Big Data Archives - TechInformed https://techinformed.com/tag/big-data/ The frontier of tech news Tue, 03 Sep 2024 15:26:20 +0000 en-US hourly 1 https://i0.wp.com/techinformed.com/wp-content/uploads/2021/12/logo.jpg?fit=32%2C32&ssl=1 Big Data Archives - TechInformed https://techinformed.com/tag/big-data/ 32 32 195600020 Clearview AI fined over $33m for “illegal” facial recognition database https://techinformed.com/clearview-ai-fined-over-33m-for-illegal-facial-recognition-database/ Tue, 03 Sep 2024 15:26:20 +0000 https://techinformed.com/?p=25552 US facial recognition firm Clearview AI has been fined €30.5 million by the Dutch data protection watchdog (DPA) for hosting an “illegal database”. Clearview AI… Continue reading Clearview AI fined over $33m for “illegal” facial recognition database

The post Clearview AI fined over $33m for “illegal” facial recognition database appeared first on TechInformed.

]]>
US facial recognition firm Clearview AI has been fined €30.5 million by the Dutch data protection watchdog (DPA) for hosting an “illegal database”.

Clearview AI uses data scraping technology to harvest people’s public photographs from websites and social media platforms to create an online database of 20 billion images of faces and data.

According to the watchdog, Clearview has not objected to the DPA’s decision and would therefore be unable to appeal against the fine.

“Facial recognition is a highly intrusive technology, that you cannot simply unleash on anyone in the world,” DPA chairman Aleid Wolfsen said in a statement.

“If there is a photo of you on the Internet – then you can end up in the database of Clearview and be tracked. This is not a doom scenario from a scary film. Nor is it something that could only be done in China,” he added.

It also banned Dutch companies from using Clearview’s services.

The DPA ordered an imposing penalty of up to €5 million if Clearview doesn’t halt the breaches of the regulation.

In a statement to The Associated Press, Clearview’s chief legal officer, Jack Mulcaire said that the decision is “unlawful, devoid of due process and is unenforceable.”

“Clearview AI does not have a place of business in the Netherlands or the EU, it does not have any customers in the Netherlands or the EU and does not undertake any activities that would otherwise mean it is subject to the GDPR,” Mulcaire added.

Two years ago, the UK watchdog (ICO) also fined Clearview AI £7.5m for the same reason.

At the time, the ICO said that even though the firm does not offer its services to UK organisations, Clearview still had customers in other regions and found it “likely” that it could still use the personal data of UK residents given the nation’s high number of social media users.

The post Clearview AI fined over $33m for “illegal” facial recognition database appeared first on TechInformed.

]]>
25552
A Coffee with… Andy Wilson, senior director, Dropbox https://techinformed.com/a-coffee-with-andy-wilson-senior-director-dropbox-stem/ Wed, 07 Aug 2024 11:40:21 +0000 https://techinformed.com/?p=24894 In this insightful episode of “A Coffee With…”, we sit down with Andy Wilson, senior director of new product solutions at Dropbox. Tracing his career… Continue reading A Coffee with… Andy Wilson, senior director, Dropbox

The post A Coffee with… Andy Wilson, senior director, Dropbox appeared first on TechInformed.

]]>
In this insightful episode of “A Coffee With…”, we sit down with Andy Wilson, senior director of new product solutions at Dropbox. Tracing his career from working on the 2000 Aardman classic Chicken Run through 15 years at the BBC to transforming workflow at Dropbox.

Wilson also discusses Dropbox’s collaboration with McLaren and how they’re helping the Formula One team navigate their data faster. He also touches on his passion for impacting the future of STEM with his involvement as a non-executive director at TeenTech, a charity that inspires young people, especially girls, to pursue careers in STEM.

Penning down his journey with a techie’s enthusiasm on his e-ink tablet — his solution for his penchant for physically writing notes — Andy Wilson proves an uplifting and insightful guest on this month’s episode of “A Coffee with…”

 


Still got a few sips left in your cup? There’s plenty more where this came from. 

The post A Coffee with… Andy Wilson, senior director, Dropbox appeared first on TechInformed.

]]>
24894
Top 10 most scraped websites revealed: Google, Amazon, Tripadvisor top list https://techinformed.com/top-10-most-scraped-websites-data-scraping-google-amazon-tripadvisor-top-list/ Tue, 09 Jul 2024 17:35:04 +0000 https://techinformed.com/?p=24257 A new report has identified Google, Amazon, and Tripadvisor as the top three most targeted websites for data scraping through APIs. The report comes from… Continue reading Top 10 most scraped websites revealed: Google, Amazon, Tripadvisor top list

The post Top 10 most scraped websites revealed: Google, Amazon, Tripadvisor top list appeared first on TechInformed.

]]>
A new report has identified Google, Amazon, and Tripadvisor as the top three most targeted websites for data scraping through APIs.

The report comes from web data collection platform Smartproxy, which has analysed millions of unique data requests made by users of its API scraping solutions over the past year. The firm has not released the total number of users who scraped a particular target, however, as it says, these figures are treated as internal data only.

According Smartproxy’s report, Google is the most scraped web platform request, with users largely collecting data for SEO analysis, keyword research, content optimisation, and market trend tracking.

Extracting data from a website is usually done using an API transfer or an automatic ‘data scraping’ program that harvests content and data from the website over the Internet.

While the latter, in some cases, has proven controversial — particularly concerning training LLMs for chatbots such as ChatGPT — Smartproxy’s report comprises data gathered via API users.

According to the firm, the results show that in 2023 and 2024 Q1, search engines comprised over 42% of all scraping API requests.

The most scraped Google categories are Search, Travel, Shopping (search, product, pricing), Images, and Ads.

Fellow search engine Bing, from Microsoft, attracts 1.2 billion visitors monthly and has reached sixth place on Smartproxy’s most scraped list.

eCommerce

 

E-commerce giants are another prime target for API data scraping. Over 18% of all requests were attributed to online shopping platforms, and over half of the sites in the top ten list are online retailers. Amazon ranked second overall.

Users scrape eCommerce websites to gather product data for price comparison, market analysis, trend tracking, and competitive intelligence.

Publicly available web data allows companies to optimise their pricing strategies, identify top-selling products, and analyse the peak shopping hours to better their offering's positioning.

According to Smartproxy CEO Vytautas Savickas, businesses tend to ramp up scraping efforts during peak shopping times like Black Friday, Halloween, and Christmas to capture the value of data generated by the rush of online shoppers seeking discounts and special offers.

Smartproxy CEO Vytautas Savickas
Smartproxy CEO Vytautas Savickas

 

“This data is crucial for gaining comprehensive insights into consumer behaviour, identifying popular product categories, and spotting emerging market trends,” he says.

“By analysing the rich datasets collected during these periods, companies can fine-tune their marketing strategies, optimise inventory levels, and tailor their offerings to better meet consumer demand,” he added.

Other popular retail sites regularly scraped for information include Walmart, which comes fourth on the list and accounts for 25% of all online grocery sales in the US.

According to Smartproxy, online businesses collect and analyse real-time data from the market leader to adjust their pricing, research bestselling products in various locations, and compare their product attributes with those of the major sellers on Walmart’s marketplace.

Online auction site eBay, ranked seventh on the list, was also described by the web scraping solutions provider as a "treasure trove of valuable data" for eCommerce businesses.

Other retailers with data that users thought was worth mining include Shopify and the popular Southeast Asian e-commerce platform Lazada, which rank 8th and 9th, respectively.

Other categories

 

With sheets of insights behind every review, Tripadvisor was the sole travel company to make it to the top of the most mined site list, coming third overall.

Businesses leverage real-time data collected from Tripadvisor to improve their services, optimise pricing, and analyse competition.

According to Smartproxy, US premier online real estate agent Zillow was tenth, with house-selling websites comprising over 3% of all scraping API requests.

In addition to house hunters, businesses are leveraging real estate data to monitor competitor listings, adjust pricing strategies, analyse user reviews, and identify areas for improvement.

Strangely absent from the top ten were social media sites such as Facebook or Instagram — despite social media emerging as the second most popular category for web scraping, with over 27% of all API scraping requests.

Similarly, Quora and Reddit were not included in the list, despite community forums accounting for 7% of all API requests.

This may be because firms often seek a multiplatform scraping solution to collect real-time data from sites, eliminating the need for a custom-built scaping API for each social media platform.

Another reason may be that some social media sites and forums do not offer APIs (only 1% of websites currently do), causing users to revert to alternative means of scraping the data.

Businesses use social media-extracted data for various purposes, including monitoring brand sentiment, analysing market trends, and conducting competitor research.

Real-time data for AI training

 

According to Smartproxy, new industries are also adopting web scraping for real-time data to improve predictive models and NLP technologies.

What is Predictive Modelling?

Predictive modelling, also known as predictive analytics, involves creating AI models that recognise patterns in historical data to forecast future events and outcomes.

This enables businesses to make more data-backed decisions and strategic plans.

What is Natural Language Processing?

Natural language processing (NLP) is a branch of AI that focuses on the interaction between computers and human language. It underpins conversational AI applications.

For example, a customer support chatbot can be trained using recent user interactions and feedback scraped from review sites, improving its ability to handle inquiries and provide helpful responses.

According to Savickas, the trend of scraping search engines highlights the critical need for real-time search data across various sectors, including the AI field, where data plays a crucial role in training AI models, optimising NLPs, and helping AI agents scrape web pages efficiently.

To access the full report, click here. Interested in getting more informed about data scraping? Click here.

The post Top 10 most scraped websites revealed: Google, Amazon, Tripadvisor top list appeared first on TechInformed.

]]>
24257
A coffee with… James Fisher, CSO, Qlik https://techinformed.com/data-literacy-coffee-with-james-fisher-cso-qlik-big-data-analytics-ai/ Tue, 09 Jul 2024 17:17:51 +0000 https://techinformed.com/?p=24240 James Fisher, CSO at Qlik, oversees the business analytics platform’s corporate strategy, vision, and strategic planning. Prior to his current role, Fisher began his career… Continue reading A coffee with… James Fisher, CSO, Qlik

The post A coffee with… James Fisher, CSO, Qlik appeared first on TechInformed.

]]>
James Fisher, CSO at Qlik, oversees the business analytics platform’s corporate strategy, vision, and strategic planning. Prior to his current role, Fisher began his career as an intern at KPMG and PwC. With two decades of experience in global software and consulting businesses — including roles at SAP, BusinessObjects, Cartesis, and PwC — the strategist has a wealth of knowledge in analytics, performance management, finance, and mobile solutions.

In this episode, Fisher delves into the critical need for data literacy and AI literacy to unlock the true potential of data and AI. He shares impactful use cases of data and AI, including efforts to address domestic violence, and advocates for the responsible use of existing data over the collection of more.

Tune in to hear Fisher’s perspectives on the tech industry and his vision for the future of data.

 


Still got a few sips left? Check out some of our previous episodes, including Zoom, eBay, and Lush Cosmetics.

The post A coffee with… James Fisher, CSO, Qlik appeared first on TechInformed.

]]>
24240
How Nordisk Film’s data strategy drives box office success https://techinformed.com/how-an-entertainment-companys-data-strategy-keeps-them-in-the-charts/ Thu, 20 Jun 2024 15:06:08 +0000 https://techinformed.com/?p=23711 Film and entertainment providers are continually exploring ways to maximise revenue streams – from franchising and merchandise to ticket sales. But the sector is undergoing… Continue reading How Nordisk Film’s data strategy drives box office success

The post How Nordisk Film’s data strategy drives box office success appeared first on TechInformed.

]]>
Film and entertainment providers are continually exploring ways to maximise revenue streams – from franchising and merchandise to ticket sales.

But the sector is undergoing a revolution, thanks to the growth of streaming services, which have allowed those at the forefront to capture real-time data around viewing figures and patterns which can be used to guide investment decisions.

Data in this sector isn’t a new thing – traditional cinemas have heavily relied on data to thrive. But what kind of data do they collect now, and how can analytics help enhance their operations?

At Qlik Connect in Florida, TechInformed caught up with Nordic entertainment company Nordisk Film to hear how it used Qlik’s data analytics engine to boost productivity and achieve significant cost savings.

Big data in Little Denmark

 

One of the oldest movie production companies still in operation, Nordisk Film has been a cornerstone of the Nordic entertainment industry since its founding in Copenhagen in 1906.

Now owned by the Egmont Group, Nordisk Film has expanded its operations across Scandinavia and beyond, encompassing movie production, distribution, and even the largest cinema chain in Denmark and Norway.

Additionally, Nordisk Film is the official distributor of Sony PlayStation consoles in Scandinavia, produces computer games and operates a significant gift card business.

As a diverse portfolio company, Nordisk Film collects data from various business ventures. According to Mikkel Hecht Hansen, head of business intelligence at Nordisk Film, the company gathers in-house data such as cost and revenue analysis.

“We also collect external data from content that is publicly available on Facebook, Twitter, TikTok, or other platforms. We manage the interactions, likes, and comments on our content on social platforms. But also, how much our movies are talked about in public areas and public groups,” he said.

But collecting so much unstructured data comes with its own set of challenges.

Houston, we have a (data) problem!

 

Previously, Nordisk Film faced several challenges in managing and utilising its vast amounts of data. The company needed to ensure that its creative processes were informed by data-driven insights, from understanding movie performance to optimising cinema operations.

However, the unstructured nature of much of its data, particularly opinion and sentiment analysis from social media and public reviews, posed significant hurdles, especially for gamers.

“Gamers are very talkative, and they’re very emotionally connected to the computer games they like. But the biggest problem we have on the gaming side is that people use a lot of irony,” he explained. This made it difficult to gauge genuine sentiment in gaming reviews.

Palads Teatret, a cinema in Copenhagen operated by Nordisk Film

 

According to Hansen, the sheer volume of cinema reviews was also overwhelming: “Each time people buy a movie ticket, we send out a survey. People are actually quite good at responding, but that’s too much information to get a handle on. There could be 50,000 reviews, each with a lot of information.”

Recognising the need for a robust data analytics solution, Nordisk Film embarked on a journey to find the right tool to address these issues.

Eureka!

 

Enter Qlik, a data analytics platform that Nordisk Film chose after a strategic vendor selection process in 2018. According to the entertainment company, the platform’s flexibility and existing competencies within Nordisk Film made it an ideal choice.

Nordisk claimed that another deciding factor was Qlik’s ‘enterprise-ready setup,’ which allowed for complete control over data sharing and access.

“We have six different business units. Some of them are in open competition with other companies, and we need to ensure to the authorities that they don’t gain any unfair access to data that they shouldn’t have access to,” says Hansen.

The implementation of Qlik transformed Nordisk Film’s data processes. By centralising data ingestion and analytics, the company was able to provide daily operational dashboards to a large user base, enabling informed decision-making across various business units.

One notable success was reducing cinema energy consumption, achieved through a detailed analysis of energy usage patterns.

“We can see the energy consumption on all our different sites, and we use that analysis for our ESG so we can find how much carbon we’re actually consuming. But it was the analysis and dashboards we created in Qlik that enabled us to find out we were turning on the lights in cinema screens hours before the screens were actually utilised,” he adds.

By fine-tuning when it turned on the power in cinema screens, Nordisk Film was able to save 10-15% on energy consumption. As such, it is now working on using the same strategy to decide when to cool and heat its different sites.

Cloudy, with a chance of migration

 

Moreover, Qlik’s cloud migration further streamlined operations, allowing Nordisk Film to maintain a small but efficient team to manage the platform. This shift also facilitated remote access to data, enhancing collaboration across different regions. Hansen explains:

“We are three people maintaining this entire platform for six different business areas with 600 users. We have over 300 different apps and solutions on the platform. So, we need to minimise the time we spend on maintenance service and stuff like that — none of which is a problem in the cloud.”

At the same time, with people located across Scandinavia, migrating everything to the cloud makes it much easier for their team to access data. “As long as they can connect to the internet, they have access to our systems.”

Hansen explains that maintaining a data warehouse platform on an internal system makes little sense when almost all of the systems his team interact with are migrating to the cloud.

“We’re actually sort of like a hybrid setup because we have a lot of our production systems still onsite. But at the same time, I see almost all of the internal Nordisk Film systems we work with are either on the way to the cloud, in the cloud, or being changed to SAS or PAS solutions. I need to follow the trend and the trend is every interaction surrounding my ecosystem is migrating in some way to the cloud.”

A coffee with... Mark Monte-Colombo, head of refurbished, eBay

Mission impossible?

 

Egmont’s strategy to focus on users meant pivoting the types of data they collected. But for Hansen, the most difficult challenge with Nordisk’s transformation was convincing everyone to come along for the ride.

“We must know how the consumption of our users is actually changing because we’re moving into a digital world; they’re changing from magazines and books to streaming,” he says.

“We had to fight the “We’ve done this before; we know our consumers” sentiment that was very strong within our organisation. We had to prove the need to be data-driven because, from a strategic point of view, things are changing very rapidly. We need to understand why.”

Hansen’s biggest challenge was to gain the organisation’s trust and demonstrate the value of the new processes. His team achieved this by creating organic use cases in areas they knew people were interested in or had the skillset to work with and showcasing them internally — but that wasn’t enough.

“It was like a two-front war; we had to find it from both sides. We started looking for the best possible use cases that we could produce for them. Slowly but surely, people started building solutions and assets on the platform. In half a year, it changed from us trying to push things to being overwhelmed by demand.”

Hansen explains that although many people are initially resistant to change, worried about their jobs being replaced, frustrated with changes to their workflow, or feeling as though they are losing ownership of their work, they eventually become ambassadors once they can tangibly see its value.

He suggests that without the Qlik software, the change would have been slower and less exponential. Indeed, it would have been harder to fight for it to happen at all.

“I think that without a central platform, different parts of the business would have attempted to create their own solutions, but they would have been small in scale. Without a centralised platform, implementing Power BI in these small pockets would have been time-consuming and less successful.”

Whilst at Qlik Connect, we also caught up with another Qlik customer, Centrica’s director of data and engineering, for a coffee.

Back to the future

 

According to Hansen, Nordisk Film is well-positioned for future growth with Qlik’s data analytics capabilities. The platform has enabled the company to build a more stable analytical framework, particularly for complex areas like its gift card business, where fraud detection has been significantly improved.

As the entertainment industry evolves, Nordisk Film is prepared to leverage data analytics to stay ahead of trends and make informed production and distribution decisions.

In addition to optimising operational efficiency and driving down costs, Nordisk Film’s ability to understand and predict audience preferences has enabled it to remain competitive.

“Otherwise, the audience will consume some of the other entertainment products out there. We’re competing for time, and it’s getting harder and harder to convince people that the great content you have is actually something that they should spend time on,” Hansen explains.

That’s a wrap!

 

Nordisk Film’s journey with Qlik offers valuable lessons for the wider entertainment and film industry. Hansen’s key takeaway is the importance of a strong data foundation and the need for effective change management to drive cultural shifts towards data-driven decision-making.

“You need to either read up on or get help with change management. Because this is not really a technical process. It’s a people process,” he says.

By embracing advanced analytics, companies can gain a competitive edge, optimise operations, and better understand their audiences. But Hansen explains it all amounts to nothing if the data is good to begin with.

He adds that despite the advances in AI across all sectors, none of the advanced analytics that people would benefit from could be achieved if time and energy aren’t invested in building a strong data foundation: “You can’t do any of the fancy stuff if you don’t have complete control over the boring bits.”

“It’s a cultural thing we need to work on continuously. Otherwise, it will never be part of business. It will be a white elephant project that people spend enormous amounts of time and energy on, but when you look at the data, no one is utilising it, still using all the small Excel sheets that they did before.”

Read about what else TechInformed learnt this year at Qlik Connect here.

 

TI:TALKS weekly podcast by TechInformed

The post How Nordisk Film’s data strategy drives box office success appeared first on TechInformed.

]]>
23711
A coffee with… Eddie Edwards, director of data and engineering, Centrica https://techinformed.com/big-data-and-ai-in-the-energy-sector-net-zero/ Tue, 18 Jun 2024 14:47:16 +0000 https://techinformed.com/?p=23545 After leaving school, director of data and engineering Eddie Edwards started his career in tech, working on CAD design and customer service systems at Centrica… Continue reading A coffee with… Eddie Edwards, director of data and engineering, Centrica

The post A coffee with… Eddie Edwards, director of data and engineering, Centrica appeared first on TechInformed.

]]>
After leaving school, director of data and engineering Eddie Edwards started his career in tech, working on CAD design and customer service systems at Centrica (formerly British Gas PLC). A company ‘lifer’, he’s still with the British multinational energy and services provider 36 years later.

A retro-gaming enthusiast who cites his vintage Donkey Kong flip screen as his favourite piece of tech, Edwards has played a key role in leading Centrica’s data analytics evolution from centralised mainframe systems to a federated lake house concept.

Today, Edwards is a strong advocate for sustainable initiatives at Centrica, recognising the importance of investing in the future. He also champions the journey of technology adoption and guiding employees towards a more tech-savvy future. He spoke with Ricki Lee at Qlik Connect in Orlando, Florida, earlier this month.

How do you take your coffee?

White, with two sugars. Stirred anti-clockwise — not for any reason, just to be different.

What was your journey into tech?

I’ve been working with the same organisation for a long time. In the beginning, it was very much a case of bringing in the technology: large mainframes, having people write on slips of paper to send them off, collating the data, scanning the data, and bringing it all together. Even then, you could see that it was possible to get more value, information, and usage out of that data.

So, I went on that journey, starting with building call centre forecasting and call demand applications, predicting, forecasting, scheduling people and agents, etc.

Now, it’s reached the point where the actual skill level to get into tech is much lower — which could be dangerous because you’ve got people using our models who don’t understand what it’s doing in the back.

You joined Centrica straight out of school, so where did your interest in tech begin?

I’d built applications during school. When I was 14, I used to do a newspaper round for a shop. I built an application that would work out what papers the customers wanted. A paperboy would come in every morning and receive a printout of how many papers they needed: 10 Times, 4 Stars, 2 Suns.

But it all began when I was around 10. I bought a PC, and I was just fascinated. Back then, a lot of time was spent waiting for your games to load, whereas I found it a lot easier just to start coding. It was fascinating to me that you could write a programme and, just by a few lines of code, achieve an outcome. Things haven’t really changed now; understanding the outcome you want and how best to achieve it.

What has been one of the biggest challenges in your career?

 One difficulty has been getting people to accept change and learning how to take them on that journey.

A lot of the conversations we’ve been having more recently are the same conversations we had ten years ago, but people were just not ready for that change. With things like digital twins and speech recognition, I can take a presentation written ten years ago and present it today, and people will go, “That’s exactly what we’re looking for.”

In the last four months, conversations with our C-suite have been about AI. They say, “We need to do AI,” but we’ve been doing it for the last ten years. Really, we’ve been using AI since the 80s, all the way from your first spell-checker. So, one of the key lessons is how to bring people along and not leave them behind.

More broadly, what kind of challenges do you think your industry is facing right now?

We’ve been going through the AI hype cycle, and we’ve arrived at this trough of disillusion. I’m starting to see a number of areas where people are thinking, “Hold on a minute. Is this really going the right way?”

There’s a lot of nervousness around security, how the data will be used, and people’s rights around the data — the GDPR-type challenges that arise. Then, there are hallucinations within that data and some of the broader impacts that could occur.

Also, our organisation works within utilities. Look at how many data centres are being built; data centres need power. For example, in Ireland, something like 23% of power consumption is from data centres, and it’s going to grow to over 40% within the next five years. That level of generation impacts the environment.

When you’re trying to drive the green agenda, as we’re doing at the moment — we’re all about a sustainable future — getting that balance right is really on our mind.

Just weeks away from a General Election in the UK, there’s a lot of talk about net zero and renationalising energy. Do you think that renationalisation is likely?

We’ve created a new net zero part of the organisation focused on heat pumps, solar, batteries, and peer-to-peer trading — community-based power generation and how that’s distributed outwards. We’ve always been big on this.

I don’t think energy will get renationalised. There were a few blips a few years ago with COVID-19, global wars, and energy prices going through the roof, but I’d say the approach taken to manage and support was good.

In my personal view, renationalisation wouldn’t resolve many of those issues. What we demonstrate as an organisation is investing in the right initiatives. We’ve committed to, I think, around £3 billion of investment in more sustainable initiatives and infrastructure. It’s just picking the right opportunities.

Other than caffeine, how do you recharge your batteries?

Walking with my dog. When working from home, you need that transition at the end of the working day. So, I get up and go for a walk to clear my head. I quite like walking down by the river, apart from when my dog dives in it.

 

Visit our 'A Coffee with...' homepage for more short articles like this.

 

The post A coffee with… Eddie Edwards, director of data and engineering, Centrica appeared first on TechInformed.

]]>
23545
Trust in AI: getting your house in order https://techinformed.com/building-trust-in-ai-qlik-connect-conference-insights/ Thu, 13 Jun 2024 18:24:06 +0000 https://techinformed.com/?p=23473 “Garbage in, garbage out” is a turn of phrase that most enterprises are now familiar with. Poor data quality can lead to incorrect or misleading… Continue reading Trust in AI: getting your house in order

The post Trust in AI: getting your house in order appeared first on TechInformed.

]]>
“Garbage in, garbage out” is a turn of phrase that most enterprises are now familiar with. Poor data quality can lead to incorrect or misleading outputs, undermining trust in AI and killing business opportunities.

According to Mike Capone, CEO of data analytics giant Qlik, success in AI begins and ends with data mastery: “Today, with the unlimited computing power and advances in generative AI available, we have the ability to spit out garbage at a breathtaking rate,” he warned.

Capone spoke in Florida last week at Qlik Connect, a gathering of industry leaders and developers from the vendor’s vast customer base. The key message at the event was the critical need to implement AI ethically and responsibly.

He explained that data quality determines any future ability to harness value from AI and analytics but recognised that managing data quality is one of the key challenges that his customers face and one that was preventing many of them from scaling AI use cases.

Capone underscored his point with a recent McKinsey survey that found over 70% of leading organisations said that managing data was one of their top growth inhibitors.

Establishing AI governance

 

How can businesses adopt AI responsibly? According to Meredith Whalen, chief research officer at market analyst IDC, who spoke at the event, companies can start by establishing an AI governance framework to help them balance pursuing new AI technologies with responsible development.

Whalen also suggested forming an AI council composed of diverse experts to provide guidance and develop practices for model transparency and data integrity.

“Our data shows that organisations are focused right now on model transparency guidelines and data integrity practices. That’s important because transparency and explainability are going to build trust among your users and among your stakeholders,” she said.

Whalen also highlighted the importance of regular employee training on ethics and responsible AI use, especially regarding security, which she recognises as everybody’s responsibility.

AI councils assemble

 

What are AI councils? Internally, it transpires that Qlik is hot on AI councils, essentially a group of diverse experts that provide guidance on AI implementation, focusing on model transparency and data integrity.

A council can also help ensure that AI development aligns with ethical standards and builds trust among users and stakeholders.

Tech entrepreneur and proclaimed AI expert Nina Schick, a member of Qlik’s very own AI Council, suggested that an AI council could also help verify AI-generated content to guarantee its integrity.

This kind of body could also facilitate discussions between industry and government to establish policies that balance innovation with fairness and societal impact.

According to Schick, AI councils can also bring together experts from various sectors to discuss AI’s future and identify necessary actions to promote responsible development and adoption.

Emphasising that computing power and data are the two essential resources driving the AI revolution, Schick believes that while computing power has increased exponentially, data is now the “new oil” that powers AI.

She argued that companies must optimise, codify, and consolidate their data to build sovereign AI and own the production of their proprietary intelligence. Schick claimed this would be critical to success in the age of AI: “All companies of the future, in my view, will be AI-first companies who build their sovereign AI.”

Qlik AI Council on stage at Qlik Connect 2024 in Orlando, Florida
Qlik AI Council on stage at Qlik Connect 2024 in Orlando, Florida

 

Implementing AI: 5 practical stages

 

The consensus at Qlik Connect around implementing AI responsibly and effectively boiled down to five practical steps.

The first involves assuring data Integration and quality control. To maintain high quality, all data needs to be integrated, transformed, and governed. Businesses also need to consolidate data from diverse sources, ensuring its integrity through rigorous quality control measures.

Only with high-quality data can AI yield reliable results. As Mayer said: “You can’t have an AI strategy without having a good data strategy.”

The second step is to form an AI council (as mentioned previously) to develop transparency and data integrity practices and provide regular ethics training for employees.

Understanding data provenance and maintaining rigorous governance are crucial. Transparency builds trust among users and stakeholders, who know that the data driving AI decisions is well-vetted.

The third step is to foster transparency and accountability, using metrics to build trust in AI systems. Equitable access to comprehensive, trusted data is essential. All stakeholders should rely on the same data pool to ensure consistency and reliability in AI outcomes. As Capone said: “You need access to complete and trusted data for everybody.”

Qlik CEO, Mike Capone

 

Adopting an agile approach was the fourth key learning from this event. Companies need to ensure that they continuously learn and adapt policies as AI technology evolves.

This includes experimenting with new techniques while maintaining alignment with ethical and market needs.

“If you are in an organisation that’s risk-averse or hesitant to get started because you’re concerned about the risks of AI, the biggest risk is to do nothing. Your competition is out there experimenting,” warns IDC’s Whalen.

The final piece of advice, more of a prediction, comes from Qlik’s VP of market readiness, Martin Tombs. He suggested focusing on AI for specific business applications rather than generic AI models. These models, he added, will need to be validated continuously to build trust over time.

“Achieving trust in unstructured data is about keeping your blast radius short and focussing on your business. I predict we will start with more generic LLMs and then evolve into domain-specific LLMs. There’ll be ones specific to call centres, support teams, salespeople, etc. And trust will come when it’s an accurate answer.”

The post Trust in AI: getting your house in order appeared first on TechInformed.

]]>
23473
Creating a data ‘gold thread’ for a building’s lifecycle https://techinformed.com/creating-a-data-gold-thread-for-a-buildings-lifecycle/ Mon, 27 May 2024 07:38:13 +0000 https://techinformed.com/?p=21965 Digitalisation is one of the greatest contemporary challenges facing construction in the UK. Whie other industries have been quick to adopt the latest software and… Continue reading Creating a data ‘gold thread’ for a building’s lifecycle

The post Creating a data ‘gold thread’ for a building’s lifecycle appeared first on TechInformed.

]]>
Digitalisation is one of the greatest contemporary challenges facing construction in the UK. Whie other industries have been quick to adopt the latest software and hardware to boost efficiency, enhance quality, and improve bottom lines, construction has lagged.

A lack of impetus and focus over the past two decades has meant that digital adoption has been slow and inconsistent.

However, that’s changing, spurred on by recent, extraordinary events that have functioned as catalysts to start bringing the sector on a par with others.

However, like the consistently used ‘slow moving tanker’ analogy, turning in a new direction takes time.

Yet seismic international events including COVID-19, building regulation changes, as well as resourcing shortages and supply change issues, have led to many construction and built environment companies embracing digital solutions to improve productivity, efficiencies, sustainability and meet regulatory compliance.

In fact, it is the weight of administration, arising from a much tighter regulatory framework for safety, which came about following 2017’s Grenfell Tragedy and the consequent Building Safety Act (BSA) 2022, as well as sustainability legislative requirements, prompted by Net Zero 2050, that is forcing the hand of many construction businesses.

Digital technology is not just a nice-to-have but a must-have to comply with the law correctly and cost-effectively.

Golden thread

 

This compliance conundrum, whilst a daily frustration, has had the welcome effect, not only of encouraging greater uptake in Con and Prop tech solutions but also prompting innovation, particularly in the way we manage and analyse information.

It is gradually moving the industry away from physical and often disjointed paper trails of documents to a digital, cloud-based ‘Golden Thread of information,’ able to map out the entire data of a building or site through its whole lifetime, from planning to occupation.

The ‘Golden Thread’ concept aims to provide the right people with the correct information when they need it.

Not only that, the information should be presented in such a way that it can be used effectively. It’s a breadcrumb trail that demonstrates how something was built, and with what, to ensure it is safe for occupation and maintained correctly.

With so many pieces of unstructured and structured data today, held by multiple systems and formats, it’s proven impossible to physically find and view all the information to utterly understand and make decisions on a building that mitigate risk.

Realising that the solution needs to be delivered digitally has accelerated the emergence of a burgeoning ConTech niche.

ConTech will continue to grow as a vertical in line with the evolving regulatory framework. As mentioned, sustainability has been a big driver for technology within the sector. However, digital building information management solutions also have a role to play with the rise of new regulation, such as the Building Safety Act (BSA), in 2022.

Safety-first approach

 

Following Grenfell, a complete regulatory overhaul was required, with a particular focus in the way the industry builds and manages high-rise and complex structures and putting accountability and transparency at the forefront of everything.

While what is required seems straightforward – having digital building information – compliance has proven more complex.

It is now a mandatory requirement for all building information to be digitised to ensure its accuracy, accessibility, and understandability, to ensure the right people can access the right information at the right time to make informed decisions, better build and manage buildings, and mitigate risk.

A golden thread of information is also required and kept in a digital format, starting at the design phase and continuing through construction, handover and all the way through the building’s lifecycle.

This building information should be a live repository, which brings all the data together about how a building has been built, and how it should be managed and operated, and providing an audit trail of any changes and decisions.

New bodies have been created, such as the Building Safety Regulator, to facilitate effective oversight with new roles and responsibilities mapped out.

Having a seamless solution for digitising and organising documentation related to building safety is vital in successfully building and maintaining higher risk residential buildings (HRRBs).

By having a central repository of digital and structured HRRB documentation, building owners will be able to be compliant and stay compliant whenever documentation is called upon — keeping people and buildings safe.

A safer, smarter sustainable future

 

While the focus of this article is on the growth of digital building information management solutions as part of Con and PropTech tools, it only scratches the surface of the tech changes that both UK and global construction are undergoing.

A bonus to all this, with skills shortages at an all-time high, is that a greater degree of process automation is taking the pressure off contractors struggling to source personnel. Digitising information management is effectively removing tedious admin, driving up productivity and freeing up staff to concentrate on higher value, ‘thinking’ activity, and effectively making businesses more effective.

It’s fair to say we’ve come a long way in a brief time, and we’re now clearly treading a pathway toward a safer, smarter, and more sustainable sector.

Digital technology is now rapidly bringing construction into the Twenty-First Century and no doubt it will soon be on a par with other more digitised sectors. As momentum continues to build, we expect to see a universal, industry-wide sea change, as a ‘digital first’ attitude becomes the rule, rather than the exception.

The post Creating a data ‘gold thread’ for a building’s lifecycle appeared first on TechInformed.

]]>
21965
How ISG is using data to lay the foundations for the future of construction https://techinformed.com/how-isg-is-using-data-to-lay-the-foundations-for-the-future-of-construction/ Thu, 11 Apr 2024 16:21:43 +0000 https://techinformed.com/?p=20587 Bringing a building to life is no easy task. Beyond all the legal issues and architectural planning, the actual building of any property, from a… Continue reading How ISG is using data to lay the foundations for the future of construction

The post How ISG is using data to lay the foundations for the future of construction appeared first on TechInformed.

]]>
Bringing a building to life is no easy task. Beyond all the legal issues and architectural planning, the actual building of any property, from a small home to a giant skyscraper, is likely to involve hundreds of parts – be that people, tools, or just plain old bricks and mortar.

So having the right procurement system in place to manage all these functions is vital to a giant construction firm such as ISG Ltd. The construction specialist has offices on three continents, but it was facing challenges when its previous pre-qualification system for procurement began to reach its end of life.

ISG head of supply chain Robert Scriven explains: “As we began to look at our pre-qualification system, we realised that the platform could in fact be a transparent view for the entire supply chain, offering access to all that data.”

In other words, the entire business and its partners could have access to one system that allows oversight of invoicing, purchase orders, subcontracts, and audits.

“We started to see that a unified system was the best way of communicating with our supply chain,” he adds. “Rather than having to speak to accounts about an invoice being paid, we could put all of the information into one place that showcases their KPIs, highlights potential risks, and allows us to work closer on ESG goals. Our aim is for those partners to grow with us.”

Speak to most of the tech world about procurement and they will say it is all about the dollar – how much can you shave off the cost off buying and where can efficiencies be found. That was the previous model many companies – including ISG – operated on. But in the sector, core goals have changed, and new technologies are being used to support this shift.

Now, heads of procurement aren’t just tasked with finding amazing deals on supplies and services, they also need to consider ESG targets. Not just their own, either, but how these are impacted by supply partners.

“There is always that pressure to get things cheaper,” agrees Scriven, “but we’ve got to build things. Sometimes you must take the hits on costs to do the right thing. But if we can make savings in how we approach procurement, that allows us to focus on hitting those other goals.”

For ISG, its supply line and partner profile is as wide and complex as the portfolio of properties it builds. It has over 7,500 supply chain partners, and 94% of these are SMEs, with over 1,000 of these comprising companies with less than ten employees. Many of these partners themselves sub-contract work, often to even smaller firms or sole traders.

ISG is a $2.5 billion turnover company and spends around $1.8 billion per year, much of which is spent down the supply chain.

“We need to know how we can watch the material flow, so we have looked at technologies like blockchain to do this. The problem is those sole traders are operating on a cash model – so there is little-to-no transactional data.

“We are in a world where we have to rely upon the data from our supply chain and the data in our systems and make sure they are aligned, so we have got a single version of truth for everything.”

Laying the first bricks

 

Overhauling the procurement system is the first step in a wider tech transformation at ISG, but as is often the case, the first step was one of the most difficult to take.

Starting in 2019, ISG execs identified the need to overhaul its procurement workflow system, and, after a tender process, turned to French unicorn Ivalua the following year, and launched its “Smartsource” supply chain tool.

For Scriven, that meant getting everyone in the room and convincing them that the “worst thing you can do is stay still”.

He identified several key areas that needed work including KPI modules, performance modules, subcontractor modules, and transparency along the supply chain, that needed work. This involved a massive analysis of data sets and a huge consolidation of vendor listings onto the Ivalua platform.

“If I was to offer advice to anyone about to transform their procurement system, it would be to try and understand your data first,” the former data analyst explains. “Most data probably is clean to some extent. Eventually you need to create a unique ID for your suppliers, but once you’ve created that unique ID out of any number of fields, you can start to utilise it, whether you use AI or not.”

This means linking unique ID codes to different providers, so they become much easier to track, to issue tenders to, and to pay, without duplication.

The initial adoption was “challenging” said Scriven, because it was one of the biggest IT upgrades implemented at ISG in over a decade. His team also found it challenging to determine what parts of Ivalua’s platform it needed, given the scope of the service available.

“The Ivalua solution can do a lot of things, but there is a question of how much you want to use it for and how much you want to pay for it,” he explains. “There are certain things we configured which we probably didn’t need to.”

This, he adds, was one of the biggest challenges of building out ISG’s procurement system – piecing together how each part fitted was as complex as laying out the bricks on a massive building project.

“When you’re building out complex services – from terms and conditions to verification, to integration with the relevant tax authorities, and much more – it can be a minefield.

“Now we have the foundations to build out all those workflows and make it as easy as possible for our stakeholders and our partners to use. We can push companies on to the platform and get them through pre-qualification in record time.”

Cementing ESG success

 

One function key to the Ivalua platform for ISG is the ability to measure risk to the supply chain. It uses nine data points from integrity to construction industry schemes with the UK government, and gives project managers a red, amber or green measurement for each part of the project. This helps the team mitigate risks and, since launch, more than 1,000 members of staff have been trained to use this.

ISG’s partnership with Ivalua is set to enter “phase three” later this year which will allow the platform to manage all subcontracts under governance rules implemented by Scriven’s team.

This means that, rather than supply partners manually discussing invoicing or payments with ISG’s accounts team, there will be one, simple system they can access for contracts, data, payments, KPIs and any other interactions with the company.

This will also benefit ISG because it allows project managers to identify which partners carry the most risk in terms of delivery, or if suppliers may face problems further down the chain, so this can be pre-empted.

“It means we can be really open with them and show them their KPIs or risks,” adds Scriven. “Often in a supply chain you don’t know exactly where the risks lie, whether it comes to ESG or companies further in the chain. Our aim is to get companies to grow with us.

“It also means we can be much more selective as to who we go out to tender with.”

It also allows ISG to manage other functions, such more controls over site access. Project managers can determine which suppliers need access to the site when and manage that through the platform, as opposed to manually checking each time.

The other key benefit will be in hitting ESG goals. The firm has committed to achieving net zero carbon emissions by 2030, and is set to introduce a carbon ‘tax’ this year which will fund sustainable innovation projects across its business.

Improving the datasets used across its supply chain will help meet this goal, explains Scriven, because it is easier for the firm to track site-by-site ESG data and put it all in one place.

“Next, we will look at what problems we’re facing in engagement with our ESG goals across the supply chain and look at the trigger points for failures, and identify these earlier,” he adds.

“If we can run an algorithm behind the scenes to identify when this might happen, we can help support our partners through early payments, or look at moving them to a different plan, or even just have a frank conversation with them, as they might not realise some of the challenges they face down the road. And that helps the industry.”

The post How ISG is using data to lay the foundations for the future of construction appeared first on TechInformed.

]]>
20587
Meet the tech firms combating global food waste https://techinformed.com/meet-the-tech-firms-combating-global-food-waste/ Fri, 05 Apr 2024 10:26:00 +0000 https://techinformed.com/?p=20454 Supermarket retail chains including Marks and Spencer, Aldi, Sainsbury’s, Tesco and Waitrose signed a letter addressed to the UK government’s Department of Environment, Food, and… Continue reading Meet the tech firms combating global food waste

The post Meet the tech firms combating global food waste appeared first on TechInformed.

]]>
Supermarket retail chains including Marks and Spencer, Aldi, Sainsbury’s, Tesco and Waitrose signed a letter addressed to the UK government’s Department of Environment, Food, and Rural Affairs (Defra) calling for food waste reporting last month.

The open letter, addressed to secretary of state of the department, Steve Barclay, requested “reconsideration of mandatory public food waste reporting,” stating that it was needed to help “measure and judge if meaningful impact is being achieved, and to encourage more action to be taken across the whole industry.”

The UK government’s current position is that it is holding back on mandating waste measurement and reporting for food businesses until 2026.

Food waste costs the UK £20.8 billion annually, and worldwide, the issue contributes to 10% of global greenhouse gas emissions.

According to the United Nations, around 13%  of food produced is lost between harvest and retail, and significant quantities are also wasted in retail and at consumption level. With this, an estimated 17%  of total global food production is wasted in households, food service, and retail altogether.

Still, the letter confirms that retailers and supply chains want to improve not only their food waste efforts but carbon emissions, and they are using technology to do so, as the letter refers to the progress that is being made with “AI and innovative tools.”

In this context, we spoke with three firms using big data, AI, and sensor technology to help the food industry become more sustainable from the farm to the shop floor.

Sustainable fields – Varda

 

Varda, a corporate spinout established by big fertiliser firm Yara, is a Norwegian state-owned company looking to bring transparency and sustainability to farming worldwide.

The firm currently has two tools. The first, ‘Global FieldID’ (GFID), uses satellite imagery to identify field boundaries across the globe. This feature assigns a unique ID number to each agricultural land plot with the aim of hosting all farming data in one place and enable sharing and collaboration across the industry.

The other, SoilHive, compliments GFID by collecting and encouraging open access to soil data in order to allow stakeholders to compare and make informed decisions.

“SoilHive is a data repository and search engine starting with publicly available information,” explains Davide Ceper, Vada’s CEO. “But then ideally, venturing into the big, submerged part of the iceberg which is the enormous amounts of data that farmers produce because they run soil tests on a regular basis.”

Varda hopes that farmers will share the data, which Ceper believes to be “extremely valuable information for firms which source from farms and those that sell to farms to improve practices.”

“If you can assess the quality and health of the soil you can help with monitoring what’s going on in a massive system at the country level or regional level, then you have a proxy for food security in the long term,” explains Ceper.

If farmers can improve soil health it can also help them yield more product out of their land, limiting the need to expand, preventing deforestation in the long run, he adds.

The way that food is sourced and distributed comprises a complex supply chain but it’s important to trace origins argues Ceper: “If you want to reduce the environmental impact of that, you need to be able to boil it down to what’s happening in each individual field around the world,” he says.

As well as increasing yield, farms can use the platform to compare practices to become more sustainable and reduce emissions. Plus, it can help in other areas such as balancing competition.

“It’s a very complicated domain, but one that ultimately is one of the most important sectors in the world because without it we would hunting and gathering berries.”

Transit – Relex

 

Finnish firm Relex claims to use artificial intelligence to reduce the food waste that happens along the supply chain.

With data collected from the end user, during transport, and trend predictions, Relex’s AI claims to forecast the number of specific products stores will need down to the individual shopfront.

The firm has collaborated with retailers such as M&S Food, Waitrose, Morrisons, LIDL and Co-op.

“We calculate the demand forecasts for each product in that store for the coming days, weeks, months and then based on that we are able to optimise inventory levels,” explains head of sustainability Svante Gӧthe.

“We’re able to calculate how much should be ordered to that store of which product and then because we’ve ordered optimal levels, we’re able to balance shelf availability and waste and keep it on the right level,” he adds.

This can vary store to store across the world, or even by country, depending on the weather or even holidays. While it may be raining one end of the country, it might be clear skies on the other and the demand for BBQ food may have risen.

Gӧthe notes that 8 to 10 % of all global emissions come from food waste and by optimising and planning for food items they can cut the waste of supermarket customers.

“What we typically see is a range somewhere between 10 to 40% in food waste reduction when we have implemented the solution,” he adds.

Altogether, Relex software claims to have saved 280 million kilograms in food waste in 2022, which equates to approximately 950,000 tons of CO2 emissions.

Packaging – Senoptica

 

Finally, to the packaging. Irish-based firm Senoptica Technologies is combating food waste using an array of complimentary technologies.

The company’s packaging technology is a colour-coded sensor that uses food-safe ink that indicates in real-time the condition of the food inside.

About half of the world’s fresh food, including fish, meat, and cheese is secured in ‘modified atmosphere packaging’.

This means that packagers will change the ratio of gases humans breathe – oxygen, nitrogen, and carbon dioxide – inside, which allows the food to stay fresher for longer without adding chemicals.

According to Brendan Rice, CEO and co-founder of Senoptica, the problem with this type of packaging is that sometimes it fails, which the sensor is designed to detect through the amount of oxygen in a package.

“It fails because the atmosphere designed to keep the food fresh has escaped because either they haven’t sealed it properly or they haven’t put the right mix of gas there to start with, or there’s potentially a defect with the packaging,” he observes.

This may be one explanation as to why food that we have bought has gone off before it’s used-by date.

Senoptica is still in its initial stages, but in the tests it has carried out so far, the firm has found as many as 5% of the food packets that it has tested had already failed by the time they reached the supermarket shelves.

“It might sound like 5% is a relatively small number, but when you consider the scale of these categories globally, you’re talking billions of packs that have failed if that figure is applied across all categories,” says Rice

“So, in the first instance, we’re trying to stop failed packs getting out into the supply chain.”

With its machine vision system, as the packets head out of the factory, its sensor technology can indicate whether an item hasn’t been sealed properly and it can be repackaged before it has the chance to expire.

Supermarket assistants can also use handheld machines to read the condition of a product, saving them time manually checking.

With digital cameras scanning the products along the supply chain, this also allows for tracking where required, and to help with security and traceability.

While supply chain tracking is nothing new, Rice claims that Senoptica’s technology is unique in its ability to automatically detect faulty or expired packs along the supply chain before they hit the supermarket shelves.

In its test, Senoptica claims that it was able to improve the detection rate of faulty packs up by 11,000 times, and it also found that a significant proportion of the food was still good after the use by date.

“That finding should be useful information for both the food companies and the retailers because it may allow them to look at the possibility of extending the use by dates,” and help keep perfectly edible food from going to waste.

The post Meet the tech firms combating global food waste appeared first on TechInformed.

]]>
20454