KFC’s “finger lickin’ good” data warehouse strategy

When Mick Ayling joined KFC’s UK and Ireland operations a little over a year ago, the data engineering manager wanted to change the way that the fried chicken chain worked with its data.

His aim was for the business and its parent company Yum! Brands (also owners of Taco Bell and Pizza Hut) to be able to order data extracts with the same ease and speed that its chicken wings and skin-on fries are served up to customers.

He also wanted to be able to use warehouse data to answer the questions that will help support KFC to move forward and expand.

Speaking at a seminar at Big Data London in Olympia last month, Ayling started by outlining the size of the business and some of the challenges it faced as it sought to move its legacy systems into a cloud-based warehouse.

To kick off, he served up some sizzling stats: There are over 1,000 KFC restaurants in the UK and Ireland, and the chain sells half a million meals every day. Its restaurants in these territories sell a billion items a year, which amounts to £1.7bn in sales annually.

But these tasty morsels are dwarfed by the data sets the Colonel’s empire produces. “Our biggest table has 8.5bn records in it,” Ayling explained.

However, KFC’s data platform was getting older: Its hardware was about five years old, running off a single server that was around 10 years old, and the business needed to move to the cloud.

According to Ayling, these legacy systems were starting to cause performance issues. “Once you start getting big tables, indexes slow down, queries slow down, so it just builds up and causes problems,” he said.

The chicken operator’s journey to the cloud meant transferring huge swathes of data that was still sitting in old systems, much of which needed to be integrated for wider projects.

“Our parent company Yum! wanted a better overview about what is happening across all their brands in all their territories. We could provide data to them this way, but it is not streamlined; we needed to create extracts and send them over; as an organisation we needed a better way of working,” he said.

Cloud warehouse

The answer was a new platform, a new data warehouse – and the first few tech decisions were straightforward, Ayling claims.

The data team selected AWS as its cloud platform of choice as KFC had a pre-existing relationship with the vendor.

For its databases, SaaS-based storage and analytics solution Snowflake was chosen, according to Ayling, “for its ability to scale”.  And for front-end reporting and analytics the fast-food chain opted for MicroStrategy – a product the data engineer claimed, “works well with Snowflake”.

Strategic partner Brillio, a US-based technology and digital transformation partner, also supported the move to a new data warehouse.

But the fast-food firm still needed tools to do the heavy lifting. While there were a huge number of Extract Transform and Lift (ETL) tools on the market to help them transfer data to the cloud-based warehouse, many of them performed this task slowly, due to the scale of data and the different systems it was all sitting on.

Ayling was keen, however, to try Matillion‘s low code/no code platform, which integrates data from legacy systems into cloud data warehouses.

Low-code/no-code (LCNC) platforms have gained popularity in recent years because they allow enterprise developers to quickly build applications by relieving them of the need to write code line by line.

Popularised by the likes of app platforms such as SalesForce’s Lightening, Microsoft PowerApps and Google AppMaker, LCNC platforms enable business analysts, office administrators, small-business owners and others who are not software developers to build and test applications.

For data transfer and analysis, Ayling was convinced of the merits of Matillion – based on his experience of using the solution at his former workplace.

He recalled: “We were working on a warehouse project – a like-for-like moving data into the cloud. The management was not prepared to accept aggregate testing for user acceptance testing – only field-testing. Yet our biggest table had 400m rows and was fifty columns wide; with 20bn data fields – and they wanted us to carry out a comparison analysis between two datasets.”

He continued: “We realised that we could do it on Snowflake – but the challenge was how to get the data into Snowflake? We tried Microsoft‘s SSIS but found that it took a long time to run – and we could only move a year’s worth of data before the thing just came to a standstill.

“We looked at other tools and experienced similar issues. And this was all taking up development time… So we looked at Matillion’s pay-as-you-go option: and we got everything connected at 9am in the morning, and within 15 mins I had created my first extract into Snowflake, and it worked brilliantly.”

Ayling added that within the space of a day they’d managed to achieve everything they needed to, leaving him convinced this was the right solution for KFC too.

He added that the fast-food firm was also keen to employ more people capable of analysing the data’s value to the business, and less on engineers storing and extracting it.

“Instead of having to recruit engineers and coders that have a whole load of skills in Python, Airflow and dbt – we want to be able to recruit people who can work with the data so the business can understand it and Matillion helps with that. It doesn’t require an extra level of skills to get the thing working,” Ayling explained.

Management buy-in

He still had to convince his bosses that he’d made the right choice, however. Although it was a two-year project, he was asked to produce some benefits to the business by the end of year one.

The data team’s solution was to break it into waves: delivering different bits of the entire warehouse at different times so that management could start seeing this early delivery.

“Now that’s a really great way of approaching it – until you start thinking of how that’s going to work with analytics if the data is in different places,” he pointed out.

“For any other analytics we wanted to do, if we’re trying to connect data from two separate sources there’s only two ways to do it. Extract everything into a third system or choose one of them to be your primary server – which means we gain no benefit from the work we’ve been doing with our warehouse – because we’re pushing everything back into a single server…”

Matillion came to the rescue again with a feature called Data Loader which enabled the data team to ‘point and click’ to copy tables into [Amazon Simple Storage Service] S3 and to load into Snowflake.

“You can effectively simulate the data being produced in our current warehouse in Snowflake without having to do all that heavy lifting through ETL,” Ayling added.

Keen to assert his impartiality and independence, but aware that he was sounding like a Matillion flag waver, KFC’s data guy then served up a disclaimer to the Big Data LDN audience.

“I’m not being paid by Matillion to say any of this – they haven’t even given me a free T-shirt – or even a coffee!”

In fact, he’s talking to the vendor on where it could make some improvements, which he also shared with delegates.

“Tighter integration with Git would be good so that we can keep track of source code; and there needs to be more support for data governance and data lineage – Matillion used have this in there but they took it out and replaced it with an API – which is handy but there are only a handful of vendors who support that at the moment,” he noted.

Another criticism with the tool, he added, was given Matillion is a pay-as-you-go tool, it’s quite hard for the user to see how many credits they’ve got left or when they’re going to run out.

According to Ayling, in the future KFC would like to use its new data process to make basket analysis easier, which is a data mining technique used by retailers to increase sales by understanding customer purchasing patterns.

“That’s one of those things that, in the past, the business asked us to do, but we did it very rarely because it takes around four hours.

“But now we’ve got this historic data in Snowflake, and we can port that over for tweaking – these jobs that were taking four hours on prem can now run in under 30 seconds. And that’s a real game changer for us.”

And yes, he even added, “It’s finger licking good!”

At the start of 2023 KFC was forced to close around 300 outlets across the UK after it was hit with a ransomware attack.