AWS re:Invent 2023 - Unlocking Data Insights: Amazon Redshift's Latest Innovations for Modern Cloud Data Warehousing

Name: AWS re:Invent 2023 - Unlocking Data Insights: Amazon Redshift's Latest Innovations for Modern Cloud Data Warehousing
Uploaded: Nov 22, 2024

Transcript

Auto Scroll On

A

All right. Thanks everyone, for joining our session today. My name is Neja Rintala. I lead the product management for Amazon Redshift. Today I'm very delighted to introduce my cos Speaker Matt Sandler, who is a senior director of data and analytics at McDonald's who is here with us to share their journey at, uh, McDonald's with AWS and Amazon Redshift. All right, so this is our agenda today. First, we'll start with a brief discussion on the trends in modern cloud data warehousing. We'll talk about Amazon redshift evolution advancements in the last few years, including some of the exciting announcements at invent this year. And then Matt is going to talk about how all these innovations come to life at scale using, uh, McDonald's as an example. Broadly, there are four trends that we see in data warehousing. The first one is the boundaries between the Data lake data Warehouse and machine learning are blurring and the use cases are converging. We see that customers just want to maintain a single copy of the data, and they want to do all types of analytics. On top of it, SQL Spark Machine learning and more. The second trend is near real time access to the data. They want to set up seamless flow of transfer of the data from operational systems from their transactional systems into the analytical stores so that they can be much more proactive in their business operations and improve their customer experiences and take it as a competitive advantage. And as you have all these data in one place, you want to make it available for all your users in the organisation. This could be data scientists, data engineers, developers and it ease of use is a critical element of it. And as you make data more and more broadly available in the organisation, making sure that you have the right security and governance controls in place is very important so that you can make sure the right users have right access to the data and all the data is centrally. Access is centrally managed and you are able to govern the data access many of you might know. In 2013, AWS has introduced Amazon Redshift as the first fully managed petabyte scale enterprise grade data warehouse, which made it really simple and cost effective to analyse large amounts of data. And this is a significant leap from the data warehousing the traditional data warehousing on premises because they were not elastic, they were not. They were expensive and much more harder to tune and operate. Since then, we have added a lot of innovations to Redshift to keep pace with customer demands and expanded the horizons of data warehousing use cases it could. It includes business, intelligence and analytics at scale, but also machine learning, Spark integration, data sharing, monetizing your data and others. So I just wanted to quickly walk through the evolution of redshift in the last couple of years, since since its inception. So in 2013 we launched redshift as the best price performance cloud Data warehouse. And in 2017, we have introduced Data Lake querying. So you're able to query the open data formats in Park KOR CJ. So in Amazon S3 data Lake. In fact, Redshift was the first query engine to introduce High Performance Analytics on top of data lake. In 2018, we have introduced elastic scaling to redshift, whereby when you have more users and more queries, redshift is able to automatically add capacity to maintain the performance. In 2019, we have introduced Redshift Manage storage, which I would say is a breakthrough for redshift, with which you, as a customer, was able to scale and pay for, compute and storage independently with this Model. The durable storage is in S3 and the local cluster nodes. The SSD nodes really serve as caches for the data and the redshift manage storage is a software layer intelligent software layer that decides what data should stay in caches and what should be in the durable storage. So building on the redshift manage storage in 2020 we have introduced redshift data sharing with which you are able to actually spin up separate compute for your analytic workloads, for example, ad hoc queries, a different type of compute data science, different type of compute. They were all sharing the same data. So without making any copies of the data, you are able to provide compute isolation for your read workloads. We also started investing in deeper integration with a W services. So we have introduced streaming injection, direct streaming from streaming services like K, sis and MSK manage streaming for Kafka. We have introduced last year zero etl from operational databases like Amazon, Aurora and we also started expanding on the analytics side, we have made Spark access easy from em, R and Glue. And last year in July, we have introduced Redshift Serverless with which we have automated compute management. So with Redshift serverless, it automatically provisions the compute, it scales the compute and it takes care of things like patching so you can just focus on getting the business insights and serverless is pay for use. So, for example, if you had a situation where you had in an hour, you are just doing work for 55 minutes, you literally would pay for that workload duration, which is five minutes. So it's a pay for your system. We also integrated with AWS Data Exchange so that you can easily consume third party data sets from your data warehouse easily. And you are also able to actually take your data warehouse data and make it available to your customers, your partners, your business ecosystem, licence it and monetize it. So with all the work that we have been doing, we have tens of thousands of customers using redshift. Today they are collectively processing exabytes of data. We have tens of billions of queries per week. Even some of our new capabilities are showing tremendous numbers. For example, zero L from Amazon Aurora. In internal tests, we see 1 million transactions per minute data coming from multiple Aurora databases landing in Redshift, making it available for analytics within 15 seconds. Our streaming injection for from a single stream, we see a throughput of 2.5 gigabytes per second. And with redshift ML, which is a capability with which you can actually use SQL to create train and invoke models We have customers doing in database inferences more than 10 billion per day. So when we think about redshift innovation and where do we want to spend our time? What should our road map look like? There are broadly four pillars that we work on from day one. We have been focused on offering best price performance at scale. So we do everything behind the scenes so that we can make sure you are getting best value from the system and you are also able to scale seamlessly as your needs grow. The second pillar is unifying all your data, so we want to make it as simple as possible to bring data into Redshift and also, when relevant, query the data in place. The third, the third pillar, is being able to work with all the data for each analytics. We want you to do a SQL Spark machine, learning more and more generate via applications, using all the data in Redshift and all this doesn't matter if you don't have a secure and reliable platform. So this is a fundamental for a W services where we want to make sure that the platform is secure and reliable and you are able to collaborate on data with confidence. So let's start to dive in a little bit to understand kind of what are what are some of the innovations and what are some of the new features here On the price performance, Redshift offers up to 66 better price performance than alternative Cloud data warehouses. We have a lot of performance capabilities. Columnar storage, MP P processing, pretty sophisticated cost based optimizer materialised views, compilation caches, results at caches. There are a lot of performance improvements that contribute to the lead that we have in terms of price performance, and redshift also scales as the number of users and queries grow so that we can maintain the maintain the costs as your usage grows. For example, this is a scenario of a real life workload dashboarding application where you have lots and lots of like high concurrency workload, low latency, very small queries. These are very common pattern that we see in the fleet. We have done a lot of optimisations, and in this scenario, redshift is able to offer up to seven X better price performance than alternative cloud data warehouses. So we continuously look into the query patterns in our fleet to see what kind of optimisations we need to do so that we can be a lot more customer focused in terms of the innovations we are doing around performance. For example, earlier in the year we have introduced a efficient CPU encoding with which we are able to speed up the spring processing by 63 X. Similarly, we have speeded up encryption. We have made our Auto Scaling to be much more functional by adding right operation support, and sometimes we do these performance optimisations behind the scenes. You just experience the benefit. For example, we recently started on boarding graviton instances on our server list, with which we are seeing up to 20% better price performance and customers do see these benefits in their deployments. Global Foundries is one of the world's leading semi semiconductor manufacturing company, and they see three X better performance with RedShift compared to their own premises systems. They were able to improve their Ines SLS. They were able to do their load data loads much faster and bring a lot more data to cloud for analytics using redshift in addition to the core performance, Optimisations Redshift is also has industry leading predictive optimisations, which means that the system is continuously monitoring your workload to figure out the strategies for improving the data layout. Improving the compute management Redshift already has things like automatically figuring out the sort key distribution, key and coding sorting the data automatic materialised views so that we can seamlessly speed up, um, repetitive queries, automatic workload management. So these are all the capabilities we have in the product. The benefit of this is as you use the data warehouse more, the performance automatically speeds up at re:Invent. We are very excited to introduce a new capability in Redshift called multidimensional data layouts, so this is also a predictive optimisation. This is a very powerful sorting technique with which we are able to speed up repeatable queries. Unlike traditional sorting methods, which sort the data by a single column like what you see here. A Cost column. The multidimensional data layout sort the data based on the incoming queries. Actual query philtres And the benefit is when you have large amounts of data and tables, the scans, the time it requires for us to read the data is much, much faster. And in our internal experiments, we see up to 74 perform 74% performance improvement compared to the tables that have no sort keys. And the good thing with this feature says, you don't have to do anything. The system is observing the workload, it figures out. OK, is there single sort key is better. Single column sort key is better. Or should I go for a multidimensional data layout and it adopts the right strategy. And when it comes to the performance, it's also not a single cluster performance with Redshift data sharing. We have thousands of customers that have deployed Redshift in a multi cluster architecture with data sharing. You can share live and transaction consistent data within an AWS account across accounts across regions across Organizations. There is no copy or movement of the data. So we have customers deploying in these kind of common multi cluster architecture patterns, like a data mesh where you have different business groups. They are all collaborating on data. No data movement required our hub and spoke Model where you bring data from different sources, put it in a central etl or a data processing environment. And then you make it available for different analytics like ad hoc queries, machine learning, different workloads using different compute. So at re:Invent, we have extended our data sharing capabilities and we have introduced something called multi data warehouse rights. So with the data sharing capability we had before you were able to scale read workloads so you can put your dashboarding workloads in a separate cluster your data science in a separate cluster. But with multi data warehouse rights, you can actually scale your data processing as well. For example, you you might want to say OK in the night time. I have a large amount of data to be processed, so I want to put that in a different cluster a large cluster. But throughout the day I have incremental data trickling feed coming in so I can use a different size cluster and maybe once in a while you have, like a weekly compliance report or monthly compliance reporting to be done, you can use a different cluster, and the beauty is all of them are able to operate on the same data. So with multi data warehouse rights, a data warehouse can write to a database in a different Redshift data warehouse. And once the data is committed, it is visible to all the other data warehouses that have permissions to access the data. And another use case for multi data. Warehouse rights is live collaboration. So many Organizations have Entity 360 like customer 3, 60 type of data sets. And they want different teams to collaborate and build those data sets. So this is an example where you have S sales, marketing, finance, everybody working on that customer 360 data and they can populate different columns. They can add data. So you are collaborating on live data sets. We talked about how, uh, how important it is to have ease of use when it comes to scaling analytics in the organisation last year, as I mentioned in July, we have made Serverless generally available and serverless simplifies the overall data warehouse management. It takes care of provisioning, the compute scaling, the compute, the patching workload management. Everything is taken care of and it's a pay for use system. And one very interesting pattern that we have seen in the last year with our customers is they use a combination of provision and serverless redshift data warehouses to form multi cluster architectures. So this is an example. Peloton is a fitness equipment company. They are known for their Internet connected bikes, where actually the fitness lessons can be streamed on demand. So they bring data from different data sources and they process it with Spark D BT. They use airflow for orchestration. They bring the data into a central redshift cluster, which is a provision cluster. And then on the consumption side, they have different patterns of workloads, ad hoc queries, looker dashboards and tableau dashboards. So all these different workloads, they have a combination of provision and serverless, and the nice thing is, they are not duplicating any data. The data is in the central cluster and all the consuming clusters are simply reading the live data from the central environment. Play is another example where they have hundreds of terabytes of data. They are one of the early users of serverless, and they have used serverless, too, as a data sharing consumer so that they can make make it available for ad hoc queries to their users. And the benefit is by clearly separating these kind of workloads. They are able to maintain their SLS on the recording, the steady state workloads, but also improve the data warehousing costs in their in their case on a monthly basis up to 20 percent. So we have been enhancing serverless with a lot of capabilities throughout the year since its general availability at re:Invent, we have launched a number of monitoring and manageability improvements in Serverless, for example, cross region snapshot support with which you can do disaster recovery. We have things like mac RP with which you can say at any point of time, I don't want serverless to go beyond this much resources, then secret manager integration. We have something called custom domain name so you can give a user friendly name to access the cluster. So a number of these improvements really make the experience with serverless much more seamless, simplified in terms of operations and also have a robust data protection. One of the exciting capabilities we launched, actually as part of Peter's keynote, is a redshift server, less AI driven, optimisations and scaling. And this is one feature we are super excited about. The idea is very simple. As a customer now, with serverless, you have a very intuitive way to specify your price performance requirements to Redshift. So you can say I want to optimise my system for cost or I want to optimise system for performance, give me the best performance possible or keep it balanced. The default is going to be balanced, but you can adjust the slider as needed, and serverless is going to take care of everything. The remaining, which is provisioning the resources, scaling the resources, deploying the optimisations so that we can actually meet this price performance level. And the other nice thing with this feature is as your workloads change. For example, you might be processing a terabyte of data to today and you are meeting that SL in 10 minutes and tomorrow your data size double, but you still want that job to be finished in 10 minutes. This particular capability is going to help with it. So it is going to automatically scale the resources so that not just we meet the performance level, but we maintain the performance level as workloads change. This is a capability again. In preview, uh, we request you to try it out and give us feedback. So let's move to the next pillar, which is unifying the data with zero TL. So in the last 12 to 18 months, we have been spending a lot of investment to simplify data ingestion into Redshift because we have been hearing from customers that building and managing pipelines is harder. And sometimes these are fragile, which means you are going. It's going to lead to delays in getting access to the data. So and sometimes these delays can be missed business opportunities right where the insights from the transactional data may be are time sensitive. They are valid only for a certain amount of time. So they really want to get access to the near real time data. So this is what we have been focusing on near real time ingestion into Redshift. And this is where AWS vision is around the Zero L, where we want to provide an easy and secure way to enable near real time analytics on petabytes of transactional data without you having to worry about the injection pipelines and management. Once the data is in Redshift, once you start to explore the data, you might still want to say OK, I want to transform this data. I want to aggregate this data. You can do that. But the point of zero L is getting you the access to the data for analytics as quickly as possible. So we have launched the general availability of Aurora MySQL zero ETL integration. Um, end of October early November time frame. It has been in preview for several months. We have been getting very good feedback and it is all incorporated. We have launched the MySQL general availability at re:Invent. We have launched three more zero etl integrations into Redshift Aurora Post SQL to redshift DynamoDB to Redshift and RDS, my sequel to redshift. So all these capabilities are in preview. You can try it out. And working with zero L is very, very easy and this is a consistent experience across all these zero TL integrations. So you go to your source system. Could be Aurora could be RDS. And in this case, this is dynamo example where you have some new option called integrations and you start working with it. You just click on it. You choose your source dynamo db table and then you choose your target RedShift data warehouse. You configure the resource policies, you want to make sure it's all secure. And on the Redshift side, you just need to accept the integration. The way you accept the integration is just create a database from that integration that you have created in your source system and that's it. And the data automatically starts flowing. It starts replicating. It's a change. Data capture based replication. So as soon as the data lands in your source system, within seconds, the data appears on redshift and you can go to redshift query editor or whatever the tool the B I tool that you are using and start querying the data. This example is a DynamoDB example, which is a key value system. So we are bringing the data as a super data type in Redshift Super is a semi structured, data json data type in Redshift. So we are bringing the DynamoDB data super. You can navigate it. You can convert into relational so you can do any post processing that you want to do after the data lands in Redshift. So zero EL has been tried, tried by many, many customers, just a couple of examples to highlight what some of the common feedback the the things in our range is to highlight the pattern. One common theme is it used to take months for us to set up the analytics environment. Super easy. Now that's a common pattern. Another pattern is I am able to replicate data into Redshift without impacting my production systems. Because the zero etl happens at a storage level. It's a storage level replication. You are actually not reading using any of the compute, either on the source or on the target, so that that's a helpful, uh, scenario for customers and, of course, the near real time access to the data so that you don't have to do this in a Batch fashion. Another capability we have added last year is streaming ingestion. This is a direct streaming ingestion into redshift by avoiding S3 as a staging area. So with this, you can take the data from Amazon, Kinesis or Kafka and directly ingest data into redshift. And the way it works again is pretty straightforward. Which is you create a materialised view. In redshift, you point it to a Kinesis stream or a Kafka topic, and data automatically comes into Redshift. As new data lands on the Kinesis Kafka streams. The materialised view on redshift is refreshed, so the idea, as you can see here the near real time data, is a focus. And also simplifying the data ingestion into Redshift is a focus for us, and we have enhanced streaming injection with capabilities like cross account support and purging some of the streaming MVS. The data may not be relevant for too long, so you need, like a delete capability from this materialised views. This is something we have introduced. There are many customers, thousands of customers using streaming injection. In fact, streaming injection is one of our most widely adapted features. It's because of its simplicity. Join me is one example where they are using streaming injection for risk control on their user financial activity like recharge refunds, rewards et-cetera where they can see all the risk related activity in one place. And they can evaluate on a near real time basis, unlike the previous situation where they were doing it hourly basis, thereby improving the efficiency of the overall business. So we have seen how we have made it easy to bring data into redshift. We are also continuing to expand on querying data in place. So at re:Invent, we have launched the general availability of Apache iceberg support. So redshift already supports par json. We have added support for Delta Lake hoodie and Apache Iceberg is in addition to it, so you can easily query the data from iceberg. You can join it with the data in redshift and just start leveraging it for your analytics. It is one of our customers who has standardised on iceberg is able to leverage redshift to read the data. We also invested on Data Lake performance. So when customers standardise on data lake keeping the data in open formats, we want to make sure we are providing the best possible performance as well. So it, uh, with the, uh, with the Data Lake, we have improved performance up to 45%. 11 of the interesting features that we have done we actually launched. A invent is incremental refresh support for materialised views on data lake data. So, for example, you have your data in Data Lake, but you want to power a dashboard with that data, this feature will come in handy where you go to redshift, create a materialised view, and it automatically gets refreshed. And you get access to the latest data with, uh, with very good latency moving to the third pillar of focusing on the analytics and machine learning. The first type of analytics that customers do with redshift is SQL. So we continue to invest in SQL. We have launched a lot of new SQL capabilities like merge, roll up cubing grouping sets so different types of SQL syntax and recently qualify. And the idea is you are able to build rich applications but also migrate easily from the traditional data warehouse systems. And one of the new capabilities we launched at re:Invent around SQL is called Glue data catalogue views. So this is a cross a W service feature and the idea with this feature is you create the view in Glue catalogue Glue data catalogue once, and you can use it across multiple engines so you can query it from Redshift. You can query it from Athena. You can query it from Spark, but you are secure. The data once you are creating that, creating the view once you are securing it once. So you are not repeating it three times for access by three engines. So this is a pretty handy feature. This is one of our common requests that we get from customers. As you can see in this example here, the first point here shows you create the view in Redshift. Then you can say I want to customise it for Athena dialect. So we have. You can customise as much as possible for the specific Athena. Then you the Athena user would come in and they can query the view. But as an administrator, you are just managing the permissions for the view. Once we are, we are also excited to introduce a few generated bi capabilities in redshift. The first one is now redshift. Machine learning is able to access large language models LMS so redshift machine learning as I mentioned briefly earlier. You can actually use SQL to create train and invoke models. We have extended the redshift redshift machine learning with what is called Super, which is the semi structured data type. We have added support to it because of which now we can do integrate with SageMaker jumpstart and you can start in working these large language models directly from Redshift ML. And there are a lot of models like text generation, text summarization and then text classification. And we hear the requirements from customers saying I want to summarise the product feedback. I want to classify the entities on the data that is stored in redshift, so the large language models LLM support is going to help with it. Another capability we have introduced is the generative SQL capability in Amazon Redshift Query Editor, which is our out of the box query experience with which you can actually now improve your query authoring experience by asking queries by asking questions in natural language and get SQL recommendations. So the generative SQL analyses the user intent looks at the past query history. It analyses the scheme of metadata that is available to provide accurate recommendations, and this is a conversational experience, so you can actually get business insights by asking questions without having to understand complex organisational metadata. So this is a feature available in Query Editor that you can write it out, moving on to the final pillar here. One of the nice things with Redshift is we offer a lot of security capabilities out of the box, so we have many authentication mechanisms. Fine grain access, control, auditing, encryption. Everything is there out of the box. There is no extra cost for security with RedShift, and we continue to improve our security capabilities. Now we have column level permissions, row level permissions, dynamic data masking. And at re:Invent, we have added support for metadata security. So if a user has one access to one schema out of the 300 schemas that are there, they can just see one schema and query the data so metadata. Security is very important in multi-tenant environments, and we also added support for fine grain access controls on nested data, lead binding views. We have introduced table level controls on data sharing consumers. One of the capabilities we have introduced you might have heard as an announcement at invent is redshift integration. This is again a cross analytic service feature, which is a unified identity across all AWS analytic services. And the idea is redshift integrates now with I Identity Centre to enable Organizations to propagate trusted identities throughout the stack. So it's pretty straightforward, which is you as a customer. You go to identity Centre, you configure your provider, it could be Microsoft and one log in you just configure it in one place and those organisation identities have a Single Sign-On experience throughout. Aw, stack so you can come through quick Sigh query editor. Go to Redshift, Go to Lake Formation, go to S3 that entire stack users don't need to re ath it again and again. The same identity is propagated. And the nice thing is administrator, because these are identities that are passed through you can. You can do fine grain access controls using the organization's identities. So this is going to simplify the use cases where you have more than one use more than one red ship more than one, a W service involved. Another capability we recently launched a few weeks back is redshift multi, a Z deployments with which we are raising the S a of redshift to four nines instead of three lines in a multi multi a deployment. Redshift is deployed in multiple availability zones. So in the rare cases of an A failure, your business continues to operate. There are a couple of nice things with the multi az that are that are differentiating and that you don't get with other solutions. One is the endpoint. The application connectivity is not interrupted, so we meet the same endpoint. But behind the scene we are failing over the second one is the capacity that we are going to keep in the other availability zone. It's not just sitting around doing nothing, it's not active passive. Instead, when you are in the healthy situation, both availability zones, the capacity in both availability zones are actually serving the workloads. So you get much better throughput. And in the case of a failure, which is a rare scenario, the it will be the request will be continued to serve from the available availability zone. So just to summarise, we have been enhancing Redshift with the different innovations and the idea is bring together the data from all your data sources make it very, very easy to access and make it available for all types of analytics not just SQL but Spark machine learning gender TV applications and have a secure, reliable, performant and scalable platform. So with this, I would like to invite Matt Sandler to talk to us about, uh, their journey with McDonald's. And Matt has been a great ally and advocate for Redshift great partner. So I'm eager to hear from you. Thank you so much. Very cool new features to kind of summarise. And, uh So I'm gonna talk about with Redshift and how we use it and, uh, applied it here over time. So let's see where there we are. So this started a little bit of a blog post where we sort of shared, uh, you know, driving the McDonald's business one for one bite at a time. And, uh, i'll go forward here. So, background context. McDonald's. All of you probably know your local McDonald's. And when you travel, you, uh, hit a few. But a lot of people don't realise just how big we really are. We have 40,000 restaurants around the globe. Only about 14,000 of those are in the US, so by far the vast majority are now overseas. We're in over 100 and 20 countries. Last year we did about 100 and 18 billion in restaurant sales. And just under 1% of the world's population actually eats at McDonald's every day. Yeah, so quick question. Think about this for a moment. What's your favourite menu item? I'm gonna count to three. Yell it out. I'll hear it. 123. Oh, I think I I I've gotten different answers. I think I heard Quarter pounder and cheese best. I don't know how it was said. Sometimes it's fries. I love the McRib II. I know I'm out there and my, uh, my three year old loves happy meals. It's it's fantastic. So we all have memories of McDonald's or many of us do, uh, going to the play places and being there. It's, um, it's a great gathering place for a lot of people, So it's a It's a huge chain, really quite dominant in the restaurant area. So So let's tell talk about technology at McDonald's. We've been around since 1955. They're actually reasonably large, you know, not as big as we are now by you know the eighties without a lot of technology, right? We we were known for if you ever seen the Founder, it's actually kind of a great movie. Um, they really known for this operational reliability, consistency, value, cleanliness. These are sort of the hallmarks of of our brand when you think about it. But we've really grown in the last five years. From a technology point of view. I have just a little bit of there. We go, so digital channels are now driving our growth. And so our we we now have a mobile app. That is, last I looked top food app on the Apple store and Google it comes with we're now nearing to be one of the world's largest loyalty programmes. On top of that, our kiosks are super popular. I've been to the airport a bunch of times, and I see like a long line at the kiosk. No one in line for the front counter and then, you know, outdoor menu drive, uh, outdoor driver boards, kind of very engaging. We also have a delivery programme that's quite big, and now you can keep having new features with our mobile app. For example, when you order The food is hot and ready, right? As you show up, there's no weight. You can also do delivery to the home through the mobile app. So as you can imagine, this creates an enormous amount of data, a lot of opportunity, both to understand our business, but also to build data products that drive the business on top of these digital platforms. From a technology point of view, given our growth, like a lot of very large companies that have been around for a while, we have a lot of country specific solutions that kind of emerged over time. So we have variations in our point of sale. We have different supply chain systems around the globe, different mobile app, variations as we got going fast and marketing. But now we've got a big push at the technology level to bring it all together. Globalise, standardised, make it operate as one global stack and we're making progress on this. So when it comes to the data side of all that digital data and all this up systems we we find at the start of this journey we had silos of data, performance bottlenecks, surging amounts of data as these things start to roll out and get so much usage and then big delays in data processing and, of course, data, insights or business insights. So we needed to bring the that together in a unified way for analysis, look at on a timely basis and then make it available for for self service and access across the company. So I'm gonna tell you a little bit of our journey with Redshift in brief. Um, and I'll go back through these. We started with a A one departmental Redshift cluster. We then went to all in on AWS and Redshift to be fully based there. And then we've evolved with these new cloud native features, some of which you saw in the prior presentation that we'll cover here. So when we started, this is a few years ago, we had our on premise data warehouse there in the middle of the slide. And you can see, uh, we we fed S3 with some of our core transaction, and we brought in this new digital capabilities of the mobile app information the back end services and we created a US customer 360. Many of you probably have something similar for your, uh, from a large company. That's customer. Um uh, large company and facing And, um, we brought We have customer sliced and disable. We have offer redemption information, sales by channel, restaurant menu. And so we did this in close collaboration with Redshift, the teams sizing and tuning and resizing as needed. And as you've seen today, a lot of the need to do some of these things has has gradually gone away. So then we made this was successful and we had a number of good, good quality results with that. But we made the big shift based on that success to go all in. And this was where we started with our, um all that core data that was in that on premise data Warehouse went to for customers offers sales transaction, all the supply chain information. We put that into S3, and we use talend as a traditional ETL to fill our global data warehouse. However, at our scale, that wasn't a Redshift at the time, Wasn't able to handle that. And so we added four more, uh or, uh, country specific data marts, in addition to the one we had before and this took over 500 E ETL and ELT jobs in order to bring that all together. So a lot to manage. But it's working. It's actually giving us a lot of value along the way. We had a lot of AWS engineering visits, um, new feature requests that we put in for performance and workload management, and we'll see some of the benefits of that in a moment. And the usage we now are able to support over 3000 , B, I tool users and opened up for the first time to direct access on these data sets. So these are our large scale main data sets and over 100 B I user or, uh, SQL analysts at the time start to work with the data, our first exposure to self service for the large data at McDonald's. Now where it gets really interesting. For those of you who've already got into this journey is these new features that have come on board and enabled us OK, so this is a very similar diagram. And in the middle, you see that large global data warehouse with the country specific data marts, each getting data from the main one but also having some other incremental data. The that's country specific And as the, um, capabilities, um grew particularly around things like R a three nodes for independent compute and storage. Um, automated concurrency scaling. So as people are working away, we can scale up the clusters and then especially data sharing. We've been able to simplify dramatically to produce the pattern you see here where we have another consumer now of the global data warehouse for all these self service loads. And we got rid of an enormous fraction of all our ETL and, uh, jobs in order to simplify. And all that data sharing, uh, was taking is taking care of all that propagation for us. We also have, uh, row and column level security features that were mentioned have been very valuable for us in making this work and then on filling out the story in terms of how we get the data in. As you may remember, in the in a prior diagram, we had S3 with talent. But now we've also been able to take advantage of transient EMR to spin up clusters. We still get a lot of, uh, Batch data overnight. Then we can very efficiently process it all in the wee hours of the morning to have our, you know, daily sales go ready and MSK managed service. Kafka is now our new venting platform. We're able to use Glue and Lambda in order to feed our primary producer and have very low latency propagation of that data for availability. So this is a great example of where Redshift, which is more the right hand side, is now getting the benefits of a lot of upstream AWS integration for us. So high performance cost effective, easily managed. It's been working well for us to give you some summary statistics on how this pattern is coming together at McDonald's. Let's see here it's very performant. We're have over 12 petabytes of actively used Long live data that we process. We have 50,000 users serving all parts of the company through data products as well as self service analysts. And then we have over 260,000 queries a day, coming in from all the different B I dashboards and data products and users. So it's a very it's really met our needs that you we saw, um, from a few slides ago that were the big problem. So it At the same time, we've done a lot of data quality work, not really the topic of this presentation, but we've been able to bring all that together with the performing data systems, high quality data to drive business value. And it's a couple of applications I'll mention here next. So one of the things we're doing to drive our business forward is we're big on marketing. We've had national campaigns for marketing you've known for a long time. But now we're also getting big into, uh, martech, where we look at customer specific information and a great example Here, uh, abandon cart. So we have Redshift with all sorts of historical data on purchase. We have new information coming in from Redshift on re very recent purchases. And then we also have real time data from the mobile app coming in through manage service Kafka. And then we combine them to now just launched recently our first abandoned CART campaign. So if you're using the mobile app and based on what's in your cart, you haven't we think you're very likely to purchase that, but haven't touched it in a little while we can send you a push in the app and it's been it's been fantastic. Another example is in supply chain. We're long known for incredibly, uh, great availability of our core products, like the fries and the Big mac. When we do promotions, it can be really hard to tune. We're not always sure how much of it people are gonna want. We try to have enough product for the promotion window, but not too much more so. Capacity planning, and now real time data can come streaming in. We combine that with historicals and marketing calendars, and we can ever tune tighter that right amount of availability during promotions. So these are just a couple of examples of things we're doing right now. And and what does the future look like? So, from an Amazon Redshift point of view, we see, uh, manage service ka, uh, Kafka. The whole data streaming capabilities as they grow within the ecosystem and tie into our data systems, really driving our event platform across the company. Right. Support for data sharing. This allows us to drive down the latency of across the data share cluster pattern you saw earlier redshift ML allowing us to up our game ever more on advanced analytics, being more efficient and more effective, and then multi a Z Redshift, which will for us as you've seen some of these applications. Take that marketing. If our cluster isn't available, we lose now transactional kinds of Integra integration. So multi a Z Redshift is important for us for that resiliency at scale. As we build on top of our data platform how we use the data, you know, we have all this clean, great data in our restaurants and how things are going. We're looking to use generative AI to access that structured data to bring it to life in a more effective way than we've ever been able to before for our restaurants. And then we've talked a lot about sort of our business facing data, our supply chain, our marketing. But we also have some other back end data systems, like some people data and some finance data. And we're now working on bringing that unified layer at SQL together for the company. So analysts can just sit down, write a query that accesses any day across any data across the company. It's that standardisation meets these data sharing features to really just drive tremendous gains in data accessibility at McDonald's. So that's sort of the last, uh, slide. I wanna take a moment to thank you know, the Redshift team for being a great partner. I also thank my teams who've done so much great work on driving this over time and then incorporating these great new features. And it's, uh, great to share my story with all of you. McDonald's. Yeah, an insightful presentation. Um, and it's it's nice to see how you're leveraging these innovations to kind of improve your data platform even more so just to wrap up, we have a lot of redshift resources. We have, uh, you can go to our feature page, explore our new features. There are a lot of demos use cases, customer stories that you can try it out. And if you want to get in touch with experts, there are links here, so the slide deck will be there. So feel free to refer to to refer to the resources that are mentioned in the deck so we can open up

Select your cookie preferences

AWS re:Invent 2023 - Unlocking Data Insights: Amazon Redshift's Latest Innovations for Modern Cloud Data Warehousing

Amazon Web Services

Up Next

Amazon Aurora MySQL Zero-ETL Integration with Amazon Redshift: Public Preview Demo and Setup Guide

AWS Clean Rooms ML and Differential Privacy: Revolutionizing Secure Data Collaboration

Enhancing Security Operations with Amazon OpenSearch Service: Introducing Security Analytics for Efficient Threat Detection and Investigation

AWS Summit Los Angeles 2024: Unleashing Generative AI's Potential - Insights from Matt Wood and Industry Leaders

AWS re:Invent 2023: Latest Innovations in Cloud Governance and Compliance Services