How Vistry is making restaurants more efficient using computer vision on AWS

The How We Built This blog series features conversations with C-suite executives that are building innovative restaurant technologies on AWS

The price of potatoes is up 13 percent, America’s “fry attachment rate” is above pre-pandemic levels, and restaurants must pivot efficiently to serve both on- and off-premise diners amidst a labor shortage. Restaurant operators are stretched to focus on quality food and customer interactions while cutting costs. How can they reach these objectives? Atif Kureishy, Founder and CEO of Vistry, has a vision for that.

Kureishy, a former NASA engineer and manufacturing expert, has conceptualized a way to see the restaurant as a connected operating plant. He formed Vistry to provide restaurant operators the intelligence they need to overcome these challenges using computer vision, which allows machines to identify people, places, and things in images with accuracy at or above human levels with much greater speed and efficiency. In doing so, he is forming a new industry model for how restaurants can run efficiently while focusing on their customers.

I had the pleasure of hearing Kureishy’s story firsthand as he explained how each seemingly disparate element of the restaurant environment connect—from the drive-thru line to the kitchen line to the parking lot throughput. Vistry, built on Amazon Web Services (AWS), can drive synergies throughout the whole system. It does so by providing operators with the level of insight and intelligence needed to support an innovative operating model that puts the customer at the center.

Keep reading to get insights and takeaways from our discussion in the following Q&A.

You started your career as NASA engineer. What was the aha moment that sparked your transition into restaurant technology?

I started at NASA and then worked for the Intel community and built smart cities across the world. On that journey, I got an understanding of the reality of delivering advanced capabilities into complex enterprises.

In 2020, I was a global artificial intelligence (AI) and deep learning leader, and we worked with one customer in the casual dining space at the beginning of the pandemic. We saw the pandemic causing a fundamental shift in the hospitality industry. And we thought we could bring the experiences we had in industrial concepts into restaurants. We hypothesized that the digital industrial manufacturing concepts that were making industries more successful could make restaurants more successful, too. We thought, why not bring these experiences to bear?

We launched Vistry in September 2020 and started working with customers on day one, tackling their problems and inserting advanced capabilities in an approachable, manageable, and practical way.

Restaurant challenges abound: food costs are rising, supply chains are constrained, and there’s a labor shortage. How does Vistry help restaurants tackle these?

Ultimately, we can put it in simple terms: data equals understanding. At a high level, we empower restaurants with the understanding to run operations with more efficiency and customer impact. We create new data from information that exists in the kitchen by using AI and deep learning that uses physical observation or measurements from sensors. With that real-time level of intelligence, the restaurant is armed with a better understanding of how these overarching challenges affect its unique brand and how to address these challenges.

So how is a restaurant like manufacturing plant?

You can think about a restaurant as a mini manufacturing plant because it uses raw materials and turns it into a finished product. It is often opaque to restaurant operators what happens along the way––sometimes, all they see is the end product resulting in a guest complaint. And there is so much variability in why a guest would complain. So, similar to a manufacturing plant, the restaurant—when given insight into where and why that complaint occurred—can solve the issue quickly and without unnecessary steps stemming from a lack of information.

For instance, a customer complaint might lead an operator to believe there is a food quality issue. But maybe the issue could be solved more easily through reducing missing items. We are working with one brand on building a vision system that detects missing items and integrates with their point of sale (POS) to understand when a part of the order is incomplete. That may influence guest complaints because you forgot a side dish. And that, of course, isn’t a food quality issue. Data equals understanding.

And using AWS, the more data we process and the more rigorous analytics we apply to that data in the context of an efficient business domain, the better our understanding of how to help restaurant brands win. The machine learning services we use on AWS help us constantly refine our analysis to help restaurant brands succeed.

The statement that “data equals understanding” is powerful. We frequently hear from restaurants that their data sources are fragmented, and it is challenging to get a single view of customer behavior to improve customer satisfaction. There’s so much data out there. Phil Le–Brun, an Enterprise Strategist at AWS, might have said it best: “Where is my data, is it clean, who owns it, and what does it mean?”

We provide data for the restaurant across the whole business as a mini manufacturing plant so that the restaurant can make sense of what is happening at any part of the process. We apply natural language understanding to drive-thru voice interactions and even apply vehicle make-and-model tracking to help predict demand cycles. With a data and analytics solution in place, a restaurant can answer all kinds of questions to gain intelligence.

You may want to know what your abandonment rate is when queues get really long during peak hours. We can develop a model with license plate data to understand loyalty, or use vehicle make-and-model tracking from vision data. We can also place loop sensors in the ground to combine with that vision data, and add in the POS data to get a sense of speed-of-service. This can all be combined into one data collection solution and viewed in a data visualization platform.

In other instances, you might want to understand what is going on with voice interactions when someone is ordering. You have a microphone, so you can provide natural language understanding on that data—and the same exists in the kitchen or the dining room, as dining room activity is returning. So, you can think of a palette of applications and capabilities that quickly come to bear after you get the Vistry environment and solution in place.

We often hear from restaurant customers that technology solutions are fragmented, despite many technologies intending to provide a single source of truth. One unique aspect of Vistry is the ability to bring data together—from IoT data, to computer vision, to voice technology—and synthesize that.

Fragmentation and integration become the biggest challenge over time with analytics at scale. We knew this was going to occur—that is why we had an Edge-first mindset when we built Vistry on AWS. We predicted that restaurants would seek a single source of truth from various data sources—sensors, loyalty, POS, cameras, smart equipment, and robotics—and would want to make sense of all that. Because eventually, a restaurant might want to action a human, a robot, or a notification to a guest’s mobile device, but you cannot do that across 20 systems. The idea of a unified technology solution has been our “day 1” concept, and this is one of the reasons we built on AWS.

How does AWS help support the relationships you have with your customers and the growth of your brand?

Our AWS account team has been supportive beyond our expectations, and we have taken advantage of  many startup-focused programs from AWS. We are customer obsessed, and our relationship with AWS amplifies that because we share that focus with Amazon. It has been a fun journey, and we are constantly learning from our customers and inventing the art of the possible. It’s still early days.

The AWS Activate program has been wonderful in getting us connected with the AWS SaaS Factory Program.

The admiration goes both ways – and you have been an AWS Outpost launch partner and part of our ISV Accelerate program.  

I consider our AWS relationship a true partnership. We wanted to build on a solution that is future proof, aligned with where customers are going, and scalable. Most of the large brand-name restaurant customers we have encountered in the food space are on AWS. The restaurant industry presence on AWS makes it possible for us to work with global brands and AWS in a connected way.

We are partners with two hybrid cloud services as well. Hybrid services blend what is happening in the cloud and what is happening at the edge, meaning in the restaurant. Restaurants, as they look at this tech, want to have more workloads run on premises because their networks might be unreliable and they have to respond in near real time to guests and orders coming in.

What other AWS services have Vistry relied on to scale and help restaurants?

We do a lot of the model build ourselves, so we use Amazon SageMaker—which builds, trains, and deploys ML models for any use case with fully managed infrastructure, tools, and workflows. And we build the datasets and labeling using Amazon SageMaker Ground Truth—which provides flexibility to build and manage our own data labeling workflows and workforce.

We also use AWS IoT Core to manage edge run times. We use Amazon QuickSight, which allows everyone in an organization to understand its data, for visualizations; Amazon Elastic Container Registry (Amazon ECR), which makes it simple to store, share, and deploy container software anywhere; and Amazon Elastic Kubernetes Service (Amazon EKS), to orchestrate Kubernetes clusters across thousands of sites. These apps we create end up being containerized microservices that are hosted in Amazon ECR.

What’s a real-world example of using AWS to quickly solve a restaurant problem at scale?

We have been working with an enterprise restaurant brand to improve quality at their virtual restaurant concept. They noticed a rise in guest complaints and wanted to understand why. We created a solution that pairs POS data with camera data that analyzes how team members are placing food in a bag so we can identify if an item is missing before the order is given to the customer.

We built a dataset using video and imagery, and we are training neural networks to understand real-world objects for rapid iteration.

Where do you imagine the future of restaurant automation—robotics, voice recognition, computer vision will go?

Forecasting the future is difficult. Ultimately, the customer will decide the future based on what experiences they value the most. Customers are becoming increasingly open to eating anywhere—a car, a sidewalk, or inside a restaurant. But sometimes customers really want that full restaurant experience and atmosphere. We have been thinking about building a flexible solution to glean intelligence in all of those different scenarios. While, I cannot predict the future, one thing is for sure, we will continually see more technology in restaurants.

You might say that AI is where on-demand food delivery was five years ago. Early adopters saw a first-mover advantage and benefitted from strong digital ordering channels once the pandemic hit.

These capabilities are very real—sometimes AI is met with skepticism in a busy restaurant, like: “that is cool and all, but I have to deal with real problems today.” My message to the restaurant leaders of the future is that this is real, this is happening, and the folks embracing this technology now will be leaders tomorrow.

Conclusion

As restaurants become more technically complex, operators will need to capture and analyze multiple data sources to streamline and improve processes. Treat your restaurant like a manufacturing plant with happy customers as the output. To learn more about how AWS can help you improve restaurant operations and enhance guest satisfaction, visit aws.amazon.com/travel-and-hospitality/restaurants/.