AWS for Industries
Smart Store Analytics: Creating Profitable Retail Insights
You already have basic footfall data that shows the number of people in your store at any given hour, and you know your sales data per hour, so you can produce a rough “conversion” rate for your customers. But what if you could get more granular data about your customers’ behavior? What if you could tell the dwell time per category, could map the physical journey of each customer through your store, and even put them into broad demographic categories, including those who leave the store without purchasing anything. What’s more, we are talking about every customer who walks through your door, not just those who have joined your loyalty program and dutifully dig out their membership card at the checkout.
Well guess what? Now you can get all these insights – and more.
In this article I will outline how you can use AWS for the Edge technology – and specifically the AWS Panorama Appliance – to process your existing CCTV footage to produce valuable in-store analytics (and even real-time alerts) for your physical retail store network. For years ecommerce applications have provided a rich set of analytics that have enabled retailers to refine and tune their customers’ online experience, remove friction, and increase conversion. Every search term, click, and hover-over has been recorded and analyzed to uncover patterns of behavior and deliver insights about the customer and their mission. Ultimately, this leads to deeper online engagement that stimulates even greater purchases.
Now, using computer vision and other artificial intelligence technologies, you can expect to get similar data from your physical stores, enabling you to level-up your customers’ shopping experience. Whether you choose to call this smart-store technology or something else, it represents possibly the biggest shift in in-store technology since the cash register over 100 years ago.
In the past, one of the biggest barriers to using AI and computer vision technologies was that it was hard to implement. You needed a lot of compute power, specially trained algorithms, and some very smart people to put it all together for you. But this problem has been solved over the last decade with the arrival of cloud computing, which came out of nowhere to become the de facto computing platform for new companies and a big part of the IT operations for most major enterprises and governments.
The power and elastic nature of cloud computing has allowed AI and computer vision to make huge leaps forward. You can rent super-powerful computers for just a few hours to train your computer vision model, or even download a pre-trained model and no longer spend a fortune on hardware. Today, computer vision is now a very well-understood technology and the latest models are getting more and more efficient. It’s now a simple process to identify a human being in a picture – and you can do it on a mobile phone or embed it into the camera that captures the image. You can also stream all your camera footage into the cloud and process it centrally. Unfortunately, most store networks lack sufficient bandwidth to do that, and there is a cost to transferring all that data.
This is where AWS for the Edge comes in. Using the AWS Panorama Appliance, you can process multiple video streams in-store, and significantly reduce the amount of data being sent for analysis from gigabytes per second to just kilobytes. You can now convert multiple CCTV images into simple text and leave all personally identifiable information (PII) at the point of capture. This preserves your customer’s privacy and ensures only anonymous, aggregated data makes it into your central analytics portal.
Technical solution
Existing in-store CCTV cameras can stream their footage over Real-Time Streaming Protocol (RTSP) that can be read by the AWS Panorama Appliance. A previously trained computer vision AI model can be used by the AWS Panorama Appliance to identify humans in each frame using a bounding box, and then analyze to determine the exact position of the human skeleton within that box. This is typically known as “human pose detection,” and there are many open-source models available to perform this task.
In addition to detecting key points of interest, such as ankles, knees, hips, and shoulders, you can apply logic within the AWS Panorama Appliance to identify the center point of each person. To ensure privacy, each video frame is discarded after processing. The only data that leaves the AWS Panorama Appliance are text-based, anonymous coordinates and aggregated demographic data, such as gender and age group.
A simple diagram is shown here to describe the end-to-end process.
In the example below, you can see a single input camera view of a convenience store with several customers browsing and shopping. This viewpoint is typical of many in-store CCTV camera placements and is perfectly adequate for in-store behavior analytics. In order to make use of the coordinate data you will have to remove the projection distortion and create a bird’s-eye view of the shop floor. More about that later.
This single camera view can be sent over RTSP to the AWS Panorama device where the AI model and logic have been deployed. Each video frame streaming from the IP camera will be used as an input to the AI model.
The raw video frames are read directly by the AWS Panorama Appliance and passed through multiple computer vision AI models. The process first detects the object (in this case a “person”) and then identifies the pose (“human pose detection”) of each of the people in the frame. In the following diagram, we’ve also added a midpoint between the ankles (shown as a green dot) that represents the center of where the person was standing in the store. In practice you would not see this output data as video, but it is useful for visualization of the data, and for further development of the use case.
Multi-object tracking can be used to trace a simplified journey of a customer through the store. The idea is that people will only vary their position slightly over the course of multiple video frames.
The coordinate data points are transformed to remove perspective distortion and provide a bird’s-eye view of the store for subsequent mapping to a planogram.
This can easily be done using the transform library in scikit-image:
Almost any amount of logic can be loaded onto the AWS Panorama Appliance. At this point, you could also decide to stream the data points to AWS Cloud storage since you’ve drastically reduced bandwidth requirements and removed all confidential data from the image.
High-level architecture view of the solution
Before finishing, you will want to store all data points securely in a data lake for subsequent analysis and dashboarding. One of the most interesting insights that can be extracted from the data is customer dwell time, which when combined with a planogram and plotted as a heatmap will reveal the areas of your store where visitors spend most of their time. This information can be used to re-organize your store around a more efficient layout, or steer traffic to less-busy parts of the store.
The data can also be monetized by using it to optimize the length of time and demographics of customers in a specific zone. It can help you place targeted advertising material or higher-margin products in strategic locations. You now have the data you need to make informed, profitable decisions in your store. For example, you could create real-time alerts to quickly deploy staff in response to emerging sales opportunities – like sending your best salesperson to assist the customer who has been looking at expensive handbags for the last 10 minutes.
What’s next?
Get started on your transformation journey today. Start experimenting with computer vision and see what insights you can uncover about how your customers behave in your stores. I guarantee it will be a game changer for your business. The technology is available today and does not require a PhD to get useful results. If you don’t have a digital development team at your disposal that’s ready to burn down your retail backlog, we can put you in touch with a partner who can deliver what you need. AWS will do all the heavy lifting so you and your partner can concentrate on delivering the vision that makes your business unique and keeps you ahead of your competitors. Reach out to your AWS account team, or contact me directly. We’d love to help.