AWS for Industries
Automating anomaly detection in ecommerce traffic patterns
Year after year, the ecommerce market continues to expand, driving further transformation of the retail industry. Retailers are striving to draw more shoppers to their ecommerce websites, which account for a growing share of their revenue, especially during peak events like Black Friday and Cyber Monday. According to Statista, in the third quarter of 2022, ecommerce’s share of total US retail sales was 14.8 percent, higher than the quarter before and equaling nearly $266 billion.
It’s not surprising that retail leaders are looking for ways to get more value from their ecommerce operations. That’s why many companies are seeking to monitor their website traffic for any sign of change in online activity that could impact revenue. A dip in traffic, for example, could indicate a drop in demand, due perhaps to less-than-optimal product pricing or other variables, such as a system outage. Either way, the top line can take a hit.
Surging ecommerce traffic, on the other hand, might sound like a good thing for retailers, but that’s not necessarily the case. Imagine that there’s a pricing mistake and online visitors start scooping up loads of merchandise at ridiculously low prices. That’s what happened recently at a major retailer that misplaced a decimal point on its website and accidentally listed a popular $100 item for just $10. Online buyers went wild and started ordering items in bulk, leaving the retailer scrambling to fix the snafu and stem further losses.
In both scenarios, the key is to stay on top of ecommerce traffic and mobilize teams quickly to solve any problems that emerge. This is where artificial intelligence and machine learning solutions from Amazon Web Services (AWS) can help. Retailers can gain a huge benefit from services like Amazon Kinesis, which helps collect, process, and analyze video and data streams in real time, and Amazon Lookout for Metrics, which automatically detects anomalies within metrics and identifies their root causes.
Dealing with traffic volatility
Retailers know that ecommerce traffic can vary significantly based on the season, month, date, and time of day. For example, many ecommerce websites experience high traffic during evening hours versus morning hours. Others experience a spike in traffic on weekends instead of weekdays. Meanwhile, traffic on holidays and other peak events might not follow any of these trends. Due to such dynamic and varying patterns, detecting minor anomalies in user traffic in near real time can be really difficult.
Many organizations with large ecommerce presences already have procedures in place to detect major anomalies in user traffic. However, these processes often rely on static alerts or manual monitoring techniques that struggle to detect smaller anomalies in near real time, making it hard for teams to intervene and address issues quickly.
Retailers need a smart solution that can detect the smallest deviations in user traffic based on historical data patterns. However, programming these trends based on static rules can be incredibly time intensive and less than effective after it’s deployed.
Let’s take a closer look at how the AWS anomaly detection solution can help retailers automate and detect minor (and major) anomalies while still accounting for expected traffic variations.
Figure 1. Architecture of an anomaly detection solution for ecommerce traffic
Working with the AWS anomaly detection solution
Our solution automates data collection and anomaly detection and provides a graphical user interface to interact with data and filter anomalies based on severity. Here’s how it works.
- Customers use an ecommerce application for online shopping.
The process starts when customers search for and view a product on an ecommerce website using either a mobile or desktop application. After adding items to an online cart, they complete the purchase on the checkout page. The traffic on these pages is broken down into chunks of data based on time intervals. These serve as the data points that you can use to understand traffic patterns.
- The data is ingested, transformed, and stored.
Ecommerce applications generate data in multiple formats and in different amounts. To make sense of it, you need to feed the data into a streaming platform that continuously ingests the data. The AWS solution uses Amazon Kinesis Data Streams (which helps you capture, process, and store data streams at any scale) to capture the user traffic and record any interactions with the ecommerce application.
Typically, you will need to modify or “transform” the collected data and store it in a form suitable for rapid analysis and machine learning. AWS services such as Amazon Kinesis Data Firehose (which captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services) with AWS Lambda (a serverless, event-driven compute service that lets you run code without thinking about servers or clusters) can help you transform the data and get it prepared for analysis. The data is efficiently stored in the cloud using Amazon Simple Storage Service (Amazon S3), an object storage service built to retrieve any amount of data from anywhere.
- Detect traffic anomalies and notify your teams.
The data is now ready to be analyzed in near real time to identify anomalies, which is where Amazon Lookout for Metrics comes into play. You start by creating a detector in Amazon Lookout for Metrics, which automatically pulls in data from the Amazon S3 data repository. After the detector is activated, Amazon Lookout for Metrics will begin monitoring the data and flagging any anomalies in near real time. To help decrease false positives, you can adjust the sensitivity of the detection system on a scale of 0 to 100. Using machine learning techniques, Amazon Lookout for Metrics can look at the feedback from traffic patterns and constantly improve detection results over time.
Naturally, you will want to notify team members about any anomalies so that they can find out what’s happening on the website and quickly take corrective action if needed. The AWS solution integrates seamlessly with Amazon Simple Notification Service (Amazon SNS) to automatically send out alerts and notifications through SMS text messaging, mobile push, and emails.
Conclusion
With the AWS anomaly detection solution, retailers have a powerful tool for monitoring ecommerce traffic and rapidly identifying traffic pattern anomalies that could impact revenue. It represents a significant advancement over traditional static alerts and manual monitoring techniques. For retailers looking to increase online sales and avoid unnecessary losses, the AWS solution could be an effective way to achieve that goal without investing in a costly and time-consuming homegrown solution.