How Clearly accurately predicts fraudulent orders using Amazon Fraud Detector

This post was cowritten by Ziv Pollak, Machine Learning Team Lead, and Sarvi Loloei, Machine Learning Engineer at Clearly. The content and opinions in this post are those of the third-party authors and AWS is not responsible for the content or accuracy of this post.

A pioneer in online shopping, Clearly launched their first site in 2000. Since then, we’ve grown to become one of the biggest online eyewear retailers in the world, providing customers across Canada, the US, Australia, and New Zealand with glasses, sunglasses, contact lenses, and other eye health products. Through its mission to eliminate poor vision, Clearly strives to make eyewear affordable and accessible for everyone. Creating an optimized fraud detection platform is a key part of this wider vision.

Identifying online fraud is one of the biggest challenges every online retail organization has—hundreds of thousands of dollars are lost due to fraud every year. Product costs, shipping costs, and labor costs for handling fraudulent orders further increase the impact of fraud. Easy and fast fraud evaluation is also critical for maintaining high customer satisfaction rates. Transactions shouldn’t be delayed due to lengthy fraud investigation cycles.

In this post, we share how Clearly built an automated and orchestrated forecasting pipeline using AWS Step Functions, and used Amazon Fraud Detector to train a machine learning (ML) model that can identify online fraudulent transactions and bring them to the attention of the billing operations team. This solution also collects metrics and logs, provides auditing, and is invoked automatically.

With AWS services, Clearly deployed a serverless, well-architected solution in just a few weeks.

The challenge: Predicting fraud quickly and accurately

Clearly’s existing solution was based on flagging transactions using hard-coded rules that weren’t updated frequently enough to capture new fraud patterns. Once flagged, the transaction was manually reviewed by a member of the billing operations team.

This existing process had major drawbacks:

• Inflexible and inaccurate – The hard-coded rules to identify fraud transactions were difficult to update, meaning the team couldn’t respond quickly to emerging fraud trends. The rules were unable to accurately identify many suspicious transactions.
• Operationally intensive – The process couldn’t scale to high sales volume events (like Black Friday), requiring the team to implement workarounds or accept higher fraud rates. Moreover, the high level of human involvement added significant cost to the product delivery process.
• Delayed orders – The order fulfillment timeline was delayed by manual fraud reviews, leading to unhappy customers.

Although our existing fraud identification process was a good starting point, it was neither accurate enough nor fast enough to meet the order fulfillment efficiencies that Clearly desired.

Another major challenge we faced was the lack of a tenured ML team—all members had been with the company less than a year when the project kicked off.

Overview of solution: Amazon Fraud Detector

Amazon Fraud Detector is a fully managed service that uses ML to deliver highly accurate fraud detection and requires no ML expertise. All we had to do was upload our data and follow a few straightforward steps. Amazon Fraud Detector automatically examined the data, identified meaningful patterns, and produced a fraud identification model capable of making predictions on new transactions.

The following diagram illustrates our pipeline:

To operationalize the flow, we applied the following workflow:

1. Amazon EventBridge calls the orchestration pipeline hourly to review all pending transactions.
2. Step Functions helps manage the orchestration pipeline.
3. An AWS Lambda function calls Amazon Athena APIs to retrieve and prepare the training data, stored on Amazon Simple Storage Service (Amazon S3).
4. An orchestrated pipeline of Lambda functions trains an Amazon Fraud Detector model and saves the model performance metrics to an S3 bucket.
5. Amazon Simple Notification Service (Amazon SNS) notifies users when a problem occurs during the fraud detection process or when the process completes successfully.
6. Business analysts build dashboards on Amazon QuickSight, which queries the fraud data from Amazon S3 using Athena, as we describe later in this post.

We chose to use Amazon Fraud Detector for a few reasons:

• The service taps into years of expertise that Amazon has fighting fraud. This gave us a lot of confidence in the service’s capabilities.
• The ease of use and implementation allowed us to quickly confirm we have the dataset we need to produce accurate results.
• Because the Clearly ML team was less than 1 year old, a fully managed service allowed us to deliver this project without needing deep technical ML skills and knowledge.

Results

Writing the prediction results into our existing data lake allows us to use QuickSight to build metrics and dashboards for senior leadership. This enables them to understand and use these results when making decisions on the next steps to meet our monthly marketing targets.

We were able to present the forecast results on two levels, starting with overall business performance and then going deeper into needed performance per each line of business (contacts and glasses).

Our dashboard includes the following information:

• Fraud per day per different lines of business
  
• Revenue loss due to fraud transactions
  
• Location of fraud transactions (identifying fraud hot spots)
  
• Fraud transactions impact by different coupon codes, which allows us to monitor for problematic coupon codes and take further actions to reduce the risk
  
• Fraud per hour, which allows us to plan and manage the billing operation team and make sure we have resources available to handle transaction volume when needed

Conclusions

Effective and accurate prediction of customer fraud is one of the biggest challenges in ML for retail today, and having a good understanding of our customers and their behavior is vital to Clearly’s success. Amazon Fraud Detector provided a fully managed ML solution to easily create an accurate and reliable fraud prediction system with minimal overhead. Amazon Fraud Detector predictions have a high degree of accuracy and are simple to generate.

“With leading ecommerce tools like Virtual Try On, combined with our unparalleled customer service, we strive to help everyone see clearly in an affordable and effortless manner—which means constantly looking for ways to innovate, improve, and streamline processes,” said Dr. Ziv Pollak, Machine Learning Team Leader. “Online fraud detection is one of the biggest challenges in machine learning in retail today. In just a few weeks, Amazon Fraud Detector helped us accurately and reliably identify fraud with a very high level of accuracy, and save thousands of dollars.”

About the Author

Dr. Ziv Pollak is an experienced technical leader who transforms the way organizations use machine learning to increase revenue, reduce costs, improve customer service, and ensure business success. He is currently leading the Machine Learning team at Clearly.

Sarvi Loloei is an Associate Machine Learning Engineer at Clearly. Using AWS tools, she evaluates model effectiveness to drive business growth, increase revenue, and optimize productivity.