Mitigating Bot Traffic Using AWS Edge Services with Instacart

Instacart partners with more than 1,800 national, regional, and local retail banners to deliver groceries from more than 100,000 stores. Customers first select products through Instacart’s app or website, and they are then connected with a personal shopper, who shops for and delivers those products.

The company achieved early delivery and low latency while reducing costs by delivering its ads using Amazon CloudFront—a content delivery network service built for high performance, security, and developer convenience. Given those positives, Instacart decided to use Amazon CloudFront for its service load balancers to strengthen its security posture and enhance performance.

Amazon CloudFront provides scalability and reliability while integrating natively with other AWS security services that Instacart uses. In addition, Instacart has benefited from Amazon CloudFront built-in protection against layer 3 and layer 4 distributed denial of service (DDoS) events. “Amazon CloudFront is a great tool to work with from an infrastructure-as-code standpoint,” says Peter Shannon, senior manager on the cloud infrastructure team at Instacart.

Using Amazon CloudFront, the company can provide a smooth customer experience, speed up resource availability, and reduce system-generated errors. “Amazon CloudFront is very stable, and the latencies are really good, whether you are in an urban or rural environment,” says Shannon.

Instacart was also facing heavy bot traffic, and it needed a robust bot-control solution to gain granular insights into its web traffic—a proactive approach to protecting its website and mobile app. “The bot score that our previous solution provided was not accurate,” says Peerakit Somsuk, software engineer on the infrastructure edge team at Instacart. “We also needed more fine-grained control on different signals that return different meanings.”

Instacart decided to use AWS WAF, which organizations use to protect web applications from common bugs. It also used AWS WAF Bot Control, which gives visibility and control over common and pervasive bot traffic.

The Instacart team integrated AWS WAF Bot Control with Application Load Balancer—which lets organizations load balance HTTP and HTTPS traffic with advanced request routing—and set up a dashboard to monitor web traffic and create alerts. That helped the team understand and categorize the types of bots—for example, verified bots, such as those from search engines, and malicious bots, which try to access various endpoints. The team built the dashboard using the event logs that were generated by AWS WAF.

Using AWS WAF Bot Control, the Instacart team can effectively protect the public endpoints of the company’s platform. The team uses various bot-control signals to challenge suspicious traffic and block dangerous traffic, significantly reducing scraping activity. “The AWS WAF team consistently provides valuable feedback and delivers new features so that we can stay ahead of emerging threats,” says Somsuk.

To maximize application security protection, Instacart uses AWS Shield, a managed DDoS protection service that safeguards applications running on AWS. In particular, the company adopted AWS Shield Advanced, which provides additional detection and mitigation against large and sophisticated DDoS events, NRT visibility into threats, and integration with AWS WAF. Using AWS Shield Advanced also gives the company access to the Shield Response Team, a dedicated team of AWS engineers who assist customers in mitigating DDoS events. “It gives us an extra layer of protection,” says Shannon.

The advantages of using AWS WAF include usage flexibility and more control over the system. “AWS WAF provides flexibility in configuration compared with our previous solution,” says Somsuk. “We have improved our signals on each request, which gives us fine-grained control.” Instacart has reduced its total traffic by about 20 percent by blocking bot traffic without affecting the user experience. The application error rate that is caused by bot traffic is down by five times, and the company has also saved about 90 percent on costs since adopting AWS WAF.

After migrating its website, Instacart migrated its mobile clients to AWS WAF. “After protecting all our legitimate clients, we know who is accessing us, and we can filter out the requests that are not coming from our clients,” says Somsuk. That has further strengthened the company’s security posture by simplifying the detection and mitigation of malicious bot traffic.

The company wants to go some steps further in mitigating bot traffic. “We want to automate rule generation, detect some kind of pattern, and get recommendations from the system—for example, what we should add to the rules,” says Somsuk. “We want to look into incorporating artificial intelligence to improve the customer experience.”

Instacart will work alongside AWS on those future steps as well. “AWS has talented technical account management and solutions architect teams, and there’s a lot of education for us,” says Somsuk. “The staff works with many customers, and we benefit greatly from its expertise.” In its journey ahead, Instacart will continue to follow the approach of AWS—in particular, in regard to customers. “Working alongside AWS, we can figure out how to solve our issues,” says Shannon. “It is customer focused and deeply addresses our business and technical needs.”