Bot management

Automated bot traffic can have negative impact on your web application, in terms of availability, increased infrastructure costs, skewed analytics and fraudulant activities such as account takeover. Bot management designates the controls used to identify traffic coming from bot, and then blocking undesired ones. The sophistication of such controls depends on how much malicious actors are incentivized financially and politically to target a web application. The more they are motivated to target a web application, the more they will use invest in detection evasion techniques, requiring more advanced mitigation capabilities. It's recommended to adopt a layered approach to managing bots, with differentiated tooling, each adapted to specific bot sophitsitcation. For more information on where to get started, see AWS Prescriptive Guidance for Implementing a bot control strategy.

Traffic generated by common and pervasive bots such as scanners and crawlers could be identified and managed on the server side by analyzing the signature of requests based on HTTP attributes such as IP, User-Agent header or TLS fingerprint. AWS WAF allows you to use the following signature based rules:

• Rules used to block HTTP Floods (L7 DDoS attacks), such as Rate limits, Managed IP Reputation rule groups (Amazon IP reputation list, Anonymous IP list, etc..), and rules automatically created by Shield Advanced's automatic application layer DDos mitigation.
• AWS WAF Bot Control managed rule group configured with common protection level to block self identifying bots, or with high confidence bot signatures. This rule group can be configured with granularity to differentiate the management of different bot categories such as Http Library or Scraping Framework.This blog gives you concrete examples a granular configuration of AWS WAF Bot Control with labels and scope-down statements.
• Managed rules provided by security vendors in the AWS Marketplace, such as Bot Protection Rules by F5 and Active Malicious Bots by ThreatSTOP.
   

When malicious actors have higher incentives for automating traffic to your website (e.g. financial motivation to scrape content to resell it, steal credit card data and resell it, etc...), they invest more efforts and money in techniques to evade detections (e.g. using IPs from residential networks, using advanced browser automation frameworks, using CAPTCHA farms, etc...). For sophisticated bots, signature based detections are less efficient, requiring you to deploy more advanced and expensive behavioral detections.

Configured rules in AWS WAF can have CAPTCHA action, in addition to block, count or rate limit. When a rule is configured with a CAPTCHA action, users are required to solve a puzzle to prove that a human being is sending the request. When a user successfully solves a CAPTCHA challenge, a token is placed on their browser to avoid challenging future requests, using a configurable immunity time. Learn about best practices for configuring CAPTCHA.

Configured rules in AWS WAF can have Challenge action, in addition to block, count or rate limit. When a rule is configured with a Challenge action, the browser is presented with a silent challenge (interstitial challenge) that requires the client session to verify that it's a browser, and not a bot. The verification runs in the background without involving the end user. This is a good option for verifying clients that you suspect of being invalid without negatively impacting the end user experience with a CAPTCHA puzzle. When a user successfully solves a silent challenge, a token is placed on their browser to avoid challenging future requests, using a configurable immunity time Learn about best practices for configuring Challenges.

Another option to acquire an AWS WAF token is by using the AWS WAF application integration SDKs. The SDKs require programming in your client applications, but they can provide a better customer experience, they're free to use, and they can be used with browsers executing JavaScript, or natively in Android or iOS mobile applications. The SDK integration is helpful in cases where Challenge or CAPTCHA rule actions are not an option, e.g Single Page Applications. AWS WAF provides two levels of client application integration :

• Intelligent threat integration SDKs – These are designed to work with Intelligent threat mitigation rules.They verify the client application and provide AWS token acquisition and management. They function similar to AWS WAF Challenge rule action.
• CAPTCHA integration JS API – These APIs verify end users with customized CAPTCHA puzzle that customers manage in their application. This is similar to the functionality provided by the AWS WAF CAPTCHA rule action, but with added control over the puzzle placement and behavior. This feature is available for JavaScript applications.

AWS WAF provides a set of managed rules focued on detecting fraudulant activities in the login or registration worklflows.Account takeover is an online illegal activity where the attacker gains unauthorized access to a user's account, by either using stolen credentials or guessing the victim's password through a series of attempts. You can monitor and control account takeover attempts by implementing the AWS WAF Fraud Control Account Takeover Prevention (ATP) managed rule. For each user session, ATP monitors the rate of login attempts, including failed attempts, to detect for password or username traversal attempts. In addition, ATP checks username and password combinations against a stolen credential database, which is updated regularly as new leaked credentials are found on the dark web.

Account creation fraud is an online illegal activity in which an attacker tries to create one or more fake accounts. Attackers use fake accounts for fraudulent activities such as abusing promotional and sign up bonuses, impersonating someone, and cyberattacks like phishing. You can monitor and control fraudulent account creation attempts by implementing the AWS WAF Fraud Control Account Creation Fraud Prevention (ACFP) managed rule. For each user session, ACFP monitors the use of compromised credentials, IP risk score, client interactivty with the page, detected automation framework or inconsisitent browser behavior, high usage of the same phone/address/email to create multiple accounts, etc..

To use the full capabilities of the different Fraud Control managed rules, you need to add a client side SDK to your application to track behaviors at session level.

Bot Control rule group, configured with Targeted Bots protection level, provides sophisticated bot detection and mitigation by creating an intelligent baseline of traffic patterns. Bot Control for Targeted Bots uses browser fingerprinting techniques and client-side JavaScript interrogation methods to help protect your application from advanced bots that mimic human traffic patterns and actively try to evade detection. AWS WAF targeted controls for bots also provide predictive ML technology to defend against distributed, proxy-based attacks. The managed AWS WAF Bot Control rule group uses automated, ML analysis of website traffic statistics to detect anomalous behavior that is indicative of distributed, coordinated bot activity.

These options include dynamic rate-limiting, challenge actions, and the ability to block based on labels and confidence scores. Learn mode about this advanced capability in this talk and this blog.

At your application level, you can use custom signals to identifiy abnormal behavior, based on what is expected by your application. For example, you might expect users to navigation your application in a certain order, or you do not likely expect a user to order certain goods from/to certain countries based on his registered address. Using such signals, you can automate your response using AWS WAF, for example by blocking or challenging using CAPTCHA requests coming from IPs with suspicious application level behavior. To get started with the concept of WAF automation based on application signals consider the examples in this AWS Solution.

Advanced automations include:

• Consuming high risk events emitted by Cognito during signin/sign up process.
• Consuming high risks events identified by Fraud Detector. Fraud Detector uses machine learning (ML) and 20 years of fraud detection expertise from Amazon Web Services (AWS) and Amazon.com to automatically identify potential fraudulent patterns performed by humans and bots in real-time. Fraud Detector allows detections of Fraud by analyzing application-level user behavior, using your own historical fraud data to train, test, and deploy custom fraud detection machine learning models tailored to your use case.

Additional protection layer can provided by security vendors on the AWS Marketplace that are specialist in advanced bot detection. Vendors include DataDome, Distill Networks, PerimeterX, Cequence, Kasada and Imperva.

Note that each security vendor has different strengths in terms of industry specific protections, capabilities and cost. In general, your application using CloudFront can integrate vendor solutions in one of two ways:

• SaaS reverse proxy based solution, sitting between CloudFront and your origin.
• Globally replicated Bot Mitigation API that can be called by Lambda@Edge for every incoming request (i.e. configured on viewer request event) to decide how to manage the request.

• AWS re:Inforce 2024 - Demos on AWS WAF bot detection rulegroups

• AWS re:Inforce 2024 - How Catch Group uses AWS WAF Bot Control on their ecommerce platform
• AWS re:Inforce 2022 - Advanced protections against bots using AWS WAF
• The Routing Loop - Protect your applications against Bot traffic
• OLX Case Study
• Use AWS WAF CAPTCHA to protect your application against common bot traffic
• Best practices for intelligent threat mitigation
• Find AWS WAF Delivery Partners
• Fraud prevention and Bot control with AWS WAF - AWS Online Tech Talks
  
• Using AWS WAF intelligent threat mitigations with cross-origin API access
• Kasada beats bots at their own game: How to identify and eliminate bot attacks