Automated bot traffic can have negative impact on your web application, in terms of availability, increased infrastructure costs, skewed analytics and fraudulant activities such as account takeover. Bot management designates the controls used to identify traffic coming from bot, and then blocking undesired ones. The sophistication of such controls depends on how much malicious actors are incentivized financially and politically to target a web application. The more they are motivated to target a web application, the more they will use invest in detection evasion techniques, requiring more advanced mitigation capabilities. It's recommended to adopt a layered approach to managing bots, with differentiated tooling, each adapted to specific bot sophitsitcation.
Signature-based detections for common and pervasive bots
Traffic generated by common and pervasive bots such as scanners and crawlers could be identified and managed on the server side by analyzing the signature of requests based on HTTP attributes such as IP, User-Agent header or TLS fingerprint. AWS WAF allows you to use the following signature based rules:
- Rules used to block HTTP Floods (L7 DDoS attacks), such as Rate limits, Managed IP Reputation rule groups (Amazon IP reputation list, Anonymous IP list, etc..), and rules automatically created by Shield Advanced's automatic application layer DDos mitigation.
- AWS WAF Bot Control managed rule group configured with common protection level to block self identifying bots, or with high confidence bot signatures. This rule group can be configured with granularity to differentiate the management of different bot categories such as Http Library or Scraping Framework.This blog gives you concrete examples a granular configuration of AWS WAF Bot Control with labels and scope-down statements.
- Managed rules provided by security vendors in the AWS Marketplace, such as Bot Protection Rules by F5 and Active Malicious Bots by ThreatSTOP.
Behavioral detections for advanced bots
When malicious actors have higher incentives for automating traffic to your website (e.g. financial motivation to scrape content to resell it, steal credit card data and resell it, etc...), they invest more efforts and money in techniques to evade detections (e.g. using IPs from residential networks, using advanced browser automation frameworks, using CAPTCHA farms, etc...). For sophisticated bots, signature based detections are less efficient, requiring you to deploy more advanced and expensive behavioral detections.
Configured rules in AWS WAF can have CAPTCHA action, in addition to block, count or rate limit. When a rule is configured with a CAPTCHA action, users are required to solve a puzzle to prove that a human being is sending the request. When a user successfully solves a CAPTCHA challenge, a token is placed on their browser to avoid challenging future requests, using a configurable immunity time. Learn about best practices for configuring CAPTCHA.
Configured rules in AWS WAF can have Challenge action, in addition to block, count or rate limit. When a rule is configured with a Challenge action, the browser is presented with a silent challenge (interstitial challenge) that requires the client session to verify that it's a browser, and not a bot. The verification runs in the background without involving the end user. This is a good option for verifying clients that you suspect of being invalid without negatively impacting the end user experience with a CAPTCHA puzzle. When a user successfully solves a silent challenge, a token is placed on their browser to avoid challenging future requests, using a configurable immunity time Learn about best practices for configuring Challenges.
AWS WAF Fraud Control
AWS WAF provides a set of managed rules focued on detecting fraudulant activities in the login or registration worklflows.Account takeover is an online illegal activity where the attacker gains unauthorized access to a user's account, by either using stolen credentials or guessing the victim's password through a series of attempts. You can monitor and control account takeover attempts by implementing the AWS WAF Fraud Control Account Takeover Prevention (ATP) managed rule. For each user session, ATP monitors the rate of login attempts, including failed attempts, to detect for password or username traversal attempts. In addition, ATP checks username and password combinations against a stolen credential database, which is updated regularly as new leaked credentials are found on the dark web.
Account creation fraud is an online illegal activity in which an attacker tries to create one or more fake accounts. Attackers use fake accounts for fraudulent activities such as abusing promotional and sign up bonuses, impersonating someone, and cyberattacks like phishing. You can monitor and control fraudulent account creation attempts by implementing the AWS WAF Fraud Control Account Creation Fraud Prevention (ACFP) managed rule. For each user session, ACFP monitors the use of compromised credentials, IP risk score, client interactivty with the page, detected automation framework or inconsisitent browser behavior, high usage of the same phone/address/email to create multiple accounts, etc..
To use the full capabilities of the different Fraud Control managed rules, you need to add a client side SDK to your application to track behaviors at session level.
AWS WAF Bot Control for targeted bots
Application level behavioral detection
At your application level, you can use custom signals to identifiy abnormal behavior, based on what is expected by your application. For example, you might expect users to navigation your application in a certain order, or you do not likely expect a user to order certain goods from/to certain countries based on his registered address. Using such signals, you can automate your response using AWS WAF, for example by blocking or challenging using CAPTCHA requests coming from IPs with suspicious application level behavior. To get started with the concept of WAF automation based on application signals consider the examples in this AWS Solution.
Advanced automations include:
- Consuming high risk events emitted by Cognito during signin/sign up process.
- Consuming high risks events identified by Fraud Detector. Fraud Detector uses machine learning (ML) and 20 years of fraud detection expertise from Amazon Web Services (AWS) and Amazon.com to automatically identify potential fraudulent patterns performed by humans and bots in real-time. Fraud Detector allows detections of Fraud by analyzing application-level user behavior, using your own historical fraud data to train, test, and deploy custom fraud detection machine learning models tailored to your use case.
3rd Party security vendors
Additional protection layer can provided by security vendors on the AWS Marketplace that are specialist in advanced bot detection. Vendors include DataDome, Distill Networks, PerimeterX, Cequence, Kasada and Imperva.
Note that each security vendor has different strengths in terms of industry specific protections, capabilities and cost. In general, your application using CloudFront can integrate vendor solutions in one of two ways:
- SaaS reverse proxy based solution, sitting between CloudFront and your origin.
- Globally replicated Bot Mitigation API that can be called by Lambda@Edge for every incoming request (i.e. configured on viewer request event) to decide how to manage the request.
- AWS re:Inforce 2022 - Advanced protections against bots using AWS WAF
- The Routing Loop - Protect your applications against Bot traffic
- OLX Case Study
- Use AWS WAF CAPTCHA to protect your application against common bot traffic
- Best practices for intelligent threat mitigation
- Find AWS WAF Delivery Partners
- Fraud prevention and Bot control with AWS WAF - AWS Online Tech Talks
- Using AWS WAF intelligent threat mitigations with cross-origin API access
- Kasada beats bots at their own game: How to identify and eliminate bot attacks