How to Configure Rate-Based Blacklisting with AWS WAF and AWS Lambda
One security challenge you may have faced is how to prevent your web servers from being flooded by unwanted requests, or scanning tools such as bots and crawlers that don’t respect the crawl-delay directive value. The main objective of this kind of distributed denial of service (DDoS) attack, commonly called an HTTP flood, is to overburden system resources and make them unavailable to your real users or customers (as shown in the following illustration). In this blog post, I will show you how to provision a solution that automatically detects unwanted traffic based on request rate, and then updates configurations of AWS WAF (a web application firewall that protects any application deployed on the Amazon CloudFront content delivery service) to block subsequent requests from those users.
As you will see throughout this post, this process is executed by an AWS Lambda function that processes CloudFront access logs in order to identify bad requesters. This function exposes execution metrics in Amazon CloudWatch so that you can monitor how many request entries were processed and the number of origins blocked. The solution also supports manually adding IP ranges that you want to block prohibitively, such as well-known bot networks.
The previous illustration shows an infrastructure trying to respond to all requests, an approach that exhausts the web server’s resources. The following illustration shows an infrastructure that uses the solution proposed in this blog post, which blocks requests originating from blacklisted sources.
One way to prevent overburdening your resources is to deploy a rate-based blacklisting solution. This allows you to set a threshold of how many requests your web application can serve. If a bot, crawler, or attacker exceeds the threshold, you can use AWS WAF to block their requests automatically. All the AWS services used for this solution are highlighted in the gray area in the following diagram. CloudFront is configured to deliver both the static and dynamic content of a website that uses Amazon S3, Amazon EC2, and Amazon RDS. Whenever CloudFront receives a request for your web application, AWS WAF inspects the request and instructs CloudFront to either block or allow the request based on the source IP address. The following diagram shows the architecture and flow of the solution.
- As CloudFront receives requests on behalf of your web application, it sends access logs to an S3 bucket that contain detailed information about the requests.
- For every new access log stored in the S3 bucket, a Lambda function is triggered.
- The Lambda function analyzes which IP addresses have made more requests than the defined threshold and adds those IP addresses to an AWS WAF block list. AWS WAF blocks those IP addresses for a period of time that you define during the provisioning of the solution. After this blocking period has expired, AWS WAF allows those IP addresses to access your application again, but it continues to monitor the behavior of the traffic from those IP addresses.
- The Lambda function publishes execution metrics in CloudWatch, such as the number of requests analyzed and IP addresses blocked.
This is how the AWS services shown in the preceding diagram are used to make this solution work:
- AWS WAF – A service that gives you control over which traffic to allow or block to your web application by defining customizable web security rules. This solution uses three rules:
- Auto Block – This rule is used to add IP addresses identified as unwanted requesters—those that don’t respect the request-per-minute limit. In our solution, after creating this rule, new requests from those IP addresses are blocked until Lambda removes them from the block list after the specified expiration period (by default we use 4 hours).
- Manual Block – This rule is used to add IP addresses manually to the auto block list. The IP addresses are permanently blocked—they can only access the web application if you remove them from the block list.
- Auto Count – This is a quarantine rule: the requests are not blocked, but you track in near real time the number of requests from previously blocked IP addresses. This exists only to give you visibility into an IP address’s behavior after being removed from the Auto Block rule.
- Lambda – A service that lets you run code without provisioning or managing servers. Just upload your code and Lambda takes care of everything required to run and scale your code. You can set up your code to trigger automatically based on activity in other AWS services. In this solution, I’ve added a trigger in S3 to execute the Lambda function every time a new access log file is uploaded. This function is responsible for processing the log data to identify offending IP addresses and block them in AWS WAF.
- CloudFront – A content delivery web service that integrates with other AWS services to give developers and businesses an easy way to distribute content to end users with low latency, high data-transfer speeds, and no minimum usage commitments. CloudFront usually delivers access logs within an hour. However, some log entries can be delayed. In this solution, CloudFront distributes both dynamic and static content of the web application.
- S3 – A simple storage service that offers software developers a highly scalable, reliable, and low-latency data storage infrastructure at low costs. In this solution, the access log files are saved in an S3 bucket.
- CloudWatch – A monitoring service for AWS cloud resources and applications that run on AWS. In this solution, I use CloudWatch to track metrics about the number of access entries processed by the Lambda function and the number of IP addresses that have been blocked.
- AWS CloudFormation – A service that enables you to create and manage AWS infrastructure deployments predictably and repeatedly. With CloudFormation, you declare all of your resources and dependencies in a template file. In this solution, I’ve created a template that helps you provision the solution stack (all of the solution’s components) without worrying about creating and configuring the underlying AWS infrastructure.
This solution allows you to define the S3 bucket, the threshold of requests per minute, and the length of time to keep IP addresses in the block list.
Deployment—Using the AWS Management Console
This solution assumes that you already have a CloudFront distribution used to deliver content for your web application. If you do not yet have a CloudFront distribution, follow the instructions on Creating or Updating a Web Distribution Using the CloudFront Console. This solution also uses CloudFormation to simplify the provisioning process. See the CloudFormation User Guide for more information about how the service works.
Step 1: Launch the solution using a CloudFormation template
- Go to CloudFormation console.
- Change the region as per your requirements. When you use CloudFormation, all resources are provisioned in a region where you are creating the stack. Because this solution uses Lambda, see the AWS Global Infrastructure Region Table to check which AWS regions are available for Lambda.
- Click the Create New Stack button.
- On the Select Template page, upload the waf_template.json found in this GitHub repository.
- On the Specify Details page (as shown in the following image):
- For Stack name, type the name of your stack.
- For Create CloudFront Access Log Bucket, select yes to create a new S3 bucket for CloudFront Access Logs, or select no if you already have an S3 bucket for CloudFront access logs.
- For CloudFront Access Log Bucket Name, type the name of the S3 bucket where CloudFront will put access logs. Leave this field empty if you selected no for Create CloudFront Access Log Bucket.
- For Request Threshold, type the maximum number of requests that can be made per minute without being blocked.
- For WAF Block Period, specify how long (in seconds) IP addresses should be blocked after passing the threshold.
- For WAF Quarantine Period, specify how long AWS WAF should monitor IP addresses after AWS WAF has stopped blocking them.
- On the Options page, click Next.
- On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box, and then click Create.
This template creates all the components necessary to run the solution: a Lambda function, an AWS WAF Web ACL (named Malicious Requesters) with all necessary rules configured, a CloudWatch custom metric, and, if you selected yes for Create CloudFront Access Log Bucket, an S3 bucket with the name you specified in the CloudFront Access Log Bucket Name parameter.
Step 2: Update CloudFront distribution settings
Update the CloudFront distribution to activate AWS WAF and logging by using the resources generated in the previous step (if you already have an S3 bucket for CloudFront access logs, skip this logging configuration step):
- Open the CloudFront console. In the top pane of the console, select the distribution that you want to update.
- In the Distribution Settings pane, click the General tab, and then click Edit.
- Update the AWS WAF Web ACL settings (as shown in the following image). This option has a drop-down list with all active AWS WAF Web ACLs. Select the Web ACLs you created during Step 1 (Malicious Requesters).
- For Logging, select On.
- For Bucket for Logs, select the bucket that you specified in Step 1.
- Save your changes.
If you already have an S3 bucket for CloudFront access logs (if you selected no for Create CloudFront Access Log Bucket), enable S3 event notification to trigger the Lambda function when a new log file is added to your CloudFront access log bucket (see more details). To do that, open the S3 console and edit the bucket properties highlighted in the following image.
Step 3: [Optional] Edit CloudFormation parameter values
If you want to change the solution parameters after creating the CloudFormation stack in Step 1 (for example, if you want to change the threshold value or how long IPs are blocked), you don’t need to create a new stack—just update the existing one following these steps:
- In the CloudFormation console, from the list of stacks, select the running stack that you want to update.
- Click Actions and then Update Stack.
- On the Select Template page, select Use the current template, and then click Next.
- On the Specify Details page, change the values of Rate-Based Blacklisting Parameters (as shown in the following image):
- For Request Threshold, type the new maximum number of requests that can be made per minute without being blocked.
- For WAF Block Period, specify the new value of how long (in seconds) the IP address should be blocked after passing the threshold.
- For WAF Quarantine Period, specify the new value of how long AWS WAF should monitor the IP address after AWS WAF has stopped blocking it.
- On the Options page, click Next.
- On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box, and then click Update.
CloudFormation will update the stack to reflect the new values of the parameters.
How to create the stack by using the AWS CLI
Alternatively, you can create the stack by using the AWS CLI. The following code shows one way to create the stack (remember to change the parameter values highlighted in red).
aws cloudformation create-stack --stack-name <STACK_NAME> --template-body file:///<PATH_TO>/waf_template.json --capabilities CAPABILITY_IAM --parameters ParameterKey=CloudFrontCreateAccessLogBucket,ParameterValue=yes ParameterKey=CloudFrontAccessLogBucket,ParameterValue=<LOG_BUCKET_NAME> ParameterKey=RequestThreshold,ParameterValue=400 ParameterKey=WAFBlockPeriod,ParameterValue=1400 ParameterKey=WAFQuarantinePeriod,ParameterValue=14400
You will find the waf_template.json in this GitHub repository.
To test the solution offered in this blog post, wait until CloudFront generates a new access log file. Alternatively, you can simulate this process by uploading this sample access log file into the S3 bucket in which you stipulated to receive log files. After completing the upload, check to see if the IP addresses were populated automatically in the AWS WAF Auto Block Set section, and if the CloudWatch metrics were updated. Remember that Lambda can take a few seconds to process the log file, and CloudWatch can take up to two minutes before displaying new metrics. The following image shows how the Auto Block Set section appears after Lambda processes the sample access log file.
To demonstrate how CloudWatch displays the metrics, see the CloudWatch dashboard, as shown in the following image.
As you followed this blog post, you provisioned a solution that automatically blocks IP addresses based on exceeding a specified request-rate threshold. You can use the Lambda script I’ve provided to change how you block unwanted requests. For example, you can analyze data generated by CloudFront (the sc-bytes of access log file field) and block those requests.
If you have any questions or comments, leave them in the “Comments” section below or on the AWS WAF forum.