AWS Security Blog

How to Import IP Address Reputation Lists to Automatically Update AWS WAF IP Blacklists

Note from July 3, 2017: The solution in this post has been integrated into AWS WAF Security Automations, and AWS maintains up-to-date solution code in the companion GitHub repository.

 

You can use AWS WAF (a web application firewall) to help protect your web applications from exploits that originate from groups of IP addresses that are known to be operated by bad actors such as spammers, malware distributors, and botnets. The IP addresses used may change over time as these bad actors attempt to avoid detection. In this post, I will show how to synchronize AWS WAF Rules with reputation lists.

A number of organizations maintain reputation lists of IP addresses used by bad actors. Their goal is to help legitimate companies block access from specific IP addresses and protect their web applications from abuse. These downloadable, plaintext reputation lists include Spamhaus’s Don’t Route Or Peer (DROP) List and Extended Drop (EDROP) List, and Proofpoint’s Emerging Threats IP list. Similarly, the Tor project’s Tor exit node list provides a list of IP addresses currently used by Tor users to access the Internet. Tor is a web proxy that anonymizes web requests and is sometimes used by malicious users to probe or exploit websites.

Solution overview

The described solution uses AWS Lambda to synchronize AWS WAF Rules with available reputation lists. Lambda automates the parsing of text-based IP reputation lists, the extracting of IP addresses contained within those lists, and updating the AWS WAF IP Sets in order to create a blacklist that blocks those addresses. Amazon CloudWatch Events executes the Lambda function on a regular schedule. To simplify the deployment, I will show you how to deploy this solution by using AWS CloudFormation.

The following diagram illustrates the process covered in this post.

Diagram of the process covered in this post

Here is how the process works:

  1. Amazon CloudWatch Events execute the Lambda function on a scheduled basis.
  2. The function downloads third-party reputation lists and processes them.
  3. To create a blacklist, the function then updates AWS WAF IP Sets with the latest IP addresses and ranges defined in the reputation lists.
  4. AWS WAF then denies requests from the IP addresses that appear in the blacklist.

In addition to the reputation lists mentioned previously, you might want to include other lists maintained by a third party or by your organization. This post’s example Lambda function, written in Node.js JavaScript, supports the downloading of multiple lists and updating AWS WAF in a single Lambda execution. The Lambda function supports text-based lists with a single IP address or range specified per line. The function uses regular expressions to extract the IP address in either a.b.c.d dotted-decimal notation or a.b.c.d/n Classless Inter-Domain Routing (CIDR) notation, and supports an optional regular expression to match text that is present in the line before the address range. The function also consolidates ranges duplicated across multiple reputation lists and ranges that are contained within larger ranges. As an example, suppose we have two address ranges, 192.0.2.0/24 and 192.0.2.64/28. The second range is contained within the first, and therefore is not required in the combined list.

AWS WAF lets you apply a web access control list (web ACL) to a CloudFront distribution. A web ACL gives you fine-grained control over the web requests that your AWS resources respond to, and it can contain up to 10 rules. Each rule can specify multiple conditions that are logically ANDed to get a match in order to either Allow, Deny, or Count the incoming request. One of the conditions these rules can refer to is an AWS WAF IP Set. An IP Set defines up to 1,000 IP descriptors, which define the IP addresses and ranges in CIDR notation. Therefore, a single AWS WAF web ACL can block up to 10,000 IP address ranges in total.

AWS WAF supports IP addresses and ranges in the CIDR format a.b.c.d/n where n, the mask, must be on the octet boundaries of 8, 16, 24, or 32. Some of the reputation lists may include address ranges that have masks other than the ones supported by AWS WAF. For those address ranges, the Lambda function will “split” the range into multiple, smaller ranges that have one of the supported masks. For example, suppose we have a range defined 192.0.2.0/31, which specifies a range with two IP addresses. AWS WAF does not support a network mask of 31, so the function will split 192.0.2.0/31 into two ranges, each specifying a single address, 192.0.2.0/32 and 192.0.2.1/32, covering the range of 192.0.2.0/31.

The example Lambda function shards your reputation lists across multiple AWS WAF IP Sets that are made available to it, and prioritizes the list of IP addresses by size of ranges such that it blocks as many individual IP addresses as possible.

Deploy the solution using AWS CloudFormation

Now that I have explained how the Lambda function will process reputation lists, I will show how to use CloudFormation to create a stack consisting of the AWS WAF web ACL, Rules, and IP Sets, as well as the Lambda function, CloudWatch Events Rule, and supporting resources. The sample CloudFormation template defines two AWS WAF Rules and two IP Sets. You can modify the template to include more Rules and IP Sets in order to support a larger blacklist. The template also defines a CloudWatch Events Rule, which will execute the Lambda function every hour. You can alter the rate to a different value, but please respect the wishes of the reputation lists owners and do not request the lists too often.

First, I go to the CloudFormation console, click Create Stack, and specify one of the URLs for the CloudFormation templates listed at the end of this post.

Screenshot of creating a stack

Click Next and type a name for the stack.

Screenshot of providing a name and parameters for the stack

The template defines the parameter ReputationLists that, by default, includes the following JSON parameter value that defines the Spamhaus DROP, Tor exit node list, Emerging Threats IP list, and associated regular expression prefixes. You can edit the JSON parameter value to specify other lists.

[
     { "url": "https://www.spamhaus.org/drop/drop.txt" },
     { "url": "https://check.torproject.org/exit-addresses","prefix":"ExitAddress "},
     { "url": "https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt" }
]

I click Next and Next again. On the next page, I select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box in order for the stack to create the IAM role assumed by the Lambda function. I click Create to create the stack.

After the CloudFormation stack is created, I select the stack in the CloudFormation console and click the Resources tab.

Screenshot of the Resources tab

I make a note of the physical ID for the Lambda function. In my example, it is called WAFReputationLists-LambdaFunction-PM6D5GNW6EDD, but yours may have a different ID. I will use this physical ID later to find the function in the Lambda console.

Monitor using CloudWatch

I then go to the Lambda console and select the Lambda function created by CloudFormation with the name mentioned in the previous paragraph. I click the Monitoring tab, and from here, I can monitor the execution of the function on an hourly basis. Note that it may take an hour for the first scheduled event to execute the function.

Screenshot of the Monitoring tab

On the Monitoring tab, I click View logs in CloudWatch, which takes me to the CloudWatch console. Here, I can view the logs generated by the function using CloudWatch Logs, as shown in the following screenshot.

Screenshot of the CloudWatch Logs

Attach the AWS WAF web ACL to a CloudFront distribution

To use the blacklist defined in my web ACL to protect my web application, I must attach the web ACL to a CloudFront distribution used to deliver the website via the CloudFront CDN.

I open the CloudFront console and click Distributions. I click my CloudFront distribution’s ID.

Screenshot showing the CloudFront distribution's ID

I then click Edit.

Screenshot showing the Edit button

I select the web ACL to attach (I choose the web ACL I created with the stack earlier in this post), scroll to the bottom of the page, and click Yes, Edit to save changes.

Screenshot showing the selection of the web ACL

I have attached the appropriate web ACL to the CloudFront distribution, which will allow me to use the blacklist to protect my web application.

Summary

In this post, I demonstrated how you can use AWS WAF to block the IP addresses used by bad actors. I did this by creating a Lambda function that imports IP addresses and ranges from multiple third-party IP reputation lists into AWS WAF, and scheduled the import using CloudWatch Events.The function processes the addresses defined in the lists into a format suitable for AWS WAF, removing duplicates and prioritizing the list to include as many IP addresses as possible. Subsequent requests to your web application (delivered by CloudFront) from IP addresses defined in this blacklist will be denied by AWS WAF.

The CloudFormation template and Lambda function package are available in the following S3 buckets that are located in the AWS regions that currently support Lambda:

The code and other WAF samples also are available in our GitHub aws-waf-sample repository. For a solution that combines this and other AWS WAF automation solutions into a deployable CloudFormation template, see AWS WAF Security Automations.

If you have comments about this blog post, please submit them in the “Comments” section below. If you have questions about this solution or its implementation, please start a new thread on the AWS WAF forum.

– Lee