How to dynamically adapt your response to changing threat levels using AWS WAF

Customers use AWS WAF to protect their web applications and APIs. They typically use a mixture of managed rules and their own custom rules, and then tune them in order to prevent as much undesired traffic as possible from reaching their applications.

This implementation and tuning exercise typically produces a web access control list (web ACL) that performs well most of the time. But things rarely stay the same for long. What if you receive a huge influx of traffic in response to a sales event or promotion and want to filter out bots and automated browsers? Or you unexpectedly start to receive a large number of potentially malicious requests and want to start blocking them?

In these situations, you might want to switch to a more restrictive web ACL. In response to an increase in bot activity, you might wish to require users to complete a CAPTCHA puzzle to access various parts of the application. Or, in response to an increase in the number of potentially malicious requests, you might wish to change some of your rule actions to block more traffic.

Accomplishing this with AWS WAF today requires some effort to implement. To update your web ACL, you need to either make the changes manually in the console or write code to make changes using the AWS WAF API. You also need to keep a record of the changes that you made so that you can revert to your previous web ACL once the threat has subsided.

We’ve heard from customers that they’d like an easier way to do this, so in this post, we introduce a solution that makes it easy to automatically switch between multiple, predefined versions of your web ACL, in response to the threat conditions your application is experiencing.

Let’s illustrate this with an example. Imagine you operate an ecommerce website, which you have secured with an AWS WAF web ACL. Your main goal is maximizing availability for users while filtering out obviously malicious traffic.

You define a web ACL in which you have a mixture of managed rules, as well as a baseline rate limiting rule. In order to maximize availability, you have most of the managed rules in COUNT mode and a rate-based rule with a fairly high threshold to help filter out possible distributed denial of service (DDoS) attacks. We’ll refer to this as the “baseline” version of the web ACL.

You then define a second version of your web ACL, which you plan to use whenever a new product launch occurs, or when you expect higher-than-usual sales, like Black Friday. This version of your web ACL has some bot control features enabled. It will block automated browsers and require users to complete a CAPTCHA in order to access the checkout page. We’ll refer to this as the “high volume” version of the web ACL.

Finally, you define a third version of your web ACL, which you plan to use if you start to see larger-than-usual numbers of requests matching some of the AWS managed rules. This version of your web ACL switches most of the rules into BLOCK mode and introduces a lower rate limit, preventing as much unwanted traffic as possible from reaching your application. We’ll refer to this as the “under attack” version of the web ACL.

The solution we have produced will allow you to switch between these three versions of your web ACL simply by updating a parameter in Parameter Store, a capability of AWS Systems Manager. This makes it easy to automate, for example, through Amazon CloudWatch alarms or Amazon EventBridge rules.

Solution details

The solution works as follows:

Figure 1: Architecture diagram of the solution overview

1. You create multiple versions of your AWS WAF web ACL, then export them in JSON format and store them in an Amazon Simple Storage Service (Amazon S3) bucket.
2. Either a user or an automated workflow updates a parameter in Parameter Store to specify the desired version of the web ACL to be deployed.
3. Updating the parameter triggers an Amazon EventBridge rule, which invokes an AWS Step Functions workflow.
4. The Step Functions workflow reads parameters from Parameter Store to determine the current version of the web ACL, the desired version of the web ACL, the S3 bucket where the configuration is stored, and the web ACL scope (“CLOUDFRONT” or “REGIONAL”).
5. If the desired version and current version are different, the Step Functions workflow reads the web ACL of the desired version from the S3 bucket.
6. Then, the Step Functions workflow invokes AWS Lambda to update the web ACL in AWS WAF.
7. The Parameter Store parameters are then updated to reflect the change.

You deploy this solution and place the three versions of your web ACL into an S3 bucket.

You can now use the solution to automate switching between the three versions of your web ACL simply by updating a parameter in Parameter Store. You can do this using automation by setting thresholds on metrics for the number of sales to switch to the “high volume” version of the web ACL, or on metrics for the number of requests matching a particular rule to switch to the “under attack” version of the web ACL.

Step-by-step instructions to deploy and test the solution

1. To test this in your own account, first deploy the sample following the instructions in the README.
2. You will need an application to use for testing. Creating a sample application is outside the scope of this walkthrough. You can find a sample static web application at Build a Serverless Web Application. Do not test this with a production application.
3. Create a new AWS WAF web ACL and associate it with your application. Add the Amazon IP reputation list rule, from the list of AWS Managed Rules.
4. Open your web ACL in the AWS WAF console, and choose Download web ACL as JSON.
5. Create three copies of the downloaded JSON file, named <webaclname>-1.json, <webaclname>-2.json, and <webaclname>-3.json. This will allow you to define three different versions of your web ACL, corresponding to three different threat levels. For this example, version 1 is for when the threat level is low, version 2 is for when the threat level is medium, and version 3 will is for when the threat level is high.
6. Edit the rules section of each of your JSON files as follows:

For version 1:

• • Override the rule group action for the Amazon IP Reputation list to COUNT. This will not block any requests, but requests that match the rule will be logged for analysis.
  • Add a rate-based rule that evaluates requests that have matched the Amazon IP Reputation list by looking for the relevant label. If a threshold of 1,000 is breached, issue a challenge action to the viewer.
  • Add a rate-based rule that evaluates requests based on the source IP. If a threshold of 10,000 is breached, issue a challenge to the viewer.

"Rules": [
    {
            "Name": "AWS-AWSManagedRulesAmazonIpReputationList",
      "Priority": 0,
      "Statement": {
        "ManagedRuleGroupStatement": {
          "VendorName": "AWS",
          "Name": "AWSManagedRulesAmazonIpReputationList",
        }
      },
      "OverrideAction": {
        "Count": {}
      },
    },
    {
      "Name": "IPReputationRateChallenge",
      "Priority": 1,
      "Statement": {
        "RateBasedStatement": {
          "Limit": 1000,
          "AggregateKeyType": "CUSTOM_KEYS",
          "CustomKeys": [
            {
              "LabelNamespace": {
                "Namespace": "awswaf:managed:aws:amazon-ip-list:"
              }
            }
          ]
        }
      },
      "Action": {
        "Challenge": {}
      },
    },
    {
      "Name": "IPRateChallenge",
      "Priority": 2,
      "Statement": {
        "RateBasedStatement": {
          "Limit": 10000,
          "AggregateKeyType": "IP"
        }
      },
      "Action": {
        "Challenge": {}
      },
    }
  ]

For version 2:

• • Override the rule group action for the Amazon IP Reputation list to COUNT. This will not block any requests, but requests that match the rule will be logged for analysis.
  • Add a rate-based rule that evaluates requests that have matched the Amazon IP Reputation list by looking for the relevant label. If a threshold of 1,000 is breached, block the request.
  • Add a rate-based rule that evaluates requests based on the source IP. If a threshold of 5,000 is breached, block the request.

"Rules": [
    {
            "Name": "AWS-AWSManagedRulesAmazonIpReputationList",
      "Priority": 0,
      "Statement": {
        "ManagedRuleGroupStatement": {
          "VendorName": "AWS",
          "Name": "AWSManagedRulesAmazonIpReputationList",
        }
      },
      "OverrideAction": {
        "Count": {}
      },
    },

    {
      "Name": "IPReputationRateLimit",
      "Priority": 1,
      "Statement": {
        "RateBasedStatement": {
          "Limit": 1000,
          "AggregateKeyType": "CUSTOM_KEYS",
          "CustomKeys": [
            {
              "LabelNamespace": {
                "Namespace": "awswaf:managed:aws:amazon-ip-list:"
              }
            }
          ]
        }
      },
      "Action": {
        "Block": {}
      },
      "VisibilityConfig": {
        "SampledRequestsEnabled": true,
        "CloudWatchMetricsEnabled": true,
        "MetricName": "RateIPReputation"
      }
    },
    {
      "Name": "IPRateLimit",
      "Priority": 2,
      "Statement": {
        "RateBasedStatement": {
          "Limit": 5000,
          "AggregateKeyType": "IP"
        }
      },
      "Action": {
        "Block": {}
      }
      }
    }
  ]

For version 3:

• • Override the rule group action for the Amazon IP Reputation list to BLOCK.
  • Add a rate based rule which evaluates requests based on the source IP. If a threshold of 1000 is breached, block the request.

"Rules": [
    {
            "Name": "AWS-AWSManagedRulesAmazonIpReputationList",
      "Priority": 0,
      "Statement": {
        "ManagedRuleGroupStatement": {
          "VendorName": "AWS",
          "Name": "AWSManagedRulesAmazonIpReputationList",
        }
      },
      "OverrideAction": {
        "Block": {}
      },
    },


    {
      "Name": "IPRateLimit",
      "Priority": 2,
      "Statement": {
        "RateBasedStatement": {
          "Limit": 1000,
          "AggregateKeyType": "IP"
        }
      },
      "Action": {
        "Block": {}
      },
    }
  ]

7. Upload the modified JSON files to the S3 bucket you created when you deployed the sample.
8. Create three parameters in Parameter Store:

• • /wafthreat/desired/[webaclname]
  • /wafthreat/current/[webaclname]
  • /wafthreat/config/[webaclname]

The config parameter value should be a JSON string. Edit the value to reflect the name of the S3 bucket you stored your JSON files in and whether your web ACL is attached to a CloudFront distribution or a Regional resource.

{
"S3Bucket":"DOC-EXAMPLE-BUCKET",
"Scope":"REGIONAL|CLOUDFRONT"
}

The current and desired parameter values should be set to 2.

9. Modify the value of the /wafthreat/desired/<webacl-name> parameter to either 3 or 1. After a minute or so, you should see that the AWS WAF web ACL has been updated with the rules you defined earlier.
10. Because the update is invoked by changing the value of the desired parameter, you can use automation to adjust the threat level. For example, you could evaluate the number of requests per second received by your application and use this metric to determine the desired threat level.

Cleanup

• Delete the CloudFormation stack.
• Remove all parameters named /wafthreat/* from Parameter Store.
• Remove the S3 bucket you created along with its contents.

Conclusion

In this post, we discussed how it can be useful to dynamically make changes to your AWS WAF web ACLs according to the threat conditions your application is experiencing. We mentioned some of the challenges customers face in doing this today and introduced a solution that makes it easy to switch between predefined versions of your web ACL by updating a parameter in Parameter Store. Finally, we stepped through an example of how you can deploy and test the solution in your own account.

Deploy the solution, get in touch with us, and share your feedback on the solution page on GitHub.

Read more about how OLX implemented a similar solution to increase the availability of their ecommerce website.

About the authors