AWS Security Blog

How to Automatically Update Your Security Groups for Amazon CloudFront and AWS WAF by Using AWS Lambda

Updated: December 9, 2016

Amazon CloudFront can help you increase the performance of your web applications and significantly lower the latency of delivering content to your customers. Recently announced, AWS WAF (a web application firewall) gives you control over which traffic to allow or block by defining customizable web security rules. In conjunction with AWS WAF, CloudFront now can also help you secure your web applications. This blog post will show you how to create an AWS Lambda function to automatically update VPC security groups with AWS internal service IP ranges to ensure that AWS WAF and CloudFront cannot be bypassed.

When using AWS WAF to secure your web applications, it’s important to ensure that only CloudFront can access your origin; otherwise, someone could bypass AWS WAF itself. If your origin is an Elastic Load Balancing load balancer or an Amazon EC2 instance, you can use VPC security groups to allow only CloudFront to access your applications. You can accomplish this by creating a security group that only allows the specific IP ranges of CloudFront. AWS publishes these IP ranges in JSON format so that you can create networking configurations that use them. These ranges are separated by service and region, which means you’ll only need to allow IP ranges that correspond to CloudFront.

In the past, you would use these IP ranges to manually create a security group rule in the AWS Management Console and supply only the prefixes marked for CloudFront. But what would you have done if the IP ranges changed? One solution was to poll the IP ranges’ endpoint periodically with a simple cron job to make sure they were current. This meant you needed infrastructure to support the task. However, you ended up with another host to manage, complete with the typical patching, deployment, and monitoring. As you can see, a small task could quickly become more complicated than the problem it aimed to solve.

An Amazon Simple Notification Service (SNS) topic is generated whenever the AWS IP ranges change. Therefore, you can build an event-driven, zero-infrastructure solution using a Lambda function that is triggered in response to the SNS notification. Let’s get started!

Create a security group

The first thing you need to do is create a security group. This security group will allow only traffic from CloudFront and AWS WAF into your Elastic Load Balancing load balancers or EC2 instances.

You must tag your security group with the protocol it controls (for example Protocol: http or Protocol: https), and have one security group for each. For instance, if your origin receives traffic on both port 80 and port 443, set up a security group tagged with Name: cloudfront, AutoUpdate: true, and Protocol: http, and another security group tagged with Protocol: https. Then, add both security groups to your Amazon EC2 instance or Elastic Load Balancing load balancer and configure the AWS Lambda script.

In the EC2 console:

  1. Click Security Groups > Create Security Group.
  2. Give your security group a meaningful name and description.
  3. Next, view the security group you just created, and add three tags that our Lambda function will use to identify security groups it needs to update: set Name to cloudfront, AutoUpdate to true, and Protocol to either http or https. Any security groups with these tags will automatically get their ingress permissions updated with CloudFront’s IP ranges.

Create an IAM policy and execution role for the Lambda function

When creating a Lambda function, it’s important to understand and properly define the security context to which the Lambda function is subject. Using IAM, you will create the Lambda execution role that determines the AWS service calls that the function is authorized to complete. (Learn more about the Lambda permissions model.)

  1. Before you can create the IAM role, you need to create an IAM policy that you will attach to it. In the IAM console, click Policies > Create Policy > Select (next to Create Your Own Policy).
  2. Supply a name for your policy, and then copy and paste the following policy document into the Policy Document box.
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
          ],
          "Resource": "arn:aws:logs:*:*:*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeSecurityGroups",
            "ec2:AuthorizeSecurityGroupIngress",
            "ec2:RevokeSecurityGroupIngress"
          ],
          "Resource": "*"
        }
      ]
    }

To explain what this policy allows, let’s look closely at both statements in the policy. The first statement allows the Lambda function to write to Amazon CloudWatch logs, which is vital for debugging and monitoring our function. The second statement allows the function to get information about existing security groups and to authorize and revoke ingress permissions. It’s an important best practice that your IAM policies be as granular as possible, to observe the principal of least privilege.

Now that you have created your policy, you can create your Lambda execution role using that policy:

  1. In the IAM console, click Roles > Create New Role, and then name your role.
  2. To select a role type, select AWS Service Roles > AWS Lambda.
  3. Attach the policy you just created.
  4. After confirming your selections, click Create Role.

Create your Lambda function

Now that you have created your Lambda execution role, you are ready to create your Lambda function:

  1. Go to the Lambda console and choose Create function. On the next page, choose Author from scratch. (Because I’ll be providing the code for your Lambda function, you can skip the blueprint step, but for other functions, blueprints can be a great way to get started.)
  2. On the Configure triggers page, choose Next.
  3. Give your Lambda function a name and description, and select Python 2.7 from the Runtime menu.
  4. Paste the following Lambda function code in the Lambda function code box. (You can also download this Lambda function from the aws-cloudfront-samples GitHub repository.) Important note: By default, Lambda configures the SDK in its own region. If the security groups are in a different region than the Lambda function, you must update the SDK client with the correct region (client = boto3.client(‘ec2’,region_name=‘yourregion’)).
    '''
    Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
    Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at
        http://aws.amazon.com/apache2.0/
    or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
    '''
    
    import boto3
    import hashlib
    import json
    import urllib2
    
    # Name of the service, as seen in the ip-groups.json file, to extract information for
    SERVICE = "CLOUDFRONT"
    # Ports your application uses that need inbound permissions from the service for
    INGRESS_PORTS = { 'Http' : 80, 'Https': 443 }
    # Tags which identify the security groups you want to update
    SECURITY_GROUP_TAG_FOR_HTTP = { 'Name': 'cloudfront', 'AutoUpdate': 'true', 'Protocol': 'http' }
    SECURITY_GROUP_TAG_FOR_HTTPS = { 'Name': 'cloudfront', 'AutoUpdate': 'true', 'Protocol': 'https' }
    
    def lambda_handler(event, context):
        print("Received event: " + json.dumps(event, indent=2))
        message = json.loads(event['Records'][0]['Sns']['Message'])
    
        # Load the ip ranges from the url
        ip_ranges = json.loads(get_ip_groups_json(message['url'], message['md5']))
    
        # extract the service ranges
        cf_ranges = get_ranges_for_service(ip_ranges, SERVICE)
    
        # update the security groups
        result = update_security_groups(cf_ranges)
    
        return result
    
    def get_ip_groups_json(url, expected_hash):
        print("Updating from " + url)
    
        response = urllib2.urlopen(url)
        ip_json = response.read()
    
        m = hashlib.md5()
        m.update(ip_json)
        hash = m.hexdigest()
    
        if hash != expected_hash:
            raise Exception('MD5 Mismatch: got ' + hash + ' expected ' + expected_hash)
    
        return ip_json
    
    def get_ranges_for_service(ranges, service):
        service_ranges = list()
        for prefix in ranges['prefixes']:
            if prefix['service'] == service:
                print('Found ' + service + ' range: ' + prefix['ip_prefix'])
                service_ranges.append(prefix['ip_prefix'])
    
        return service_ranges
    
    def update_security_groups(new_ranges):
        client = boto3.client('ec2')
    
        http_group = get_security_groups_for_update(client, SECURITY_GROUP_TAG_FOR_HTTP)
        https_group = get_security_groups_for_update(client, SECURITY_GROUP_TAG_FOR_HTTPS)
        print ('Found ' + str(len(http_group)) + ' HttpSecurityGroups to update')
        print ('Found ' + str(len(https_group)) + ' HttpsSecurityGroups to update')
    
        result = list()
        http_updated = 0
        https_updated = 0
        for group in http_group:
            if update_security_group(client, group, new_ranges, INGRESS_PORTS['Http']):
                http_updated += 1
                result.append('Updated ' + group['GroupId'])
        for group in https_group:
            if update_security_group(client, group, new_ranges, INGRESS_PORTS['Https']):
                https_updated += 1
                result.append('Updated ' + group['GroupId'])
    
        result.append('Updated ' + str(http_updated) + ' of ' + str(len(http_group)) + ' HttpSecurityGroups')
        result.append('Updated ' + str(https_updated) + ' of ' + str(len(https_group)) + ' HttpsSecurityGroups')
    
        return result
    
    def update_security_group(client, group, new_ranges, port):
        added = 0
        removed = 0
    
        if len(group['IpPermissions']) > 0:
            for permission in group['IpPermissions']:
                if permission['FromPort'] <= port and permission['ToPort'] >= port:
                    old_prefixes = list()
                    to_revoke = list()
                    to_add = list()
                    for range in permission['IpRanges']:
                        cidr = range['CidrIp']
                        old_prefixes.append(cidr)
                        if new_ranges.count(cidr) == 0:
                            to_revoke.append(range)
                            print(group['GroupId'] + ": Revoking " + cidr + ":" + str(permission['ToPort']))
    
                    for range in new_ranges:
                        if old_prefixes.count(range) == 0:
                            to_add.append({ 'CidrIp': range })
                            print(group['GroupId'] + ": Adding " + range + ":" + str(permission['ToPort']))
    
                    removed += revoke_permissions(client, group, permission, to_revoke)
                    added += add_permissions(client, group, permission, to_add)
        else:
            to_add = list()
            for range in new_ranges:
                to_add.append({ 'CidrIp': range })
                print(group['GroupId'] + ": Adding " + range + ":" + str(port))
            permission = { 'ToPort': port, 'FromPort': port, 'IpProtocol': 'tcp'}
            added += add_permissions(client, group, permission, to_add)
    
        print (group['GroupId'] + ": Added " + str(added) + ", Revoked " + str(removed))
        return (added > 0 or removed > 0)
    
    def revoke_permissions(client, group, permission, to_revoke):
        if len(to_revoke) > 0:
            revoke_params = {
                'ToPort': permission['ToPort'],
                'FromPort': permission['FromPort'],
                'IpRanges': to_revoke,
                'IpProtocol': permission['IpProtocol']
            }
    
            client.revoke_security_group_ingress(GroupId=group['GroupId'], IpPermissions=[revoke_params])
    
        return len(to_revoke)
    
    def add_permissions(client, group, permission, to_add):
        if len(to_add) > 0:
            add_params = {
                'ToPort': permission['ToPort'],
                'FromPort': permission['FromPort'],
                'IpRanges': to_add,
                'IpProtocol': permission['IpProtocol']
            }
    
            client.authorize_security_group_ingress(GroupId=group['GroupId'], IpPermissions=[add_params])
    
        return len(to_add)
    
    def get_security_groups_for_update(client, security_group_tag):
        filters = list();
        for key, value in security_group_tag.iteritems():
            filters.extend(
                [
                    { 'Name': "tag-key", 'Values': [ key ] },
                    { 'Name': "tag-value", 'Values': [ value ] }
                ]
            )
    
        response = client.describe_security_groups(Filters=filters)
    
        return response['SecurityGroups']
    
    '''
    Sample Event From SNS:
    {
      "Records": [
        {
          "EventVersion": "1.0",
          "EventSubscriptionArn": "arn:aws:sns:EXAMPLE",
          "EventSource": "aws:sns",
          "Sns": {
            "SignatureVersion": "1",
            "Timestamp": "1970-01-01T00:00:00.000Z",
            "Signature": "EXAMPLE",
            "SigningCertUrl": "EXAMPLE",
            "MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
            "Message": "{\"create-time\": \"yyyy-mm-ddThh:mm:ss+00:00\", \"synctoken\": \"0123456789\", \"md5\": \"03a8199d0c03ddfec0e542f8bf650ee7\", \"url\": \"https://ip-ranges.amazonaws.com/ip-ranges.json\"}",
            "Type": "Notification",
            "UnsubscribeUrl": "EXAMPLE",
            "TopicArn": "arn:aws:sns:EXAMPLE",
            "Subject": "TestInvoke"
          }
        }
      ]
    }
    
    '''
    
  1. Below the code window for Lambda function handler and role, select the execution role you created earlier.
  2. Under Advanced settings, increase the Timeout to 5 seconds.  If you are updating several security groups with this function, you might have to increase the timeout by even more time. Finally, click Next.
  3. After confirming your settings are correct, click Create function.

Test your Lambda function

Now that you have created your function, it’s time to test it and initialize your security group:

  1. In the Lambda console on the Functions page, choose your function, choose the Actions drop-down menu, and then Configure test event.
  2. Enter the following as your sample event, which will represent an SNS notification.
    {
      "Records": [
        {
          "EventVersion": "1.0",
          "EventSubscriptionArn": "arn:aws:sns:EXAMPLE",
          "EventSource": "aws:sns",
          "Sns": {
            "SignatureVersion": "1",
            "Timestamp": "1970-01-01T00:00:00.000Z",
            "Signature": "EXAMPLE",
            "SigningCertUrl": "EXAMPLE",
            "MessageId": "95df01b4-ee98-5cb9-9903-4c221d41eb5e",
            "Message": "{\"create-time\": \"yyyy-mm-ddThh:mm:ss+00:00\", \"synctoken\": \"0123456789\", \"md5\": \"7fd59f5c7f5cf643036cbd4443ad3e4b\", \"url\": \"https://ip-ranges.amazonaws.com/ip-ranges.json\"}",
            "Type": "Notification",
            "UnsubscribeUrl": "EXAMPLE",
            "TopicArn": "arn:aws:sns:EXAMPLE",
            "Subject": "TestInvoke"
          }
        }
      ]
    }
  1. After you’ve added the test event, click Save and test. Your Lambda function will be invoked, and you should see log output at the bottom of the console similar to the following.
    Updating from https://ip-ranges.amazonaws.com/ip-ranges.json
    MD5 Mismatch: got 2e967e943cf98ae998efeec05d4f351c expected 7fd59f5c7f5cf643036cbd4443ad3e4b: Exception
    Traceback (most recent call last):
      File "/var/task/lambda_function.py", line 29, in lambda_handler
        ip_ranges = json.loads(get_ip_groups_json(message['url'], message['md5']))
      File "/var/task/lambda_function.py", line 50, in get_ip_groups_json
        raise Exception('MD5 Missmatch: got ' + hash + ' expected ' + expected_hash)
    Exception: MD5 Mismatch: got 2e967e943cf98ae998efeec05d4f351c expected 7fd59f5c7f5cf643036cbd4443ad3e4b
    

You will see a message indicating there was a hash mismatch. Normally, a real SNS notification from the IP Ranges SNS topic will include the right hash, but because our sample event is a test case representing the event, you will need to update the sample event manually to have the expected hash.

  1. Edit the sample event again, and this time change the md5 hash highlighted in red to be the first hash provided in the log output. In this example, we would update the sample event with the hash “2e967e943cf98ae998efeec05d4f351c”.
  2. Click Save and test, and your Lambda function will be invoked.

This time, you should see output indicating your security group was properly updated. If you go back to the EC2 console and view the security group you created, you will now see all the CloudFront IP ranges added as allowed points of ingress. If your log output is different, it should help you identify the issue.

Configure your Lambda function’s trigger

After you have validated that your function is executing properly, it’s time to connect it to the SNS topic for IP changes. To do this, use the AWS Command Line Interface (CLI). Enter the following command, making sure to replace <Lambda ARN> with the Amazon Resource Name (ARN) of your Lambda function. You will find this ARN at the top right when viewing the configuration of your Lambda function.

aws sns subscribe --topic-arn arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged --protocol lambda --notification-endpoint <Lambda ARN>

You should receive an ARN of your Lambda function’s SNS subscription. Your Lambda function will now be invoked whenever AWS publishes new IP ranges!

Summary

As you followed this blog post, you created a security group and a Lambda function to update the security group’s rules dynamically whenever AWS publishes new internal service IP ranges. This solution has several advantages:

  • The solution is not designed as a periodic poll, so it only executes when it needs to.
  • It is automatic, so you don’t need to update security groups manually.
  • It is simple because you have no extra infrastructure to maintain.
  • It is cost effective. Because the Lambda function fires only when necessary and only runs for a few seconds, this solution only costs pennies to operate.

And this is just the tip of the iceberg for AWS WAF. In the coming year, we hope to provide you additional blog posts about how to use AWS WAF.

If you have any questions or comments, please add them in the comments section below or on the Lambda forum. If you have any other use cases for using Lambda functions to dynamically update security groups or even other networking configurations such as VPC route tables or ACLs, we’d love to hear about them as well!

– Travis