AWS DevOps & Developer Productivity Blog

Customers, CloudFormation, and Custom Resources

I was at re:Invent in Las Vegas a few months ago, and my favorite part about our annual conference also happens to be what I enjoy most about my job: meeting with customers and learning not only how they use our services, but also how they would like to see them improved. One evening after a day of great talks I spent some time with engineers from a company who use CloudFormation to provision and manage pretty much every part of their application, from VPCs for both dev and prod environments, to the EC2 Instances that run their app. They had a particularly interesting use-case that involved CloudFormation and Elastic IP (EIP) addresses that we’re going to focus on today.

EIPs and CloudFormation

An EIP is a persistent, static, and public IP address that can be attached to an EC2 instance. They have been supported in CloudFormation for some time. Here’s a snippet that provisions and attaches an EIP to an EC2 instance:

"Resources" : {
    "Ec2Instance" : {
      "Type" : "AWS::EC2::Instance",
      ...
      }
    },
    "IPAddress" : {
      "Type" : "AWS::EC2::EIP"
    },
    "IPAssoc" : {
      "Type" : "AWS::EC2::EIPAssociation",
      "Properties" : {
        "InstanceId" : { "Ref" : "Ec2Instance" },
        "EIP" : { "Ref" : "IPAddress" }
      }
    }
  }

This diagram illustrates how CloudFormation uses the EC2 API to provision the IPAddress resource, returning its physical ID (i.e., a public IPV4 address) that can be Ref’d by the IPAssoc resource:

If I view the stack’s resources in the AWS CloudFormation Management Console I can see the Logical ID, Physical ID, and type of the resources:

The Customer’s Challenge

When you create a stack including the above template snippet, CloudFormation creates a brand new EIP (i.e., you don’t know the address in advance) per the IPAddress declaration, then the Ec2Instance resource, and finally associates the address with the instance. When you delete the same stack, CloudFormation removes the EIP association, then terminates the EC2 instance and deletes the EIP.

Here’s where the customer’s use-case gets interesting: the EC2 instances they were provisioning and managing with CloudFormation are connecting to 3rd-party APIs (for example, a credit card payment processing gateway) that require IP whitelisting. They had previously provisioned a pool of tens of EIPs and gone through the manual whitelisting process with their 3rd-party providers at some point in the past. Although they dynamically provision their VPC and EC2 instances for dev, test, and prod with CloudFormation, they needed the EIPs attached to those instances to come from their pre-allocated pool. Declaring and attaching a “Type” : “AWS::EC2::EIP” resource the standard way wouldn’t work for their scenario.

Fortunately, CloudFormation allows developers to define Custom Resources for exactly these types of situations. In fact, earlier that day at re:Invent a few CloudFormation engineers gave a great talk on Custom Resources and a framework they released to make developing them easier (video of their talk is on YouTube; I highly recommend you check it out). This customer’s need to provision EIPs from a pool seemed like a great fit for Custom Resources and the new framework, so I offerd to put one together.

A CloudFormation Custom Resource for EIPs

Before we get into the solution, here’s a quick background on Custom Resources: A CloudFormation Custom Resource provides a way for a template developer to include resources in an AWS CloudFormation stack that they define. In this case, the resource is still an Elastic IP Address, but it will be provided by the developer from a pool of existing addresses that they own. The allocation and deallocation of the resource is declared in the CloudFormation template and is part of the stack workflow (i.e., create, update, or delete).

Pooling EIPs in DynamoDB

Remember that the EIPs in this example will be pre-provisioned and whitelisted with some third-party service (for example, a credit card payment processing gateway). It’s reasonable to assume that certain EIPs may be associated with certain services or providers, and that we want to pool them accordingly. We also need to track – with strong consistency – which EIPs are are in use, and where (i.e., which CloudFormation stack) they are used. DynamoDB fits the bill perfectly. Here’s the table structure I defined and then manually entered my pre-provisioned EIPs into:

Pool (Hash Key) Address (Range Key) stack_id logical_id
default 54.208.124.113 arn:aws:cloudformation:us-east–1:…:stack/stack1 IPAddress
default 54.208.220.64    
CCPaymentService 54.209.55.41    
CCPaymentService 54.235.107.220 arn:aws:cloudformation:us-east–1:…:stack/stack3 IPAddress
CCPaymentService 54.236.244.12    

By using a DynamoDB hash+range key (you can read more about the DynamoDB data model at http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataModel.html), I can retreive all EIPs in a given named pool (i.e., CCPaymentService). If an item has the stack_id attribute set, that means the address for that item is in use; I can mark an address as ‘in use’ by setting the stack_id attribute to the value of the CloudFormation stack that is using the EIP.

Here’s how I might implement that functionality in a Python method using the boto library:

def get_address(pool):
    """Retrieve an EIP for the given pool from DynamoDB"""

    #Connect to ddb
    conn = boto.dynamodb2.connect_to_region(options.region)
    ddb = Table(options.table_name, connection=conn)

    # Get available EIPs from pool
    eips = ddb.query(
        pool__eq=pool,
        consistent=True
    )

    # Raise an exception if there are no EIPs in the named pool
    if not eips:
        raise FatalError(u"No EIPs found in pool %s" % pool)

    # Iterate returned addresses and find first address that does not have
    # a stack_id attribute (meaning it's in use)
    address = None
    for eip in eips:
        if not eip.get('stack_id', False):
            eip['stack_id'] = 'SOME STACK ID'
            eip['logical_id'] = 'SOME LOGICAL RESOURCE ID'
            if eip.save():
                address = eip['address']
                break

    # Raise an exception if all addresses are in use
    if not address:
        raise FatalError(u"All EIPs in pool %s are in use" % pool)

    return address

Deleting an address is simply a matter of clearing the stack_id and logical_id attributes for an item.

Integrating with CloudFormation

Now that we have a simple convention for managing our pre-provisioned, whitelisted EIPs in a DynamoDB table, and a small bit of Python to retrieve and manage those addresses, we need a way to declare these EIPs in CloudFormation so we can attach them to EC2 instances in our stacks. Let’s jump ahead a bit and look at how I’ll declare this Custom Resource in my template, then we’ll step back and talk more about the implementation:

"IPAddress" : {
    "Type" : "Custom::EipLookup",
    "Version" : "1.0",
    "Properties" : {
        "ServiceToken" : { "Ref" : "EipLookupServiceToken" },
        "pool" : "CCPaymentService"
    }
}

Compare that to the built-in EIP declaration we saw earlier:

"IPAddress" : {
  "Type" : "AWS::EC2::EIP"
}

We can see the Type is different, and that we’ve included a Version. We also see a Properties key that defines two really important things:

  1. ServiceToken: Required. This is the ARN (Amazon Resource Name) of an existing Amazon SNS (Simple Notification Service) Topic. CloudFormation will publish the contents of the Custom Resource declaration (i.e., the JSON snippet) to that topic whenever a stack is Created, Updated, or Deleted (and will include which of those lifecycle events is occurring).

  2. pool: This indicates which pool the EIP should come from, and maps to the Pool hash key in the DynamoDB table. It will be included in the message that CloudFormation sends to the SNS Topic that the ServiceToken points to.

So, if I create a stack with the above EIP Custom Resource, CloudFormation will publish a message to the SNS topic indicated in the Servie Token basically saying “Please create a v1.0 Custom::EipLookup resource from the CCPaymentService pool. Also, please let me know when you’re done, and the value of what you created.” The message published to SNS would look similar to:

{
   "RequestType" : "Create",
   "ResponseURL" : "http://pre-signed-S3-url-for-response",
   "StackId" : "arn:aws:cloudformation:us-east-1:EXAMPLE/stack-name/guid",
   "RequestId" : "unique id for this create request",
   "ResourceType" : "Custom::EipLookup",
   "LogicalResourceId" : "IPAddress",
   "ResourceProperties" : {
      "pool" : "CCPaymentService"
   }
}

The Custom Resource Bridge framework, discussed in the next section, helps us link our custom code to these CloudFormation events and notifications. For now, let’s extend our earlier diagram, replacing the Amazon EC2 API with the SNS Topic (i.e., ServiceToken) and showing the idea of our Custom Resource as a black box that queries our DynamoDB table:

Custom Resource Bridge (CRB) Framework

The CRB is a piece of software released by the CloudFormation team at re:Invent in November 2013 (and available under the Apache license on GitHub at https://github.com/aws/aws-cfn-resource-bridge). It runs on an EC2 instance (preferably in an Auto Scaling Group with min=1) and takes care of a lot of the work (everything embodied in the above diagram’s black box) to connect your Custom Resource code with CloudFormation lifecycle events by introducing a few conventions to follow:

  1. The SNS Topic you use in your Custom Resource’s ServiceToken should have an SQS Queue subscribed.

  2. Tell the CRB the name of that SQS Queue and a script to invoke when a message is received. Here’s an example CRB config file:

    [eip-lookup]
resource_type=Custom::EipLookup
queue_url=https://your-sqs-queue-url-that-is-subscribed-to-the-sns-topic-in-the-service-token
timeout=60
default_action=/home/ec2-user/lookup-eip.py

  3. When the CRB receives a message, it will parse the JSON, convert the relevant information to environment variables, and invoke your script.

  4. Your script gets information about what it should do (i.e., create, update, or delete) from the environment. Here’s a few lines of Python that infer the request type and pool from the environment, then calls the get_address method we saw earlier if the resource is being created:

    # Get the Request Type and EIP Pool
request_type = os.getenv('Event_RequestType')
pool = os.getenv('Event_ResourceProperties_pool', 'default')

...

# Get a new address from the pool
if request_type == 'Create':
    physical_id = get_address(pool)

  5. Communicate your result by printing to stdout. For example, if my resource got an EIP from the pool in response to a CREATE event, I would print it out and the CRB will communicate that back to CloudFormation:

    # Write out our successful response!
if request_type != 'Delete':
    print u'{ "PhysicalResourceId" : "%s" }' % physical_id

Handling Updates and Deletes

Any custom resource you make should handle the update and delete stack lifecycle events. In this EIP example, a user could decide to use an EIP from a different pool for an existing stack. The template might look like:

"IPAddress" : {
    "Type" : "Custom::EipLookup",
    "Version" : "1.0",
    "Properties" : {
        "ServiceToken" : { "Ref" : "EipLookupServiceToken" },
        "pool" : "ATotallyDifferentPool"
    }
}

If the user called UpdateStack with this new template, my Custom Resource would be notified of the update event, and in addition to the current declaration would also be given the previous state of the resource. This allows my code to decide if a change was made and update the EIP table accordingly:

# Get the Request Type and EIP Pool
request_type = os.getenv('Event_RequestType')
pool = os.getenv('Event_ResourceProperties_pool', 'default')

...

elif request_type == 'Update':
    old_pool = os.getenv('Event_OldResourceProperties_pool')
    old_address = os.getenv('Event_PhysicalResourceId')

    # If the updated resource wants an EIP from a different pool
    if not pool == old_pool:
        # And get a new one
        physical_id = get_address(pool)
    else:
        physical_id = old_address

Finally, a Delete event is handled by simply removing the stack_id attribute from the address in DynamoDB:

def delete_address(pool, address):
    """Mark an EIP as no longer in use"""
    #Connect to ddb
    conn = boto.dynamodb2.connect_to_region(options.region)
    ddb = Table(options.table_name, connection=conn)

    eip = ddb.get_item(pool=pool, address=address)
    del eip['stack_id']
    del eip['logical_id']
    eip.save()

...

# Get the Request Type and EIP Pool
request_type = os.getenv('Event_RequestType')
pool = os.getenv('Event_ResourceProperties_pool', 'default')

...

elif request_type == 'Delete':
    address = os.getenv('Event_PhysicalResourceId')
    delete_address(pool, address)

Try it Out.

Here’s how you can use the CRB and the EIP Custom Resource we just discussed, in 4 easy steps:

  1. Download the custom-resource-runner.template CloudFormation template from GitHub and create a new stack in the CloudFormation Management Console. This creates an SNS Topic (i.e., ServiceToken), wires up an SQS Queue, then launches an EC2 instance in an Auto Scaling Group and installs the CRB along with the Python script, and finally creates the DynamoDB table to enter your EIP addresses into. It’s this part of the architecture:

  2. After the stack launches, open the Outputs tab and copy the value for ServiceToken to your clipboard:

  3. Open the EC2 Management Console, click the Elastic IPs link, and allocate a few new addresses. Then open the DynamoDB Management Console, select the EipCustomResource table that was created by CloudFormation in Step 1 and click the Explore Table button. Click New Item and add the EIPs you just provisioned. Use default as the pool name.

  4. Download the example.template CloudFormation template from GitHub and create a new stack. Provide the ServiceToken value you copied in Step 2. This provisions a stack that uses your EIP Custom Resource to provision and attach a pooled EIP. It’s this part of the architecture:

Cleanup

When you’re done trying out the sample, be sure to delete the stacks you launched and release any EIPs you allocated as part of the trial.

Next Steps

CloudFormation Custom Resources are a really powerful tool for integrating your own custom code into the CloudFormation workflow. There are 4 other fully-functional example Custom Resources on GitHub that you can explore, try out, and use as references to build your own, including:

If you come up with a cool Custoom Resource you’d like to share, we actively maintain the GitHub repo and love Pull Requests!

Finally, Each of these samples is discussed in detail in this excellent re:Invent video by CloudFormation engineers D.J. Edwards and Adam Thomas: