I’m excited to finally answer a question I’ve been hearing from both Infrastructure as Code developers and security practitioners for years: “How do I send a signal back to my CloudFormation stack from within a private VPC without going across the public internet?”

You’ve been able to signal success or failure after pausing stack provisioning for years using the AWS CloudFormation cfn-signal helper script with a WaitCondition to coordinate with external provisioners, or a CreationPolicy while installing software on an Amazon EC2 instance, or an UpdatePolicy for Auto Scaling events. For instance, if you’ve paused stack creation with an Amazon EC2 CreationPolicy, here’s how you would signal your stack the results of your actions from inside the EC2 instance:

cfn-signal -e $? --stack ${AWS::StackName} --resource MyInstance --region ${AWS::Region}

Historically, this traffic needed to traverse the public internet to reach the CloudFormation API endpoints. This meant setting up a NAT instance or inline proxy if the script was running in a non-public part of your Amazon Virtual Private Cloud (VPC). However, with the recent addition of CloudFormation support for AWS PrivateLink, you can now make these calls to the CloudFormation API from inside your VPC using a private endpoint, routed entirely within the AWS network. Adding a CloudFormation endpoint to your VPC helps you meet your requirements to limit public internet connectivity.

I’m going to walk through a couple examples that demonstrate the bare minimum resources required to use cfn-signal from an EC2 instance inside a private subnet. The first example is for an EC2 instance sending a cfn-signal to its CreationPolicy in a CloudFormation stack when it is ready to receive traffic. The second is for a WaitCondition used for more complex orchestration with other, non-EC2 resources. In those cases you will also need an Amazon S3 endpoint in order to respond to the self-signed Amazon S3 URLs for those resources.

The templates I’m using are available on awslabs GitHub. Feel free to grab them and follow along. I’ve created versions with internet gateways and bastion instances to allow you to use SSH to connect to the private EC2 and take a look at what’s going on. I’ve also created fully self-contained VPCs with no external access (other than the VPC endpoints) to demonstrate true isolation.

Define the CloudFormation endpoint

I’ll start by defining the PrivateLink endpoint I’m going to use for CloudFormation:

    CfnEndpoint:
      Type: AWS::EC2::VPCEndpoint
      Properties:
        VpcId: !Ref VPC
        ServiceName: !Sub "com.amazonaws.${AWS::Region}.cloudformation"
        VpcEndpointType: "Interface"
        PrivateDnsEnabled: true
        SubnetIds:
          - !Ref PrivateSubnet1
          - !Ref PrivateSubnet2
        SecurityGroupIds:
          - !Ref EndpointSG

Note that there are two different kinds of endpoints for your VPC:

• Gateway endpoints for Amazon S3 and Amazon DynamoDB (I’ll be talking about them later)
• Interface endpoints (aka PrivateLink), like the one I’m using here for CloudFormation.

These two endpoint types work slightly differently, as detailed in this blog. The short version is that Interface endpoints place routable elastic network interfaces (ENIs) directly into the subnets of your VPC, providing private IP addresses for whichever service you define.

With that in mind, let’s break down the endpoint defined above. I’ve defined it as an Interface endpoint for the CloudFormation namespace. I’ve included the AWS::Region pseudo-parameter so I can use this template in any AWS Region that supports PrivateLink. I’ve provided references to the VPC ID and a list of Subnet IDs where PrivateLink will place the endpoint’s routable ENIs. This is an endpoint for an AWS service, so I need to open up HTTPS access on port 443 using a Security Group. I also need to enable Private DNS to get a fully qualified domain name (FQDN) for CloudFormation that I can use in my VPC.

With my PrivateLink defined, I can define a typical VPC (with or without public internet access) that includes the VPC and private subnets that I referenced previously.

Define the private EC2 instance

Now that I have the network pieces defined, I can move on to defining the private EC2 instance that will send the signal. Remember that in this first scenario, I’m going to be responding to a CreationPolicy. This is the typical way to tell a CloudFormation stack to wait for things like package installations or software deployment for an EC2 instance or Auto Scaling group. Here’s my instance resource:

    PrivateInstance:
      DependsOn: CfnEndpoint
      Type: AWS::EC2::Instance
      Properties:
        InstanceType: t3.micro
        SecurityGroupIds:
        - !Ref PrivateSG
        SubnetId: !Ref PrivateSubnet1
        ImageId: !Ref LinuxAMI
        UserData:
          Fn::Base64:
            !Sub |
              #!/bin/bash -x
              date > /tmp/datefile
              cat /tmp/datefile
              # Signal the status from this instance
              /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} \
                --resource PrivateInstance --region ${AWS::Region}
        Tags:
          -
            Key: Name
            Value: Private
      CreationPolicy:
        ResourceSignal:
          Count: 1
          Timeout: "PT15M"

The CreationPolicy is defined at the bottom and tells my stack to wait for one success signal. If it doesn’t receive a signal in 15 minutes it will Fail the resource and start a ROLLBACK of the stack.

The signal itself is defined in the UserData section of the instance. Nothing special here. I already configured the private DNS entry for my CloudFormation endpoint for this VPC, so cfn-signal will call the local endpoint instead of the public one.

Handling WaitConditions

Next, I’ll move on to the wait condition use case. While a CreationPolicy receives its signals directly from the CloudFormation endpoint, a WaitCondition also needs an Amazon S3 endpoint to respond to the self-signed object URLs for those resources. As I mentioned earlier, Amazon S3 uses gateway endpoints, which work a little differently than the interface endpoint I created for CloudFormation. Here’s my S3 endpoint:

    S3Endpoint:
      Type: AWS::EC2::VPCEndpoint
      Properties:
        VpcId: !Ref VPC
        ServiceName: !Sub "com.amazonaws.${AWS::Region}.s3"
        VpcEndpointType: "Gateway"
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal: "*"
              Action:
                - "s3:PutObject"
              Resource:
                - !Sub "arn:aws:s3:::cloudformation-waitcondition-${AWS::Region}/*"
        RouteTableIds:
          - !Ref PrivateRouteTable1
          - !Ref PrivateRouteTable2

This gateway endpoint lives outside my VPC, so there are no routable IP addresses, DNS, or security groups involved. Instead, the endpoint is routed to the S3 API based on an IAM policy. I’ve tightly scoped the policy for my endpoint to only allow S3 PutObject actions to the CloudFormation regional S3 bucket for WaitConditions.

And that’s it! With this additional endpoint I can now add an AWS::CloudFormation::WaitCondition to my template that depends on any other resource, and signal it from my private EC2 instance:

    PrivateWaitHandle:
      Type: AWS::CloudFormation::WaitConditionHandle

    PrivateWaitCondition:
      DependsOn: PrivateInstance
      Type: AWS::CloudFormation::WaitCondition
      Properties:
        Handle: !Ref PrivateWaitHandle
        Timeout: '3600'
        Count: 1

I’d like to stress that if you’re signaling back from an EC2 instance or an Auto Scaling group, you really should be using CreationPolicies and UpdatePolicies. They’re easier to configure, easier to code, and have some cool additional features for Auto Scaling. Save WaitConditions for coordinating more complex workflow logic with other resources.

Conclusion

You’re ready to go! You have everything you need to set up private signaling to CloudFormation without the need for proxies, NATs, or even an internet gateway on your VPC! Feel free to use these templates as a foundation for your own architectures – and you’re always welcome to reach out to me on Twitter or submit an Issue/Pull Request if you have a question or come up with an improvement.

About the Author

Chuck Meyer is a Senior Developer Advocate for AWS CloudFormation based in Columbus, Ohio. He spends time working with both external and internal customers to constantly improve the developer experience for AWS CloudFormation users. He’s a live music true believer and spends as much time as possible playing bass and watching bands. He also does the tweets (@chuckm).