How to Provision Complex, On-Demand Infrastructures by Using Amazon API Gateway and AWS Lambda

Many AWS customers are using the power of AWS CloudFormation to customize complex infrastructures. At the same time, they are moving towards self-service for their expanding customer bases. How can complex infrastructure be provisioned on-demand while minimizing customer use of the AWS Management Console?

Let’s say AnyCompany uses AWS services to process sensitive datasets owned by its customers. These customers need to be able to provision their own processing infrastructure on demand. However, AnyCompany doesn’t want the customers to access AWS CloudFormation directly, or see how customer data is processed. How can AnyCompany create resources with CloudFormation templates without exposing proprietary processing methods?

You can use Amazon API Gateway and AWS Lambda to provide complex, on-demand, data-processing infrastructure to users who have no need to access the AWS Management Console. In this post, we walk through the setup and configuration of working code examples that you (and your customers) can use to provision on-demand infrastructure. This post’s solution combines principles of immutable computing with a method to provide granular access to powerful capabilities available in the AWS Cloud. In this post, you create immutability for an Amazon EC2 instance by provisioning it without an SSH key or any other access mechanism. The instance can’t be altered after it’s launched.

Solution architecture

API Gateway simplifies the creation, management, and deployment of APIs. Integration of API Gateway with Lambda, the AWS serverless compute service, allows for further interaction with the larger family of AWS services. In this post, the Lambda function provisions on-demand CloudFormation infrastructure stacks for our example service’s primary business function:  the calculation of pi.

Two CloudFormation templates are used to provision infrastructure stacks:

• The primary template
• The business-function template

The primary CloudFormation template creates the base infrastructure that is shown in the following diagram.

The primary template creates a stack of resources:

• API Gateway resources
• The associated AWS Identity and Access Management (IAM) security roles
• An Amazon S3 bucket for the results of the data processed by the business-function template
• A Lambda function specifically for the purpose of creating multiple iterations of a second repeatable infrastructure. This repeatable infrastructure, contained within the second CloudFormation template, is referred to as the business-function template.

The API created by the primary template allows system users to pass selected parameters to the business-function template, such as cost-center tags, a unique name, and data-processing parameters. The ability to pass parameters allows the business-function infrastructure to adapt to the specific needs of each request. In this post’s example—calculating the first 15,000 digits of pi—the number of digits calculated can be specified in the request initiated by the customer. For your own business function templates, this could be the location of input data or an email address to which results are sent.

In the example business-function template, the template provisions an immutable EC2 instance that calculates a specified number of digits of pi and uploads the results to the S3 bucket. The business-function stack then self-terminates, reducing any potential extra costs.

This architecture allows you, at multiple points, to control access, processing parameters, and workflow.

In the earlier diagram, (1) all access to the AWS Management Console and API for your account’s infrastructure is segregated through the use of API Gateway. Then, (2) IAM role–based restrictions are in place for API Gateway to call the Lambda function. The Lambda function serves to selectively add, subtract, or alter input parameters from external users. The code for this function is embedded in the primary CloudFormation template. You can modify the code to add specific tags for billing information or call other AWS services such as Amazon DynamoDB or Amazon RDS to keep a record of which infrastructure is instantiated by which API users.

In our example template, the Lambda function is populated to pass only the required parameters for the business-function template’s calculation of pi. The use of the function restricts the ability of customers to inject unexpected or undesirable parameters. The solution then (3) uses a Lambda execution role to control the Lambda function’s access. Finally, (4) the Lambda function initiates the CloudFormation business-function template by using another IAM role that is restricted to instantiating only the resources required.

The use of an intermediate Lambda function here allows for heavy customization and filtering of the input requests from the customer-facing API Gateway.

Solution Features

By using this API Gateway–based solution, you can benefit in a number of ways:

• User requests are decoupled from the infrastructure that fulfills those requests.

  This post’s solution removes the need for users to have access to the AWS Management Console. Users can accomplish their infrastructure-backed requests on demand without having any direct access to the console. Additionally, processing methods and infrastructure can be switched in and out without any change to the external appearance of your service. For example, in this post, I could have a containerized solution such as Amazon EC2 Container Services (Amazon ECS) process my pi-calculation service without any visibility to users.

• Proprietary processing methods and infrastructure are obscured from users.

  The solution in this post can protect the proprietary secrets of your company’s data processing service by obscuring which processing methods you are using and which infrastructure performs the processing. The solution also isolates users from viewing each other’s operations. Without having console access, users cannot determine which running processing stacks are initiated by other users.

• On-demand infrastructure in specific configurations is created without allowing users to provision arbitrary infrastructure.

  The solution in this post complements the functionality of the AWS Service Catalog. Beyond allowing only preapproved infrastructure and preventing unapproved resources from executing outside your company’s policies, this API Gateway–driven method delivers a specific, processed result. For ease of troubleshooting and integration with DevOps workflows, you can standardize, version, and deploy complicated CloudFormation stacks for networking, compute, storage, and big data processing.

• Simplification of the user experience.

  This solution also simplifies the user experience. Commands that request a specific result are limited to a single, familiar REST API interface call that delivers only that result.

• Cost savings through self-terminating infrastructure.

  The architecture of this solution is augmented through the use of self-terminating CloudFormation stacks. Stacks can spin up expensive infrastructure to perform data processing on demand, and self-terminate as soon as the task has completed. This can save you infrastructure costs associated with idle resources.

Walkthrough: Deploy the solution

This post’s example templates create the base infrastructure and a business-function process. The business-function process can instantiate a single EC2 instance to calculate a specified number of digits of pi and deposit the results in an S3 bucket. As noted previously, this example includes built-in logic to automatically self-terminate the business-function template’s CloudFormation stacks after instantiation.

The following steps walk through how to provision this solution for creating on-demand business-function CloudFormation stacks through API Gateway. You can modify the business-function template code for your specialized infrastructure needs. However, you must upload the modified template to your own S3 bucket before completing Step 1 because the business-function template location is required for the primary template’s parameters.

Both templates are designed for use in the us-west-2 region and may require minor modifications to function properly in other regions.

Costs

API Gateway costs are based on the number of API calls requested, and the single call here leads to a near-zero cost. For current pricing, see Amazon API Gateway Pricing. The associated S3 costs and Lambda costs for this post also are negligible because you are not transferring much data or spending a significant amount of time processing serverless code. The most expensive part of this post’s solution is the on-demand EC2 instance cost invoked in each business-function template instantiation. This cost is approximately $0.012 per hour for each run with the default t2.micro EC2 instance. In total, running the solution as presented in this post should cost you less than $0.10 in order to test multiple calculations of pi.

1. Deploy the CloudFormation template

First, deploy the primary CloudFormation template for API Gateway and other resources. This template creates IAM policies, IAM roles, API Gateway resources, a new S3 bucket, and a Lambda function.

The primary template requires you to have enabled API Gateway Usage Plans.

You can deploy the primary template and launch the primary stack in the us-west-2 region by selecting the following button. Aside from Stack name, the only required parameters are the exact location of the business-function template in S3 and a unique S3 bucket name as a location for template results.

1. In the CloudFormation console, on the Specify Details page, type a stack name.
2. For NewS3BucketName, type a unique name.
3. For S3CFTLocation, leave the default value. If you adapt the primary template for your own customized business-function template, be sure to include the entire URI of the S3 location (for example, https://s3-us-west-2.amazonaws.com/mys3bucket/businessFunctionTemplate.yaml).

1. On the Options page, no changes are required, but you can specify additional tags if desired.
2. On the Capabilities page, acknowledge that CloudFormation might create IAM resources, as shown in the following screenshot. To finish creating the stack, choose Create.

2. Create an API key for the API Gateway

Though you could modify the example primary template for public use, this example restricts access to execute API calls through the use of API keys. After you create the API key, you must associate the new API key with the usage plan created by the primary template.

1. In the API Gateway console, choose API Keys.
2. Choose Actions, Create API key.
3. For Name, type a name for the key. For API Key, leave the default Auto Generate
4. (Optional) For Description, type a value.
5. Choose Save.
6. Choose Add to Usage Plan and type WrappingApiUsagePlan. This usage plan defaults your Stage to PROD because it is the only configured stage.
7. To add the key to the usage plan, choose the green check icon. Your key should resemble the following screenshot.
8. Choose Show and note the API key value for the next step.

3. Initiate a REST API query

Now, initiate a REST API query against the API that you created. in the example curl command below, replace the placeholder API key specified in the x-api-key HTTP header with your API key.

You also need to replace the example URI path with your API’s specific URI. Create the full URI by concatenating the base URI listed in the CloudFormation output for the primary template stack with the primary template’s API call path, which is /PROD/PerRunExecute. This full URI for your specific API call should closely resemble the URI path (e.g. https://abcdefghi.execute-api.us-west-2.amazonaws.com/PROD/PerRunExecute) in the following example curl command.

curl -v -X POST -i -H "x-api-key: 012345ABCDefGHIjkLMS20tGRJ7othuyag" https://abcdefghi.execute-api.us-west-2.amazonaws.com/PROD/PerRunExecute -d '{
  "StackName":"MyCalculatePiStack",
  "CostCenter":"12345",
  "DigitsToCalculate": "15000"
}'

When you run the curl command, it executes a POST query to the REST API to instantiate the business-function template and calculate the first 15,000 digits of pi. A successful API query returns information on the curl request as well as CloudFormation API output similar to the following, indicating an “HTTPStatusCode” of 200.

{"StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/MyCalculatePiStack/1234debf-9423-11e7-9a08-50399b8d8a8c", "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "RequestId": "0cbabc97-9423-11e7-97b3-395d92028ea7", "HTTPHeaders": {"x-amzn-requestid": "0cbabc97-9423-11e7-97b3-395d92028ea7", "date": "Thu, 07 Sep 2017 23:19:52 GMT", "content-length": "388", "content-type": "text/xml"}}}

4. Examine the output

After the EC2 instance completes its processing and uploads the results to the S3 bucket created by the primary template, you can examine the output. Using the configured t2.micro EC2 instance, instantiation and calculation for the first 15,000 digits of pi takes approximately 5–10 minutes. You can monitor progress in the EC2 console (where you see a new EC2 instance running) or the CloudFormation console (where you see the provisioned stack in a “created” state).

After the first 15,000 digits of pi have been uploaded to the S3 bucket, the EC2 instance self-terminates the stack. You can then download the results file from the S3 bucket, as shown below. The naming convention for pi calculating runs is “PiCalculation” concatenated with an epoch timestamp from the time of execution.

This file contains the instance ID that performed the processing, a time stamp, and the first 15,000 digits of pi. The output should look similar to the following screenshot.

5. Clean up

To help minimize your costs, clean up any infrastructure that resulted from following the solution in this post. Remember that you first have to detach the API key that you created from the usage plan in order to delete the primary CloudFormation stack. You can do this by clicking the X for associated usage plans for your API key, as shown in the following screenshot.

You also must empty the S3 results bucket of all files for the bucket to be deleted when the CloudFormation stack self-terminates.

This solution can be adapted to any CloudFormation template encompassing a business function for your service.

Adapt to Your Services

To adapt this example to your own business-function:

1. Write a CloudFormation template that implements your new business function.
2. In the primary template, customize the Lambda resource object code for “PerRunCloudFormationExecutionLambda” to accept and pass on the specific parameters needed by your business function template. Save it with a new name.
3. Upload your custom business-function CloudFormation template developed in step 1 to an S3 bucket with permissions that allow the CloudFormation service to access the template objects. For more information about S3 access control, see Managing Access Permissions to Your Amazon S3 Resource.
4. Start your new stack with the modified primary CloudFormation template. Pass it, as a parameter, the name of your new business function template as a parameter. You can reuse the same API Gateway usage plan created previously, if desired.
5. Test, using your new stack name in place of the bolded content above, and posting an API Gateway query with parameters needed by your new business function.

Conclusion

By using API Gateway, you can control complex, on-demand CloudFormation stacks that create AWS infrastructure. When you combine this functionality with self-terminating templates, you can realize significant cost savings. API Gateway also allows for standardization of infrastructure. It can enable your users to instantiate complex and costly architectures on demand and only for as long as your users need them.

Though the solution in this post assumes that the business function is to calculate the first 15,000 digits of pi (a less than profitable venture these days), you can adopt the solution to deliver significant cost savings. For example, instead of a single EC2 instance, the business-function template could instantiate Amazon Redshift (a petabyte-scale data warehouse), Amazon EMR, or any complex processing infrastructure. As with all on-demand AWS services, you pay for the infrastructure only when it is running.

If you have comments about any of this content, submit them in the “Comments” section below. If you have questions about implementing this solution, start a new thread on the API Gateway forum.