AWS Developer Tools Blog
Handling arbitrary HTTP requests in Amazon API Gateway
In this post, I walk you through the steps to build a serverless web application that can accept arbitrary HTTP requests and use custom logic to return arbitrary responses. The concepts in this post are applicable to any situation where you require flexible control over the HTTP requests received and returned by an application that uses Amazon API Gateway.
You can find all of the code and resources used throughout this post in the associated GitHub repository.
Solution background
You are building a web application for a technical, puzzle-solving competition. Some parts of the competition require entrants to construct and send specific types of HTTP requests to the application. Your application also must be able to construct and return custom responses.
You know in advance that traffic to the application will be sporadic and unpredictable as people work their way through the competition. So, you opt to build a serverless application using AWS CloudFormation, AWS Lambda, and Amazon API Gateway.
The web application has the following requirements:
- The application code must be able to accept and handle HTTP requests with these requirements:
- The HTTP responses sent by the application must also be similarly configurable, including sending custom headers and response body.
- The application must define all of its infrastructure as code, so that you can easily deploy and test multiple environments as necessary.
Groundwork
Most of the requirements are met by using the ANY method and proxy path features of API Gateway, as outlined in API Gateway Update – New Features Simplify API Development.
If you haven’t read that post, here’s a short summary of the outcome: An API Gateway API with a single resource that has the path /{proxy+} and the method ANY. The resource is configured to proxy all requests to a Lambda function using the Lambda proxy integration feature of API Gateway.
For now, the Lambda function simply returns the integration request input as JSON. The Python code for the Lambda function looks like the following code example:
import json
def handler(event, _):
return {
"statusCode": 200,
"headers": {
"Content-Type": "application/json",
},
"body": json.dumps(event, indent=4),
}
This code example instructs API Gateway to send back a 200 response, with the Content-Type header set to application/json. It also sets the response body to a JSON-encoded copy of the event object received by the Lambda function.
Because you intend to define the entire application as code, re-create the preceding API Gateway configuration as an OpenAPI document (which you later embed into an AWS CloudFormation template):
{
"openapi": "3.0",
"info": {
"title": "All-capturing example",
"version": "1.0"
},
"paths": {
"/{proxy+}": {
"x-amazon-apigateway-any-method": {
"responses": {},
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": ""
}
}
}
}
}
You can now make a request to the API endpoint and get the integration request data back:
curl -i http://abcdef.execute-api.us-east-1.amazonaws.com/Prod/foo
content-type: application/json
content-length: 3332
date: Wed, 21 Aug 2019 15:43:03 GMT
{
// Some fields removed for brevity
"path": "/foo",
"httpMethod": "GET",
"headers": {
},
"queryStringParameters": null,
"body": null,
"isBase64Encoded": false
}
Walkthrough
You now have the basis of the application, but there are still a few things left to configure to meet all of the requirements.
Handling all paths
The application must be able to receive requests at any path, including the root path: /. An API Gateway resource with a path of /{proxy+} captures every path except the root path. Making a request for the root path results in a 403 response from API Gateway with the message Missing Authentication Token.
To fix this omission, add an additional resource to the API with the path set to / and link that new resource to the same Lambda function as used in the existing /{proxy+} resource.
The updated OpenAPI document now looks like the following code example:
{
"openapi": "3.0",
"info": {
"title": "All-capturing example",
"version": "1.0"
},
"paths": {
"/": {
"x-amazon-apigateway-any-method": {
"responses": {},
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": ""
}
}
},
"/{proxy+}": {
"x-amazon-apigateway-any-method": {
"responses": {},
"x-amazon-apigateway-integration": {
"httpMethod": "POST",
"type": "aws_proxy",
"uri": ""
}
}
}
}
}
Handling all content types
Now your application can handle any type of HTTP request at any path, but there’s one significant issue remaining: handling binary data.
If you POST a binary payload to the API at the moment, the response can be corrupted by automatic encoding and decoding to and from textual representations of the binary payload. To demonstrate this, use a 1×1 pixel GIF file that is 35 bytes long. If you post it to the API, you get the following response back (again, simply echoing the request that you sent):
curl --data-binary "@1pixel.gif" http://abcdef.execute-api.us-east-1.amazonaws.com/Prod/
{
// Some fields removed for brevity
"path": "/",
"httpMethod": "POST",
"headers": {
"content-type": "image/gif",
},
"queryStringParameters": null,
"body": "GIF87a\u0001\u0000\u0001\u0000\ufffd\u0002\u0000\u0000\u0000\u0000\ufffd\ufffd\ufffd,\u0000\u0000\u0000\u0000\u0001\u0000\u0001\u0000\u0000\u0002\u0002L\u0001\u0000;",
"isBase64Encoded": false
}
As you can see from the body element in the preceding JSON response, the Lambda function has received a string containing several UTF-8 characters. There is no clear way to correctly decode that payload back into the original binary image data. You may have noticed the flag isBase64Encoded that is included in the request data that gets echoed back. That flag holds the key to solving the problem of receiving binary payloads.
When creating an API Gateway API, you can specify that certain content types should be treated as binary. When you do this, you’re telling API Gateway to encode that binary data using base64 encoding, which can safely be decoded by the receiving function. You can find this under Settings in the API Gateway console.
If you add image/gif as a binary media type to the API and retry the preceding command, you get the following response:
curl --data-binary "@1pixel.gif" http://abcdef.execute-api.us-east-1.amazonaws.com/Prod/
{
// Some fields removed for brevity
"path": "/",
"httpMethod": "POST",
"headers": {
"content-type": "image/gif",
},
"queryStringParameters": null,
"body": "R0lGODdhAQABAIACAAAAAP///ywAAAAAAQABAAACAkwBADs=",
"isBase64Encoded": true
}
You see in the preceding code that the isBase64Encoded flag is set to true and that the body looks different. In fact, the body is now a correct base64-encoding of the 1×1 pixel GIF image.
You could continue to add MIME types to the API to cover all of the options required. However, because you want the application code to be able to handle arbitrary content types, you can instead use a wildcard content type: */*.
Returning binary responses
Now that you have dealt with all of the requirements regarding incoming HTTP requests, check that you can meet the requirements for sending custom responses.
From the API Gateway documentation, all of your requirements are met by using the Lambda proxy integration’s output model:
{
"isBase64Encoded": true|false,
"statusCode": httpStatusCode,
"headers": { "headerName": "headerValue", ... },
"multiValueHeaders": { "headerName": ["headerValue", "headerValue2", ...], ... },
"body": "..."
}
You can send any HTTP status code by setting the statusCode field, any headers required by setting either headers or multiValueHeaders, and any response payload by populating the body field. If the response payload contains binary data, you can base64-encode the data and set the isBase64Encoded field to true.
To demonstrate these features, modify the Lambda function so that the response returned by the API echoes the original HTTP request that was sent to it, rather than echoing the integration request as before. Here is the Python code:
import json
def handler(event, _):
return {
"statusCode": 200,
"headers": {
"Content-Type": event["headers"]["content-type"],
},
"body": event["body"],
"isBase64Encoded": event["isBase64Encoded"],
}
For example, if you send a 1×1 pixel GIF to the API now, you receive a response containing an exact copy of the same image with the same Content-Type header.
Putting it all together
Now that you have everything working as intended, the final stage is to wrap everything in AWS CloudFormation so that you have repeatable deployments. Use the AWS Serverless Application Model (AWS SAM) because it contains useful resource types. That means that the AWS CloudFormation template can be more expressive and readable.
First, define the Lambda function:
Function:
Type: "AWS::Serverless::Function"
Properties:
CodeUri: echo.py
Handler: echo.handler
Runtime: python3.6
Events:
Base:
Type: Api
Properties:
Method: any
Path: /
RestApiId: !Ref Api
Others:
Type: Api
Properties:
Method: any
Path: /{proxy+}
RestApiId: !Ref Api
The Events section defines URL paths that map to the API definition that you add next:
Api:
Type: "AWS::Serverless::Api"
Properties:
StageName: Prod
BinaryMediaTypes:
- "~1"
DefinitionBody:
openapi: "3.0"
info:
title: !Ref "AWS::StackName"
version: "1.0"
paths:
/:
x-amazon-apigateway-any-method:
responses:
{}
x-amazon-apigateway-integration:
httpMethod: POST
type: aws_proxy
uri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Function.Arn}/invocations"
/{proxy+}:
x-amazon-apigateway-any-method:
responses:
{}
x-amazon-apigateway-integration:
httpMethod: POST
type: aws_proxy
uri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Function.Arn}/invocations"
There are a couple of things to note here. First, the BinaryMediaTypes property contains *~1*, which is an encoding required by AWS SAM that translates into the wildcard */* string. Secondly, you replaced the uri properties within the embedded OpenAPI document to use the !Sub function. This is so that you can avoid hardcoding the Lambda function ARN.
Last of all, add an Outputs section to the CloudFormation template so that you can easily access the URL of the API Gateway endpoint.
Outputs:
Endpoint:
Value: !Sub "https://${Api}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
When you deploy this template, AWS SAM translates the two resources (the Lambda function and API) into all of the AWS resources required:
You can see the full AWS CloudFormation template along with the code for the Lambda function and instructions for deploying it to your account in amazon-api-gateway-custom-request-response GitHub repo.
Summary
To recap, here is a list of the configurations required to create an API Gateway API that can send and receive arbitrary HTTP requests and responses through a Lambda function:
- Use the
ANYmethod to capture all request types (such asGET,POST, andHEAD). - Create two paths:
- One for the root URL:
/ - One for every other URL:
/{proxy+}
- One for the root URL:
- Set the binary media types setting to
*/*. - In your Lambda function, take the following steps:
- If
isBase64Encodedis true, base64-decode thebodyelement of the request. - To specify the HTTP response code, set the
statusCodeelement of the response object. - Add any custom headers to the
headersobject of the response object. - To send a binary response, base64-encoded the
bodyelement of the response and setisBase64Encodedto true.
- If