Using API Gateway with VPC endpoints via AWS Lambda

To isolate critical parts of their app’s architecture, customers often rely on Virtual Private Cloud (VPC) and private subnets. Today, Amazon API Gateway cannot directly integrate with endpoints that live within a VPC without internet access. However, it is possible to proxy calls to your VPC endpoints using AWS Lambda functions.

This post guides you through the setup necessary to configure API Gateway, Lambda, and your VPC to proxy requests from API Gateway to HTTP endpoints in your VPC private subnets. With this solution, you can use API Gateway for authentication, authorization, and throttling before a request reaches your HTTP endpoint.

For this example, we have written a very basic Express application that receives a GET and POST method on its root resource (“/”). The application is deployed on an EC2 instance within a private subnet of a VPC. We use a Lambda function that connects to our private subnet to proxy requests from API Gateway to the Express HTTP endpoint. The CloudFormation template below deploys the API Gateway API, the AWS Lambda functions, and sets the correct permissions on both resources. Our CloudFormation template requires 4 parameters:

The IP address or DNS name of the instance running your express application (for example, 10.0.1.16)
The port used by the Express app (for example, 8080)
The security group of the EC2 instance (for example, sg-xx3xx6x0)
The subnet ID of your VPC’s private subnet (for example, subnet-669xx03x)

Click the link below to deploy the CloudFormation template, the rest of this blog post dives deeper on each component of the architecture.

The Express application

We have written a very simple web service using Express and Node.js. The service accepts GET and POST requests to its root resource and responds with a JSON object. You can use the sample code below to start the application on your instance. Before you create the application, make sure that you have installed Node.js on your instance.

Create a new folder on your web server called vpcproxy. In the new folder, create a new file called index.js and paste the code below in the file.

var express = require('express');
var bodyParser = require('body-parser');

var app = express();
app.use(bodyParser.json());

app.get('/', function(req, res) {
        if (req.query.error) {
                res.status(403).json({error: "Random error"}).end();
                return;
        }
        res.json({ message: 'Hello World!' });
});

app.post('/', function(req, res) {
        console.log("post");
        console.log(req.body);
        res.json(req.body).end();
});
app.listen(8080, function() {
        console.log("app started");
});

To install the required dependencies, from the vpcproxy folder, run the following command: npm install express body-parser

After the dependencies are installed, you can start the application by running: node index.js

API Gateway configuration

The API Gateway API declares all of the same methods that your Express application supports. Each method is configured to transform requests into a JSON structure that AWS Lambda can understand, and responses are generated using mapping templates from the Lambda output.

The first step is to transform a request into an event for Lambda. The mapping template below captures all of the request information and includes the configuration of the backend endpoint that the Lambda function should interact with. This template is applied to all requests for any endpoint.

#set($allParams = $input.params())
{
  "requestParams" : {
    "hostname" : "10.0.1.16",
    "port" : "8080",
    "path" : "$context.resourcePath",
    "method" : "$context.httpMethod"
  },
  "bodyJson" : $input.json('$'),
  "params" : {
    #foreach($type in $allParams.keySet())
      #set($params = $allParams.get($type))
      "$type" : {
        #foreach($paramName in $params.keySet())
          "$paramName" : "$util.escapeJavaScript($params.get($paramName))"
          #if($foreach.hasNext),#end
        #end
      }
      #if($foreach.hasNext),#end
    #end
  },
  "stage-variables" : {
    #foreach($key in $stageVariables.keySet())
      "$key" : "$util.escapeJavaScript($stageVariables.get($key))"
      #if($foreach.hasNext),#end
    #end
  },
  "context" : {
    "account-id" : "$context.identity.accountId",
    "api-id" : "$context.apiId",
    "api-key" : "$context.identity.apiKey",
    "authorizer-principal-id" : "$context.authorizer.principalId",
    "caller" : "$context.identity.caller",
    "cognito-authentication-provider" : "$context.identity.cognitoAuthenticationProvider",
    "cognito-authentication-type" : "$context.identity.cognitoAuthenticationType",
    "cognito-identity-id" : "$context.identity.cognitoIdentityId",
    "cognito-identity-pool-id" : "$context.identity.cognitoIdentityPoolId",
    "http-method" : "$context.httpMethod",
    "stage" : "$context.stage",
    "source-ip" : "$context.identity.sourceIp",
    "user" : "$context.identity.user",
    "user-agent" : "$context.identity.userAgent",
    "user-arn" : "$context.identity.userArn",
    "request-id" : "$context.requestId",
    "resource-id" : "$context.resourceId",
    "resource-path" : "$context.resourcePath"
  }
}

After the Lambda function has processed the request and response, API Gateway is configured to transform the output into an HTTP response. The output from the Lambda function is a JSON structure that contains the response status code, body, and headers:

{  
   "status":200,
   "bodyJson":{  
      "message":"Hello World!"
   },
   "headers":{  
      "x-powered-by":"Express",
      "content-type":"application/json; charset=utf-8",
      "content-length":"26",
      "etag":"W/\"1a-r2dz039gtg5rjLoq32eF4w\"",
      "date":"Wed, 25 May 2016 18:41:22 GMT",
      "connection":"keep-alive"
   }
}

These values are then mapped in API Gateway using header mapping expressions and mapping templates for the response body.

First, all known headers are mapped:

"responseParameters": {
    "method.response.header.etag": "integration.response.body.headers.etag",
    "method.response.header.x-powered-by": "integration.response.body.headers.x-powered-by",
    "method.response.header.date": "integration.response.body.headers.date",
    "method.response.header.content-length": "integration.response.body.headers.content-length"
}

Then the body is extracted from Lambda’s output JSON using a very simple body mapping template: $input.json('$.bodyJson')

Response codes other than 200 are handled using using regular expressions to match the status code in API Gateway (for example \{\"status\"\:400.*), and the parseJson method of the $util object to extract the response body.

#set ($errorMessageObj = $util.parseJson($input.path('$.errorMessage')))
$errorMessageObj.bodyJson

All of this configuration is included in Swagger format in the CloudFormation template of this tutorial. The Swagger is generated dynamically based on the four parameters that the template requires using the Fn::Join function.

The AWS Lambda function

The proxy Lambda function is written in JavaScript and captures all of the request details forwarded by API Gateway, creates similar request using the standard Node.js http package, and forwards it to the private endpoint. Responses from the private endpoint are encapsulated in a JSON object which API Gateway turns into an HTTP response. The private endpoint configuration is passed to the Lambda function from API Gateway in the event model. The Lambda function code is also included in the CloudFormation template.

var http = require('http');

exports.myHandler = function(event, context, callback) {
    // setup request options and parameters
    var options = {
      host: event.requestParams.hostname,
      port: event.requestParams.port,
      path: event.requestParams.path,
      method: event.requestParams.method
    };
    
    // if you have headers set them otherwise set the property to an empty map
    if (event.params && event.params.header && Object.keys(event.params.header).length > 0) {
        options.headers = event.params.header
    } else {
        options.headers = {};
    }
    
    // Force the user agent and the "forwaded for" headers because we want to 
    // take them from the API Gateway context rather than letting Node.js set the Lambda ones
    options.headers["User-Agent"] = event.context["user-agent"];
    options.headers["X-Forwarded-For"] = event.context["source-ip"];
    // if I don't have a content type I force it application/json
    // Test invoke in the API Gateway console does not pass a value
    if (!options.headers["Content-Type"]) {
        options.headers["Content-Type"] = "application/json";
    }
    // build the query string
    if (event.params && event.params.querystring && Object.keys(event.params.querystring).length > 0) {
        var queryString = generateQueryString(event.params.querystring);
        
        if (queryString !== "") {
            options.path += "?" + queryString;
        }
    }
    
    // Define my callback to read the response and generate a JSON output for API Gateway.
    // The JSON output is parsed by the mapping templates
    callback = function(response) {
        var responseString = '';
    
        // Another chunk of data has been recieved, so append it to `str`
        response.on('data', function (chunk) {
            responseString += chunk;
        });
      
        // The whole response has been received
        response.on('end', function () {
            // Parse response to json
            var jsonResponse = JSON.parse(responseString);
    
            var output = {
                status: response.statusCode,
                bodyJson: jsonResponse,
                headers: response.headers
            };
            
            // if the response was a 200 we can just pass the entire JSON back to
            // API Gateway for parsing. If the backend returned a non 200 status 
            // then we return it as an error
            if (response.statusCode == 200) {
                context.succeed(output);
            } else {
                // set the output JSON as a string inside the body property
                output.bodyJson = responseString;
                // stringify the whole thing again so that we can read it with 
                // the $util.parseJson method in the mapping templates
                context.fail(JSON.stringify(output));
            }
        });
    }
    
    var req = http.request(options, callback);
    
    if (event.bodyJson && event.bodyJson !== "") {
        req.write(JSON.stringify(event.bodyJson));
    }
    
    req.on('error', function(e) {
        console.log('problem with request: ' + e.message);
        context.fail(JSON.stringify({
            status: 500,
            bodyJson: JSON.stringify({ message: "Internal server error" })
        }));
    });
    
    req.end();
}

function generateQueryString(params) {
    var str = [];
    for(var p in params) {
        if (params.hasOwnProperty(p)) {
            str.push(encodeURIComponent(p) + "=" + encodeURIComponent(params[p]));
        }
    }
    return str.join("&");
}

Conclusion

You can use Lambda functions to proxy HTTP requests from API Gateway to an HTTP endpoint within a VPC without Internet access. This allows you to keep your EC2 instances and applications completely isolated from the internet while still exposing them via API Gateway. By using API Gateway to front your existing endpoints, you can configure authentication and authorization rules as well as throttling rules to limit the traffic that your backend receives.

If you have any questions or suggestions, please comment below.

AWS Compute Blog

Using API Gateway with VPC endpoints via AWS Lambda

The Express application

API Gateway configuration

The AWS Lambda function

Conclusion

Resources

Follow