AWS Compute Blog

Going Serverless: Migrating an Express Application to Amazon API Gateway and AWS Lambda

Brett Andrews
Brett Andrews
Software Development Engineer

Amazon API Gateway recently released three new features that simplify the process of forwarding HTTP requests to your integration endpoint: greedy path variables, the ANY method, and proxy integration types. With this new functionality, it becomes incredibly easy to run HTTP applications in a serverless environment by leveraging the aws-serverless-express library.

In this post, I go through the process of porting an “existing” Node.js Express application onto API Gateway and AWS Lambda, and discuss some of the advantages, disadvantages, and current limitations. While I use Express in this post, the steps are similar for other Node.js frameworks, such as Koa, Hapi, vanilla, etc.

Modifying an existing Express application

Express is commonly used for both web applications as well as REST APIs. While the primary API Gateway function is to deliver APIs, it can certainly be used for delivering web apps/sites (HTML) as well. To cover both use cases, the Express app below exposes a web app on the root / resource, and a REST API on the /pets resource.

The goal of this walkthrough is for it to be complex enough to cover many of the limitations of this approach today (as comments in the code below), yet simple enough to follow along. To this end, you implement just the entry point of the Express application (commonly named app.js) and assume standard implementations of views and controllers (which are more insulated and thus less affected). You also use MongoDB, due to it being a popular choice in the Node.js community as well as providing a time-out edge case. For a greater AWS serverless experience, consider adopting Amazon DynamoDB.

//app.js
'use strict'
const path = require('path')
const express = require('express')
const bodyParser = require('body-parser')
const cors = require('cors')
const mongoose = require('mongoose')
// const session = require('express-session')
// const compress = require('compression')
// const sass = require('node-sass-middleware')

// Lambda does not allow you to configure environment variables, but dotenv is an
// excellent and simple solution, with the added benefit of allowing you to easily
// manage different environment variables per stage, developer, environment, etc.
require('dotenv').config()

const app = express()
const homeController = require('./controllers/home')
const petsController = require('./controllers/pets')

// MongoDB has a default timeout of 30s, which is the same timeout as API Gateway.
// Because API Gateway was initiated first, it also times out first. Reduce the
// timeout and kill the process so that the next request attempts to connect.
mongoose.connect(process.env.MONGODB_URI, { server: { socketOptions: { connectTimeoutMS: 10000 } } })
mongoose.connection.on('error', () => {
   console.error('Error connecting to MongoDB.')
   process.exit(1)
})

app.set('views', path.join(__dirname, 'views'))
app.set('view engine', 'pug')
app.use(cors())
app.use(bodyParser.json())
app.use(bodyParser.urlencoded({ extended: true }))

/*
* GZIP support is currently not available to API Gateway.
app.use(compress())

* node-sass is a native binary/library (aka Addon in Node.js) and thus must be
* compiled in the same environment (operating system) in which it will be run.
* If you absolutely need to use a native library, you can set up an Amazon EC2 instance
* running Amazon Linux for packaging your Lambda function.
* In the case of SASS, I recommend to build your CSS locally instead and
* deploy all static assets to Amazon S3 for improved performance.
const publicPath = path.join(__dirname, 'public')
app.use(sass({ src: publicPath, dest: publicPath, sourceMap: true}))
app.use(express.static(publicPath, { maxAge: 31557600000 }))

* Storing local state is unreliable due to automatic scaling. Consider going stateless (using REST),
* or use an external state store (for MongoDB, you can use the connect-mongo package)
app.use(session({ secret: process.env.SESSION_SECRET }))
*/

app.get('/', homeController.index)
app.get('/pets', petsController.listPets)
app.post('/pets', petsController.createPet)
app.get('/pets/:petId', petsController.getPet)
app.put('/pets/:petId', petsController.updatePet)
app.delete('/pets/:petId', petsController.deletePet)

/*
* aws-serverless-express communicates over a Unix domain socket. While it's not required
* to remove this line, I recommend doing so as it just sits idle.
app.listen(3000)
*/

// Export your Express configuration so that it can be consumed by the Lambda handler
module.exports = app

Assuming that you had the relevant code implemented in your views and controllers directories and a MongoDB server available, you could uncomment the listen line, run node app.js and have an Express application running at http://localhost:3000. The following “changes” made above were specific to API Gateway and Lambda:

  • Used dotenv to set environment variables.
  • Reduced the timeout for connecting to DynamoDB so that API Gateway does not time out first.
  • Removed the compression middleware as API Gateway does not (currently) support GZIP.
  • Removed node-sass-middleware (I opted for serving static assets through S3, but if there is a particular native library your application absolutely needs, you can build/package your Lambda function on an EC2 instance).
  • Served static assets through S3/CloudFront. Not only is S3 a better option for static assets for performance reasons, API Gateway does not currently support binary data (e.g., images).
  • Removed session state for scalability (alternatively, you could have stored session state in MongoDB using connect-mongo.
  • Removed app.listen() as HTTP requests are not being sent over ports (not strictly required).
  • Exported the Express configuration so that you can consume it in your Lambda handler (more on this soon).

Going serverless with aws-serverless-express

In order for users to be able to hit the app (or for developers to consume the API), you first need to get it online. Because this app is going to be immensely popular, you obviously need to consider scalability, resiliency, and many other factors. Previously, you could provision some servers, launch them in multiple Availability Zones, configure Auto Scaling policies, ensure that the servers were healthy (and replace them if they weren’t), keep up-to-date with the latest security updates, and so on…. As a developer, you just care that your users are able to use the product; everything else is a distraction.

Enter serverless. By leveraging AWS services such as API Gateway and Lambda, you have zero servers to manage, automatic scaling out-of-the-box, and true pay-as-you-go: the promise of the cloud, finally delivered.

The example included in the aws-serverless-express library library includes a good starting point for deploying and managing your serverless resources.

  1. Clone the library into a local directory git clone https://github.com/awslabs/aws-serverless-express.git
  2. From within the example directory, run npm run config <accountId> <bucketName> [region] (this modifies some of the files with your own settings).
  3. Edit the package-function command in package.json by removing “index.html” and adding “views” and “controllers” (the additional directories required for running your app).

  4. Copy the following files in the example directory into your existing project’s directory:

    • simple-proxy-api.yaml – A Swagger file that describes your API.
    • cloudformation.json – A CloudFormation template for creating the Lambda function and API.
    • package.json – You may already have a version of this file, in which case just copy the scripts and config sections. This includes some helpful npm scripts for managing your AWS resources, and testing and updating your Lambda function.
    • api-gateway-event.json – Used for testing your Lambda function locally.
    • lambda.js – The Lambda function, a thin wrapper around your Express application.

    Take a quick look at lambda.js so that you understand exactly what’s going on there. The aws-serverless-express library transforms the request from the client (via API Gateway) into a standard Node.js HTTP request object; sends this request to a special listener (a Unix domain socket); and then transforms it back for the response to API Gateway. It also starts your Express server listening on the Unix domain socket on the initial invocation of the Lambda function. Here it is in its entirety:

// lambda.js
'use strict'
const awsServerlessExpress = require('aws-serverless-express')
const app = require('./app')
const server = awsServerlessExpress.createServer(app)

exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context)

TIP: Everything outside of the handler function is executed only one time per container: that is, the first time your app receives a request (or the first request after several minutes of inactivity), and when it scales up additional containers.

Deploying

Now that you have more of an understanding of how API Gateway and Lambda communicate with your Express server, it’s time to release your app to the world.

From the project’s directory, run:

npm run setup

This command creates the Amazon S3 bucket specified earlier (if it does not yet exist); zips the necessary files and directories for your Lambda function and uploads it to S3; uploads simple-proxy-api.yaml to S3; creates the CloudFormation stack; and finally opens your browser to the AWS CloudFormation console where you can monitor the creation of your resources. To clean up the AWS resources created by this command, simply run npm run delete-stack. Additionally, if you specified a new S3 bucket, run npm run delete-bucket.

After the status changes to CREATE_COMPLETE (usually after a couple of minutes), you see three links in the Outputs section: one to the API Gateway console, another to the Lambda console, and most importantly one for your web app/REST API. Clicking the link to your API displays the web app; appending /pets in the browser address bar displays your list of pets. Your Express application is available online with automatic scalability and pay-per-request without having to manage a single server!

Additional features

Now that you have your REST API available to your users, take a quick look at some of the additional features made available by API Gateway and Lambda:

  • Usage plans for monetizing the API
  • Caching to improve performance
  • Authorizers for authentication and authorization microservices that determine access to your Express application
  • Stages and versioning and aliases when you need additional stages or environments (dev, beta, prod, etc.)
  • SDK generation to provide SDKs to consumers of your API (available in JavaScript, iOS, Android Java, and Android Swift)
  • API monitoring for logs and insights into usage

After running your Express application in a serverless environment for a while and learning more about the best practices, you may start to want more: more performance, more control, more microservices!

So how do you take your existing serverless Express application (a single Lambda function) and refactor it into microservices? You strangle it. Take a route, move the logic to a new Lambda function, and add a new resource or method to your API Gateway API. However, you’ll find that the tools provided to you by the aws-serverless-express example just don’t cut it for managing these additional functions. For that, you should check out Claudia; it even has support for aws-serverless-express.

Conclusion

To sum it all up, you took an existing application of moderate complexity, applied some minimal changes, and deployed it in just a couple of commands. You now have no servers to manage, automatic scaling out-of-the-box, true pay-as-you-go, loads of features provided by API Gateway, and as a bonus, a great path forward for refactoring into microservices.

If that’s not enough, or this server-side stuff doesn’t interest you and the front end is where you live, consider using aws-serverless-express for server-side rendering of your single page application.

If you have questions or suggestions, please comment below.