Running Arbitrary Executables in AWS Lambda

In previous posts on this blog we’ve talked about how Lambda manages container lifetimes and how you can use custom JavaScript libraries in Lambda, even native ones. In this post I cover how you can run arbitrary executables, including languages like Python, and shell scripts like bash.

First, a note on security: Lambda’s built-in sandbox lets you safely run code written in any language, because Lambda doesn’t rely on the language runtime to provide isolation between functions. You get the same protections regardless of whether Lambda starts a process for you or whether you start it yourself, and regardless of the language in which it’s written. With that out of the way, let’s look at how easy it is:

To start a background process, you can use the child_process command in Node.js to execute a binary that you’ve uploaded with your function or any executable visible to your function, such as /bin/bash or /usr/bin/python. Node.js supports both child_process.spawn, which follows the common async programming pattern by accepting a callback or returning an EventEmitter, as well as child_process.spawnSync, which waits for the spawned process to exit before returning control to your code.

Including your own executables is easy; just package them in the ZIP file you upload, and then reference them (including the relative path within the ZIP file you created) when you call them from Node.js or from other processes that you’ve previously started. Ensure that you include the following at the start of your function code:

process.env[‘PATH’] = process.env[‘PATH’] + ‘:’ + process.env[‘LAMBDA_TASK_ROOT’]

You can use all the usual forms of interprocess communication as well as files in /tmp to communicate with any of the processes you create.

If you compile your own binaries, ensure that they’re either statically linked or built for the matching version of Amazon Linux. The current version of Amazon Linux in use within AWS Lambda can always be found on the Supported Versions page of the Lambda docs.

For scenarios like running a Python script, you may find it easiest to simply run the executable once for each request. If you did old school-style CGI programming, this is a modern twist on it.

Other executables might want to retain state or may take too long to initialize to restart each time. Fortunately, Lambda has you covered there as well: Processes running when your request ends will be frozen and (if your container is reused to serve a later request) thawed out again next time, so you don’t have to restart long-running background processes repeatedly. The prologue section of your JavaScript function is a good place to locate such “once per container” activities. See Understanding Container Reuse for a fuller description of container and process reuse.

AWS Lambda offers you lots of flexibility, without the hassle (or lockin) of curated libraries or platform-specific language runtimes that differ from standard releases. Bring your own code, we’re happy to run it for you!