Using Packages and Native nodejs Modules in AWS Lambda

Tim Wagner, AWS Lambda


Bryan Liston, AWS Solutions Architect


In this post we take a look at how to use custom nodejs packages with AWS Lambda, including building and packaging native nodejs modules for use in your Lambda functions. To do the steps below, you’ll need an EC2 instance or a similar machine running Amazon Linux with nodejs installed.

Using Existing JavaScript Packages with Lambda

Using npm packages and custom modules/packages with Lambda is easy. We’ll start by including prebuilt modules then move on to native ones.

Step1: Create a new directory to hold your Lambda function and its modules.

$ mkdir lambdaTestFunction
$ cd lambdaTestFunction

Step 2: Install an npm package

For this example, we’ll keep things simple and install the AWS SDK. This is just for illustration; the current version of the AWS SDK is pre-installed in Lambda, but you could use this technique to load other pre-built JavaScript packages or if you actually needed an earlier version of the AWS SDK for compatibility reasons.

$ npm install --prefix=~/lambdaTestFunction aws-sdk
aws-sdk@2.0.27 node_modules/aws-sdk
├── xmlbuilder@0.4.2
└── xml2js@0.2.6 (sax@0.4.2)
$ ls node_modules
aws-sdk

At this point we’ve installed the ‘aws-sdk’ module into the node_modules directory of our lambdaTestFunction folder.

Step 3: Verification

Next we’ll run nodejs locally to make sure we have a valid configuration before proceeding. We’ll create a trivial test.js file that uses the AWS SDK to print some EC2 config data. It’s not interesting in its own right, but it serves to show that the SDK has been installed.

$ echo 'var AWS = require("aws-sdk");console.log(AWS.EC2.apiVersions)'> test.js
$ node test.js
[ '2013-06-15*','2013-10-15*','2014-02-01*','2014-05-01*','2014-06-15*','2014-09-01*','2014-10-01' ]

Step 4: Create Your Lambda Function!

At this point we’ve successfully created a directory containing one or more npm-installed packages and verified that the packages can load and execute by running a test script locally. You can now delete the test script and continue by creating a real Lambda function that takes advantage of the modules that you’ve just installed, testing it the same way. To deploy the resulting function and modules to Lambda, just zip up the entire lambdaTestFunction directory and use Lambda’s createFunction API, CLI, or the console UI to deploy it.

Custom JavaScript Modules

Everything you learned above for npm-installing existing modules also applies to custom modules that you create yourself. We’ll illustrate with a trivial example that just logs to the console. Create a directory under node_modules that contains your custom written packages, and add them to your function, just as you did previously for npm-installed packages:

$ echo 'console.log("This is a demo")' > node_modules/bryan/index.js
$ echo 'require("bryan")' > test.js
$ node test.js
This is a demo

Native Modules

Native modules are similarly installed and deployed, but you’ll need to build them against the Amazon Linux libraries. You’ll need to either ensure that the libraries and their transitive dependencies are statically compiled or use rpath-style linking; we’ll do it the static way in this post, and demonstrate use of rpath in a subsequent post. (Note that many, but not all, libraries can be statically linked this way.)

The following steps will walk you through updating your system, installing the required development libraries and tools, downloading nodejs and our sample library, OpenCV, and finally installing and testing the OpenCV module we create by running some basic facial detection code on a well-known face.

Step 1: Update your Amazon Linux machine and install required libraries and tools

$ sudo yum update
$ sudo yum install gcc44 gcc-c++ libgcc44 cmake –y
$ wget http://nodejs.org/dist/v0.10.33/node-v0.10.33.tar.gz
$ tar -zxvf node-v0.10.33.tar.gz
$ cd node-v0.10.33 && ./configure && make
$ sudo make install

Step 2: Download your native module

$ wget http://softlayer-dal.dl.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.9/opencv-2.4.9.zip
$ mkdir opencv_install
$ mkdir opencv_example
$ unzip opencv-2.4.9.zip –d opencv_install/ && cd opencv_install

Step 3: Configure your module for static libraries

$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_SHARED_LIBS=NO -D CMAKE_INSTALL_PREFIX=~/opencv opencv-2.4.9/

Note that we’re disabling shared libraries as we build OpenCV; you can see this in the output:

--   C/C++:
--   Built as dynamic libs?:     NO
--   Install path:               /home/ec2-user/opencv

Step 4: Build and install your module

$ make && make install

Step 5: Install the package using npm

Now that we’ve built the OpenCV library to a local folder, we’ll install the npm module to our local lambda example folder, opencv_example.

$ PKG_CONFIG_PATH=~/opencv/lib/pkgconfig/  npm install –prefix=~/opencv_example opencv

We specify a prefix of PKG_CONFIG_PATH before we run npm install to include our new static libraries into the local path. NPM will compile the OpenCV module using the statically compiled version of the OpenCV library we built above.

Step 6: Testing our newly compiled Native Module

The NodeJS OpenCV module includes some sample facial detection code that we can use to validate that the module has been built correctly:

$ cd ~/opencv_example
$ cd node_modules/opencv/examples/
$ mkdir tmp
$ node face-detection.js
Image saved to ./tmp/face-detection.png
$ ls tmp
face-detection.png

Now if we look at the newly created file we should see a red circle placed around Mona Lisa’s head:

Mona Lisa, after facial recognition analysis

Step 7: Deploying to Lambda

You’ve now successfully compiled a native NodeJS module and tested it using OpenCV’s test code. You can remove any test files and test output, write a real Lambda function, ZIP up your directory as before and deploy it to Lambda. If you’ve been following along with our actual example above, you could try this out in Lambda by using OpenCV to do facial recognition on images added to an Amazon S3 bucket.

-Bryan and Tim