Using Packages and Native nodejs Modules in AWS Lambda
Tim Wagner, AWS Lambda
Bryan Liston, AWS Solutions Architect
In this post we take a look at how to use custom nodejs packages with AWS Lambda, including building and packaging native nodejs modules for use in your Lambda functions. To do the steps below, you’ll need an EC2 instance or a similar machine running Amazon Linux with nodejs installed.
Using npm packages and custom modules/packages with Lambda is easy. We’ll start by including prebuilt modules then move on to native ones.
Step1: Create a new directory to hold your Lambda function and its modules.
$ mkdir lambdaTestFunction $ cd lambdaTestFunction
Step 2: Install an npm package
$ npm install --prefix=~/lambdaTestFunction aws-sdk firstname.lastname@example.org node_modules/aws-sdk ├── email@example.com └── firstname.lastname@example.org (email@example.com) $ ls node_modules aws-sdk
At this point we’ve installed the ‘aws-sdk’ module into the node_modules directory of our lambdaTestFunction folder.
Step 3: Verification
Next we’ll run nodejs locally to make sure we have a valid configuration before proceeding. We’ll create a trivial test.js file that uses the AWS SDK to print some EC2 config data. It’s not interesting in its own right, but it serves to show that the SDK has been installed.
$ echo 'var AWS = require("aws-sdk");console.log(AWS.EC2.apiVersions)'> test.js $ node test.js [ '2013-06-15*','2013-10-15*','2014-02-01*','2014-05-01*','2014-06-15*','2014-09-01*','2014-10-01' ]
Step 4: Create Your Lambda Function!
At this point we’ve successfully created a directory containing one or more npm-installed packages and verified that the packages can load and execute by running a test script locally. You can now delete the test script and continue by creating a real Lambda function that takes advantage of the modules that you’ve just installed, testing it the same way. To deploy the resulting function and modules to Lambda, just zip up the entire lambdaTestFunction directory and use Lambda’s createFunction API, CLI, or the console UI to deploy it.
Everything you learned above for npm-installing existing modules also applies to custom modules that you create yourself. We’ll illustrate with a trivial example that just logs to the console. Create a directory under node_modules that contains your custom written packages, and add them to your function, just as you did previously for npm-installed packages:
$ echo 'console.log("This is a demo")' > node_modules/bryan/index.js $ echo 'require("bryan")' > test.js $ node test.js This is a demo
Native modules are similarly installed and deployed, but you’ll need to build them against the Amazon Linux libraries. You’ll need to either ensure that the libraries and their transitive dependencies are statically compiled or use rpath-style linking; we’ll do it the static way in this post, and demonstrate use of rpath in a subsequent post. (Note that many, but not all, libraries can be statically linked this way.)
The following steps will walk you through updating your system, installing the required development libraries and tools, downloading nodejs and our sample library, OpenCV, and finally installing and testing the OpenCV module we create by running some basic facial detection code on a well-known face.
Step 1: Update your Amazon Linux machine and install required libraries and tools
$ sudo yum update $ sudo yum install gcc44 gcc-c++ libgcc44 cmake –y $ wget http://nodejs.org/dist/v0.10.33/node-v0.10.33.tar.gz $ tar -zxvf node-v0.10.33.tar.gz $ cd node-v0.10.33 && ./configure && make $ sudo make install
Step 2: Download your native module
$ wget http://softlayer-dal.dl.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.9/opencv-2.4.9.zip $ mkdir opencv_install $ mkdir opencv_example $ unzip opencv-2.4.9.zip –d opencv_install/ && cd opencv_install
Step 3: Configure your module for static libraries
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_SHARED_LIBS=NO -D CMAKE_INSTALL_PREFIX=~/opencv opencv-2.4.9/
Note that we’re disabling shared libraries as we build OpenCV; you can see this in the output:
-- C/C++: -- Built as dynamic libs?: NO -- Install path: /home/ec2-user/opencv
Step 4: Build and install your module
$ make && make install
Step 5: Install the package using npm
Now that we’ve built the OpenCV library to a local folder, we’ll install the npm module to our local lambda example folder, opencv_example.
$ PKG_CONFIG_PATH=~/opencv/lib/pkgconfig/ npm install –prefix=~/opencv_example opencv
We specify a prefix of PKG_CONFIG_PATH before we run npm install to include our new static libraries into the local path. NPM will compile the OpenCV module using the statically compiled version of the OpenCV library we built above.
Step 6: Testing our newly compiled Native Module
The NodeJS OpenCV module includes some sample facial detection code that we can use to validate that the module has been built correctly:
$ cd ~/opencv_example $ cd node_modules/opencv/examples/ $ mkdir tmp $ node face-detection.js Image saved to ./tmp/face-detection.png $ ls tmp face-detection.png
Now if we look at the newly created file we should see a red circle placed around Mona Lisa’s head:
Step 7: Deploying to Lambda
You’ve now successfully compiled a native NodeJS module and tested it using OpenCV’s test code. You can remove any test files and test output, write a real Lambda function, ZIP up your directory as before and deploy it to Lambda. If you’ve been following along with our actual example above, you could try this out in Lambda by using OpenCV to do facial recognition on images added to an Amazon S3 bucket.
-Bryan and Tim