How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?

3 minute read
1

I used pip to install a Python package that contains compiled code, and now my AWS Lambda function returns an "Unable to import module" error.

Short description

Python packages that contain compiled code, such as NumPy and pandas, aren't always compatible with Lambda runtimes by default. If you install these packages using pip, then the packages download and compile a module-name package for the architecture of the local machine. If you don't use a Linux operating system (OS), then this makes your deployment package incompatible with Lambda Python runtimes.

To make your deployment package or layer compatible with Lambda on a non-Linux OS, run the pip install command with manylinux2014 as the value for the --platform parameter.

Note: macOS --platform tags don't work. For example, the win_amd64 and macosx_10_6_intel tags don't install a deployment package that's compatible with Lambda.

Resolution

Note: The following example procedure shows how to install pandas for the Lambda Python 3.9 runtime that runs on x86_64 architecture.

  1. Open a command prompt. Then, run the following pip command to confirm that you're using a version of pip that's version 19.3.0 or newer:

    pip --version

    If you're using a version of pip that's older than pip version 19.3.0, then upgrade to the latest version of pip:

    python3.9 -m pip install --upgrade pip
  2. Install the precompiled Python package's .whl file as a dependency in your Lambda function's project directory:

    Important: Replace my-lambda-function with the name of your function's project directory.

    pip install \    
        --platform manylinux2014_x86_64 \
        --target=my-lambda-function \
        --implementation cp \
        --python-version 3.9 \
        --only-binary=:all: --upgrade \
        pandas
  3. Open your Lambda function's project directory. If you use macOS, then run the following command:

    cd my-lambda-function
  4. In a text editor, create a new file named lambda_function.py. Then, copy and paste the following example code into the file:

    import numpy as np
    import pandas as pd
    def lambda_handler(event, context):
        df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),columns=["a", "b", "c"])
        number = np.pi
        print(df2)
        print(number)

    Save this file in your Lambda function's project directory.

  5. Create a Lambda deployment package .zip file archive that includes all of the installed libraries and source code:

    zip -r ../my-deployment-package.zip .

    Use the my-deployment-package.zip file archive to either create a new Python 3.9 Lambda function or to update an existing one. For instructions, see Working with .zip file archives for Python Lambda functions.

    Note: You can use a similar procedure to create a Lambda layer that can be used across multiple functions. For example, the following command creates a new Lambda layer to install pandas for the Lambda Python 3.9 runtime, running on arm64 architecture:

    pip install \
        --platform manylinux2014_aarch64 \
        --target=./python/lib/python3.9/site-packages \
        --implementation cp \
        --python-version 3.9 \
        --only-binary=:all: --upgrade \
        pandas

Related information

Creating and sharing Lambda layers

AWS OFFICIAL
AWS OFFICIALUpdated 9 months ago
6 Comments

Hey there,

Thanks for the guide! One issue that I noticed is that --python 3.9 should be --python-version 3.9. Otherwise, the dependencies will be installed for whatever version of Python is running on the host.

I am deploying a lambda layer via a CodeBuild project, and this seemed to work for me.

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago

I have tried those steps mentioned above, but I keep getting the same error. I am doing that through an AWS CDK app, that I am deploying through a CodePipeline.

Error:

{ "errorMessage": "Unable to import module 'pandasLambda': Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.9 from "/var/lang/bin/python3.9"\n * The NumPy version is: "1.24.3"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n", "errorType": "Runtime.ImportModuleError", "requestId": "", "stackTrace": [] }

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago
replied 9 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 9 months ago