AWS Developer Tools Blog
AWS Lambda layers with .NET Core
Update October 15, 2020: Version 4.2.0 of Amazon.Lambda.Tools has been released that adds Lambda layers support for .NET Core 3.1. This was initially disabled due to an issue in the .NET Core SDK. A fix was released in version 3.1.400 of the .NET Core SDK and is the required minimum to be installed to create layers for .NET Core 3.1.
Lambda layers enable you to provide additional code and content to your AWS Lambda function. A layer is composed of additional files used by your Lambda function that are extracted into the /opt directory in the Lambda compute environment.
Since the release of Lambda Layers one of the common questions I hear is how can .NET Core Lambda functions take advantage of this feature. For .NET Core, there are a couple challenges that you have to overcome to take advantage of layers. First, you have to tell the .NET runtime to load assemblies from outside of the deployment bundle. The other big challenge is when the dotnet publish command is executed, which all of our .NET Lambda tools rely on to gather all of the required .NET assemblies, the publish command needs to know what assemblies not to include because they will be provided by a layer.
Thankfully, the .NET Core tooling has some lesser-known features to make this work, but they’re a bit tricky to use. With version 3.2.0 of the Amazon.Lambda.Tools .NET Core Global Tool the process of creating layers and using them with your Lambda functions is now simple to use.
The major benefit of using layers is it can dramatically reduce the size of the .zip file that has to be uploaded to Lambda whenever you deploy a function. Also, there are opportunities to improve cold-start performance that are described below in the optimizing packages section.
Runtime package stores
The .NET Core technology used by Amazon.Lambda.Tools to create layers is called runtime package stores, https://docs.microsoft.com/en-us/dotnet/core/deploying/runtime-store. This was introduced as part of .NET Core 2.0.
A manifest file is used to create a runtime package store. The project files, that is the *.csproj file in your Lambda projects, are an example of a manifest. When you create a runtime package store, all of the NuGet packages identified by the PackageReference elements in the manifest and their dependencies are captured in a directory that Amazon.Lambda.Tools will turn into a Lambda layer.
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp2.1</TargetFramework>
<GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
<AWSProjectType>Lambda</AWSProjectType>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Amazon.Lambda.Core" Version="1.1.0" />
<PackageReference Include="Amazon.Lambda.Serialization.Json" Version="1.4.0" />
</ItemGroup>
</Project>
This means you can create a layer for all of the dependencies in your Lambda project. Then when you deploy, you will only upload the assemblies for your local projects. Or you could create a custom manifest in the same style as a .csproj file with all of your common dependencies, and create a layer from that. You then reference that layer from all of your Lambda projects.
For a full description of how runtime package stores are turned into Lambda layers and how they work, I recommend checking out our .NET Lambda layers documentation, https://github.com/aws/aws-extensions-for-dotnet-cli/blob/master/docs/Layers.md.
Creating a Lambda layer
As we mentioned earlier, a layer can be created from the .csproj file of a Lambda project. To create a layer, execute the following command in the project directory.
dotnet lambda publish-layer LayerBlogDemoLayer --layer-type runtime-package-store --s3-bucket <s3-bucket>
This creates a layer called LayerBlogDemoLayer. The type will be runtime-package-store. Currently, runtime-package-store is the only valid value, but it’s a required field to allow us to create new types of layers in the future. The Amazon S3 bucket that’s specified is used to upload the runtime package store created locally, and which Lambda will use to create the layer from.
Here is output of the publish-layer command.
...
... Progress: 100%
Upload complete to s3://normj-west2/LayerBlogDemoLayer-636888783824636933/artifact.xml
Create zip file of runtime package store directory
... zipping: dotnetcore\store\x64\netcoreapp2.1\artifact.xml
... zipping: dotnetcore\store\x64\netcoreapp2.1\amazon.lambda.core\1.0.0\lib\netstandard1.3\Amazon.Lambda.Core.dll
... zipping: dotnetcore\store\x64\netcoreapp2.1\amazon.lambda.core\1.1.0\lib\netstandard2.0\Amazon.Lambda.Core.dll
... zipping: dotnetcore\store\x64\netcoreapp2.1\amazon.lambda.serialization.json\1.4.0\lib\netstandard1.3\Amazon.Lambda.Serialization.Json.dll
... zipping: dotnetcore\store\x64\netcoreapp2.1\newtonsoft.json\9.0.1\lib\netstandard1.0\Newtonsoft.Json.dll
Uploading layer input zip file to S3
Uploading to S3. (Bucket: normj-west2 Key: LayerBlogDemoLayer-636888783824636933/packages.zip)
... Progress: 52%
... Progress: 100%
Upload complete to s3://normj-west2/LayerBlogDemoLayer-636888783824636933/packages.zip
Layer publish with arn arn:aws:lambda:us-west-2:123412341234:layer:LayerBlogDemoLayer:1
There are two things I want to call out here. First, there was an artifact.xml file loaded to Amazon S3. This file is important because when Lambda functions are later deployed using this layer, the artifact.xml file is downloaded by Amazon.Lambda.Tools to tell the dotnet publish command the assemblies to not include. The use of this file will be transparent to you, but beware that this file is uploaded to S3, and is meant to stay there as long as you want to use this layer. To share your layer with other accounts, you also need to share this object in S3.
The most important information in the output is the ARN of the new version of the layer, which you can see on the last line. This value is what you’ll use when deploying functions.
Inspect your layer
If you execute the dotnet lambda help command, you can see there are several new commands added to manage your layers.
PS> dotnet lambda help
Amazon Lambda Tools for .NET Core applications (3.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
...
Commands to publish and manage AWS Lambda Layers:
publish-layer Command to publish a Layer that can be associated with a Lambda function
list-layers Command to list Layers
list-layer-versions Command to list versions for a Layer
get-layer-version Command to get the details of a Layer version
delete-layer-version Command to delete a version of a Layer
...
To inspect your layer, the get-layer-version will let you know what assemblies are in the layer and where the artifact.xml file is stored.
PS< dotnet lambda get-layer-version arn:aws:lambda:us-west-2:123412341234:layer:LayerBlogDemoLayer:1
Amazon Lambda Tools for .NET Core applications (3.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
Layer ARN: arn:aws:lambda:us-west-2:123412341234:layer:LayerBlogDemoLayer
Version Number: 1
Created: 3/22/2019 12:06 PM
License Info:
Compatible Runtimes: dotnetcore2.1
Layer Type: .NET Runtime Package Store
.NET Runtime Package Store Details:
Manifest Location: s3://normj-west2/LayerBlogDemoLayer-636888783824636933/artifact.xml
Packages Optimized: False
Packages Directory: /opt/dotnetcore/store
Manifest Contents
-----------------------
<StoreArtifacts>
<Package Id="Amazon.Lambda.Core" Version="1.1.0" />
<Package Id="Amazon.Lambda.Core" Version="1.0.0" />
<Package Id="Amazon.Lambda.Serialization.Json" Version="1.4.0" />
<Package Id="Newtonsoft.Json" Version="9.0.1" />
</StoreArtifacts>
Deploying with your layers
To use the layer when you deploy the function, use the --function-layers switch. This should be set to the layer version ARN output by the publish-layer command. You can use a comma-separated list of ARNs to use multiple layers.
PS< dotnet lambda deploy-function LayerBlogDemo --function-layers arn:aws:lambda:us-west-2:123412341234:layer:LayerBlogDemoLayer:1
Amazon Lambda Tools for .NET Core applications (3.2.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
Inspecting Lambda layers for runtime package store manifests
... arn:aws:lambda:us-west-2:626492997873:layer:LayerBlogDemoLayer:1: Downloaded package manifest for runtime package store layer
Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder 'C:\temp\LayerBlogDemo\src\LayerBlogDemo\bin\Release\netcoreapp2.1\publish'
... Disabling compilation context to reduce package size. If compilation context is needed pass in the "/p:PreserveCompilationContext=false" switch.
... publish: Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
... publish: Copyright (C) Microsoft Corporation. All rights reserved.
... publish: Restore completed in 46.95 ms for C:\temp\LayerBlogDemo\src\LayerBlogDemo\LayerBlogDemo.csproj.
... publish: LayerBlogDemo -> C:\temp\LayerBlogDemo\src\LayerBlogDemo\bin\Release\netcoreapp2.1\rhel.7.2-x64\LayerBlogDemo.dll
... publish: LayerBlogDemo -> C:\temp\LayerBlogDemo\src\LayerBlogDemo\bin\Release\netcoreapp2.1\publish\
Zipping publish folder C:\temp\LayerBlogDemo\src\LayerBlogDemo\bin\Release\netcoreapp2.1\publish to C:\temp\LayerBlogDemo\src\LayerBlogDemo\bin\Release\netcoreapp2.1\LayerBlogDemo.zip
... zipping: LayerBlogDemo.deps.json
... zipping: LayerBlogDemo.dll
... zipping: LayerBlogDemo.pdb
... zipping: LayerBlogDemo.runtimeconfig.json
Updating code for existing function LayerBlogDemo
Notice at the start of deployment that the Lambda layers were inspected and the artifact.xml file was downloaded. Under the covers, the artifact.xml file was passed into the dotnet publish command, which told it not to include the Amazon.Lambda.* and Newtonsoft.Json NuGet packages. You can see that only the project’s assembly was included with the package bundle.
When using the deploy-serverless command to deploy with an AWS CloudFormation template, set the Layers property. The deploy-serverless command performs the same inspection of the layers that we saw in the *deploy-function* command.
javascript
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Transform" : "AWS::Serverless-2016-10-31",
"Description" : "An AWS Serverless Application.",
"Resources" : {
"LayerBlogDemo" : {
"Type" : "AWS::Serverless::Function",
"Properties": {
"Handler": "LayerBlogDemo::LayerBlogDemo.Function::FunctionHandler",
"Runtime": "dotnetcore2.1",
"Layers" : ["arn:aws:lambda:us-west-2:123412341234:layer:LayerBlogDemoLayer:1"],
"CodeUri": "",
"MemorySize": 256,
"Timeout": 30,
"Role": null,
"Policies": [ "AWSLambdaBasicExecutionRole" ]
}
}
}
}
Optimizing packages
A feature of a runtime package store is that the .NET assemblies placed into the store can be optimized for the target runtime by pre-jitting the assemblies. Pre-jitting is the process of compiling the platform-agnostic machine code of an assembly, known as MSIL, into native machine code. Without pre-jitting, the assemblies are compiled into native machine code when they are first loaded into the .NET Core Process. Enabling the optimization can significantly reduce cold-start times in Lambda.
To create an optimized runtime package store layer, you must run the publish-layer command in an Amazon Linux environment. Attempting to create an optimized runtime package store layer on Windows or macOS will result in an error. If you’re creating the layer on Linux, be sure the distribution is Amazon Linux. Amazon EC2 provides an AMI with Amazon Linux and .NET Core 2.1 preinstalled and you can easily launch them from the AWS Toolkit for Visual Studio.
A nice aspect of using optimized layers is that you can create the layer once on an Amazon Linux instance to get the pre-jitted benefits, and then share that layer version ARN with all of the Lambda functions you are developing on Windows and macOS.
To tell the publish-layer command to optimize the layer, set the --enable-package-optimization switch to true.
Upcoming AWS Toolkit for Visual Studio
With today’s release you can use Lambda layers with the Amazon.Lambda.Tools .NET Core Global Tool. We’re finishing updates to the AWS Toolkit for Visual Studio to add support for the upcoming Visual Studio 2019. When we release this update, you’ll be able to reference the layers you created with the publish-layer command in either the aws-lambda-tools-defaults.json or serverless.template file, and the deployment from Visual Studio will process the layers in the same way we saw with the deploy-function and deploy-serverless commands. Be sure to monitor our .NET blog or the @awsfornet Twitter handle for the upcoming AWS Toolkit for Visual Studio release.
Summary
There’s a lot going on under the hood for Amazon.Lambda.Tools to provide a seamless experience. I recommend checking out the full docs about how this feature works from our GitHub repository, https://github.com/aws/aws-extensions-for-dotnet-cli/blob/master/docs/Layers.md. There’s a FAQ and more details about how to use layers with the package and package-ci commands for CI systems.
I’m excited to add this requested feature to our Lambda .NET tool chain. I hope you find the process of using layers seamless, and we welcome any feedback on this feature. Feel free to reach out on the .NET Lambda repository.