AWS Compute Blog

Building single binary file extensions for AWS Lambda with .NET

This post is written by Santiago Cardenas, Sr Partner SA and Dmitry Gulin, Modernization Architect.

Lambda Extensions integrate functions with other monitoring, observability, security, and governance tools. This blog post explains what functionality partners are providing via their extensions. It also demonstrates how to use Lambda extensions, using the AWS AppConfig extension provided by AWS.

.NET Core added support for self-contained executables. While these are not portable between operating systems and processor architectures, Lambda functions can benefit from not requiring a pre-installed and pre-configured .NET runtime.

This blog post shows how to create an AWS Lambda extension using C#, packaged as a self-contained binary file with no external dependency on the .NET runtime. You can then use this as an extension for any Lambda runtime. All source code for this demo extension is available on GitHub.

Creating a new project

Although single file executable support has been available since .NET Core 3.0, .NET 5.0 includes several important enhancements (see single file deployment and executable and .NET 5.0 Runtime highlights for more details). This walkthrough uses .NET 5.0.

With the .NET Core SDK installed, use these commands to create a new Lambda extension project:

mkdir csharp-example-extension
cd csharp-example-extension
dotnet new console

In a code editor, the initial project structure looks like this:

IDE file explorer view

Configuring the project file

Make the following changes to the project file:

  • Setting LangVersion to latest allows you to use the latest C# language features, like async main function, added in C# 7.1
  • The <PropertyGroup Condition=" '$(Configuration)' == 'Release' "> tag overrides the default release build configuration of the extension. It configures MSBuild to combine the .NET Core runtime and configuration files into a single executable file. Additionally, MSBuild skips debug symbols creation and packages only a subset of required assemblies, instead of the entire runtime.
  • The Target tag is used to invoke the Move command to move the resulting single executable into the extensions/ subdirectory. This ensures that the Lambda Extensions API can locate and launch the extension.

This is the project file (csharp-example-extension.csproj) after the changes are made:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
    <RootNamespace>csharp_example_extension</RootNamespace>
    <LangVersion>latest</LangVersion>
  </PropertyGroup>

  <PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
    <DebugSymbols>false</DebugSymbols>
    <DebugType>None</DebugType>
    <PublishTrimmed>true</PublishTrimmed>
    <PublishSingleFile>true</PublishSingleFile>
    <RuntimeIdentifier>linux-x64</RuntimeIdentifier>
  </PropertyGroup>

  <Target Name="RenameExtensionHook" AfterTargets="Publish" Condition=" '$(PublishSingleFile)' == 'true' ">
    <Move SourceFiles="$(PublishDir)/csharp-example-extension" DestinationFiles="$(PublishDir)/extensions/csharp-example-extension" OverwriteReadOnlyFiles="true" />
  </Target>

</Project>

For more details, read:

Creating the ExtensionClient class

Add a new ExtensionClient class to the project to encapsulate the Lambda Extensions API details.

Define the following Extensions API constants to the class, to reference these later in the code:

  • LambdaExtension*Header constants define HTTP header names for setting Extensions API context information.
  • LambdaRuntimeApiAddress defines the environment variable name that holds Lambda Extensions API URL.
  • basePath is a relative URL part, which is common for all Extensions API endpoints.

This is the first version of ExtensionClient class:

namespace csharp_example_extension
{
    public class ExtensionClient
    {
        private const string LambdaExtensionNameHeader = "Lambda-Extension-Name";
        private const string LambdaExtensionIdHeader = "Lambda-Extension-Identifier";
        private const string LambdaExtensionFunctionErrorTypeHeader = "Lambda-Extension-Function-Error-Type";
        private const string LambdaRuntimeApiAddress = "AWS_LAMBDA_RUNTIME_API";
        private const string basePath = "2020-01-01/extension";
    }
}

Add an HttpClient instance, which is used for communicating with the Lambda Extensions API. The HttpClient instance must be configured with an infinite timeout. This prevents disposing of an active connection while the function’s execution environment is frozen, when awaiting the next event from the Lambda Extensions API.

Note that the HttpClient class implements IDisposable. A correct implementation of the ExtensionClient class must also include an IDisposable implementation to dispose of the HttpClient instance correctly. See the GitHub repo for a Dispose implementation or read Implement a Dispose method for IDisposable pattern implementation details.

namespace csharp_example_extension
{
    public class ExtensionClient
    {
        // 
        // ... constants defintions ...
        // 
        
        private readonly HttpClient httpClient = new HttpClient() { Timeout = Timeout.InfiniteTimeSpan };
    }
}

Defining and initializing instance variables in the ExtensionClient constructor

The extension must communicate with the Lambda Extensions API during the function lifecycle. Define member variables to store the Lambda Extensions API URL paths and use a class constructor to construct all URL paths. The extension name is provided by the caller, so store it in an instance variable.

The Id property is used later in the code to track the extension’s identifier. This is assigned to the extension when it is registered with the Extensions API at runtime.

namespace csharp_example_extension
{
    public class ExtensionClient
    {
        // 
        // ... all other declarations ...
        // 
        
        public string Id { get; private set; }

        private readonly string extensionName;
        private readonly Uri registerUrl;
        private readonly Uri nextUrl;
        private readonly Uri initErrorUrl;
        private readonly Uri shutdownErrorUrl;
        
        public ExtensionClient(string extensionName)
        {
            this.extensionName = extensionName ?? throw new ArgumentNullException(nameof(extensionName), "Extension name cannot be null");

            var apiUri = new UriBuilder(Environment.GetEnvironmentVariable(LambdaRuntimeApiAddress)).Uri;

            // Calculate all Extension API endpoints' URLs
            this.registerUrl = new Uri(apiUri, $"{basePath}/register");
            this.nextUrl = new Uri(apiUri, $"{basePath}/event/next");
            this.initErrorUrl = new Uri(apiUri, $"{basePath}/init/error");
            this.shutdownErrorUrl = new Uri(apiUri, $"{basePath}/exit/error");
        }        
    }
}

Defining the main event loop

Add a single public method, which loops indefinitely, receiving events from the API, until a SHUTDOWN event is received. You can follow task-based asynchronous pattern guidelines to avoid blocking the main thread, while waiting for the next event to be received.

None of the functions in this class requires the async/await pattern. It is implemented so that this code can easily be extended for logic that benefits from an asynchronous pattern. You can also add several extra lines to work with Task.CompletedTask, to resolve the CS1998 warning.

namespace csharp_example_extension
{
    public class ExtensionClient
    {
        // 
        // ... constants, variables, and constructor ...
        // 
        
        public async Task ProcessEventsAsync(Func<string, Task> onInit = null, Func<string, Task> onInvoke = null, Func<string, Task> onShutdown = null)
        {
            // Register extension with AWS Lambda Extension API to handle both INVOKE and SHUTDOWN events
            await RegisterExtensionAsync(ExtensionEvent.INVOKE, ExtensionEvent.SHUTDOWN);

            // If onInit function is defined, invoke it and report any unhandled exceptions
            if (!await SafeInvokeAsync(onInit, this.Id, ex => ReportErrorAsync(this.initErrorUrl, "Fatal.Unhandled", ex))) return;

            // loop till SHUTDOWN event is received
            var hasNext = true;
            while (hasNext)
            {
                // get the next event type and details
                var (type, payload) = await GetNextAsync();

                switch (type)
                {
                    case ExtensionEvent.INVOKE:
                        // invoke onInit function if one is defined and log unhandled exceptions
                        // event loop continues even if there is an exception
                        await SafeInvokeAsync(onInvoke, payload, onException: ex => {
                            Console.WriteLine($"[{this.extensionName}] Invoke handler threw an exception");
                            return Task.CompletedTask;
                        });
                        break;
                    case ExtensionEvent.SHUTDOWN:
                        // terminate the loop, invoke onShutdown function if there is any and report any unhandled exceptions to AWS Extension API
                        hasNext = false;
                        await SafeInvokeAsync(onShutdown, this.Id, ex => ReportErrorAsync(this.shutdownErrorUrl, "Fatal.Unhandled", ex));
                        break;
                    default:
                        throw new ApplicationException($"Unexpected event type: {type}");
                }
            }
        }        
    }
}

There are three arguments defined for this method: onInit, onInvoke, and onShutdown. They are function pointers to asynchronous actions that are invoked during the lifecycle of this extension. All function pointer parameters are optional and ProcessEventsAsync does not try to invoke undefined functions:

  • onInit is invoked only once immediately after extension registration is complete.
  • onInvoke is called every time a new INVOKE event is received from Lambda Extensions API.
  • onShutdown is called after receiving the SHUTDOWN event and before the main message loop exits.

Unhandled exceptions thrown by the onInvoke lambda expression are logged to stdout and silently ignored. The ones thrown by onInit and onShutdown are reported to Lambda Extensions API initErrorUrl and shutdownErrorUrl respectively and the loop ends.

Implementing private methods

There are several private methods the implementation depends on:

  • RegisterExtensionAsync registers the current process with Lambda Extensions API to receive events. Argument is an array of event types this extension is registering for.
  • SafeInvokeAsync invokes an action, if it is defined, and catches any unhandled exception and passes it to the lambda expression, defined with the last argument.
  • GetNextAsync waits for the next event from Lambda Extensions API and return its type and raw payload buffer.

See the GitHub repository for an implementation of these private methods.

Implementing the extension’s main entry point

Extensions are Linux executables. You must include the extension lifecycle management calls in the Main function implementation that is the default entry point of our C# console application.

Setting <LangVersion>latest</LangVersion> configuration in the project file allows you to use an async signature (static async Task Main(string[] args)). This allows you to use await directly in the body of the Main method.

You can add console stdout statements to all lambda expression blocks passed to ProcessEventsAsync, so that you’ll have logs capturing all those events. If needed, any block can be replaced with a call to throw new Exception(), so that you can see how Init and Shutdown unhandled exceptions are reported to the Lambda Extensions API.

The final version of Program.cs file looks like:

namespace csharp_example_extension
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var extensionName = Assembly.GetEntryAssembly()?.GetName()?.Name;

            var client = new ExtensionClient(extensionName);

            // ProcessEvents loops internally until SHUTDOWN event is received
            await client.ProcessEventsAsync(
                // this expression is called immediately after successful extension registration with Lambda Extension API
                onInit: async id => {
                    Console.WriteLine($"[{extensionName}] Registered extension with id = {id}");
                    await Task.CompletedTask; // useless await, so that compiler doesn't report warnings
                },
                // this is called every time Lambda is invoked
                onInvoke: async payload =>
                {
                    Console.WriteLine($"[{extensionName}] Handling invoke from extension: {payload}");
                    await Task.CompletedTask; // useless await, so that compiler doesn't report warnings
                },
                // this is called just once - after receiving SHUTDOWN event and before exiting the loop
                onShutdown: payload => // this is an example of lambda expression implementation without async keyword
                {
                    Console.WriteLine($"[{extensionName}] Shutting down extension: {payload}");
                    return Task.CompletedTask;
                });

        }
    }
}

await Task.CompletedTask and return Task.CompletedTask are used to implement the asynchronous lambda expressions implementations.

Building and packaging the self-contained executable

The project file has the necessary settings for building and packaging this extension as a self-contained single file executable.

It uses Condition=" '$(Configuration)' == 'Release' " to limit packaging activity to the Release configuration only. This helps with running and debugging this console application locally, if needed. With the .NET SDK 5.0 or later installed, run this command to package into a single file in the bin/publish/extensions/ directory:

dotnet publish -c Release -o bin/publish

dotnet publish output
Lambda Extensions relies on directory structure and POSIX-compliant file permissions to discover and run new extensions. All extension entry points must be present in the extensions directory with the executable bit set. If the bit is not set, the Lambda environment reports a PermissionDenied error.

If you are using Linux or macOS for development, the MSBuild process sets the bit for you. You can validate this with the ls -l bin/publish/extensions command (note the x permission bit set for the file):

MSBuild output

Next, create a zip archive with the file data and its POSIX permissions information. For example, use the zip command on macOS to move extension executable into a new archive:

cd bin/publish
zip -rm ./deploy.zip *
cd -

Now the deploy.zip archive is in the bin/publish/ directory, ready to be deployed as a Lambda layer.

Windows file permissions are different to POSIX permissions. There is no direct mapping between NTFS access control lists (ACLs) and traditional Unix permissions. This limits options for creating zip archives with proper file permissions information. Developers can use the Windows Subsystem for Linux to set POSIX permissions and archive the file or use a custom archival tool (for example, the AWS Lambda for Go build-lambda-zip tool).

Deploying and testing the extension

Using the AWS Command Line Interface (AWS CLI), deploy the extension zip archive as a new Lambda layer:

aws lambda publish-layer-version \
        --layer-name "csharp-example-extension" \
        --zip-file "fileb://./bin/publish/deploy.zip"

Note the value of the output LayerVersionArn.

Publish layer output

Add the extension layer to a Lambda function:

aws lambda update-function-configuration \
  --function-name <<Your function name>> \
  --layers <<LayerVersionArn from publish-layer-version command output>>

The executable binary file for the extension can also be added to a container image to run in a Lambda function. To learn more, read Working with Lambda layers and extensions in container images.

This sample extension logs messages to the standard output. When you invoke the Lambda function, you see that additional log entries are added to that function’s CloudWatch Logs. All log entries created by the extension are prefixed with [csharp-example-extension]:

CloudWatch Logs output

Conclusion

This post shows how to create, build, and package a C# Lambda extension as a single binary file. Explore the example code and other Lambda extensions examples in the GitHub repository.

For more serverless learning resources, visit Serverless Land.