Configuring .NET Garbage Collection for Amazon ECS and AWS Lambda

.NET developers rely on .NET’s automatic memory allocation and garbage collection (GC) mechanisms to handle the memory needs of their applications. For most use cases GC isn’t something developers need to worry about. However, in modern architectures where .NET applications are running in memory constrained environments, like containers and AWS Lambda functions, the GC might need extra information to understand how much memory is really available to the application.

Amazon Elastic Container Service (ECS)

.NET container applications are deployed to ECS as an ECS Task, and the amount of memory allocated for the Task is configured on the Task Definition. ECS uses cgroups, a Linux kernel feature, to restrict CPU and memory resources based on the task’s configured settings. For ECS tasks launched using AWS Fargate for compute the memory and CPU are required settings in the task definition. For ECS tasks launched to EC2 instances the task memory and CPU settings are required if the containers defined in the task definition do not have memory and CPU settings defined.

A hierarchy of cgroups for the task and the individual containers in the task are created when launching a ECS Task. The task’s memory settings are configured for the parent cgroup, which limits the amount of memory for the child cgroups used for the containers. However, the .NET GC does not support traversing the cgroup hierarchy for determining the amount of available memory. This means if memory is only configured at the task definition level the .NET GC doesn’t see the cgroup’s memory restriction, and instead detects the size of the underlying host compute’s memory.

To illustrate, running the following application will report the available memory the .NET GC sees is available. If you deploy this as an ECS Task using Fargate with the minimum memory setting of 0.5GB, the application will report there is 4GB available. The 4GB is coming from the underlying host compute.

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();


app.MapGet("/", () => "Memory Test");
app.MapGet("/memory", () =>
{
    return GC.GetGCMemoryInfo().TotalAvailableMemoryBytes.ToString("N0");
});

app.Run();

This causes the .NET GC to not aggressively release unused memory in the heap when it detects the .NET process is getting near the 0.5GB limit configured for the ECS task. This can trigger an OutOfMemoryException and shut down the container.

If a hard memory limit is set on the container definition within the ECS Task definition, the cgroup for the container will have a memory limit set and the .NET GC will see the correct amount of available memory for it to manage. You can set the hard memory limit in the console for the Task definition in the Environment section. This container hard memory limit can also be set programmatically through the AWS SDKs, AWS Tools for PowerShell, AWS CLI, AWS CloudFormation, and AWS Cloud Development Kit (CDK).

Once the new task definition is deployed to ECS the example code above will now report, correctly, that there is 0.5GB of available memory for the GC.

If the task definition has multiple containers defined then you will need to divide up the task definition’s allocated memory as needed across the different containers.

AWS .NET Deploy Tools

In the latest versions of the AWS .NET deploy tooling the container hard memory limit is now set to the same value as the task memory limit when deploying to ECS Fargate. This tooling is used from either the command line or Visual Studio using the AWS Toolkit for Visual Studio.

AWS Lambda

.NET code in Lambda also runs in a memory constrained environment. The minimum memory size for a Lambda function is 128 MB. Like Fargate, Lambda uses Linux’s cgroups to restrict CPU and memory settings based on the Lambda function’s configured memory size. However, in the case of Lambda the use of cgroups is completely hidden from the .NET runtime. This again, causes the .NET GC to think that more memory is available than is really the case.

For example, if you deploy the following function with a memory size of 128MB it will likely report a memory size larger then 128MB.

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace LambdaMemoryCheck;

public class Function
{
    public string FunctionHandler()
    {
        return GC.GetGCMemoryInfo().TotalAvailableMemoryBytes.ToString("N0");
    }
}

Lambda, unlike ECS, does not have a container hard memory limit you can set. To inform the .NET GC how much memory is available you can set the DOTNET_GCHeapHardLimit environment variable that the .NET GC knows to look for. The value of DOTNET_GCHeapHardLimit is the number of bytes in hexadecimal the GC should limit itself too. For convenience the table below gives the hexadecimal values for the available Lambda configurations up to 1GB.

The DOTNET_GCHeapHardLimit environment variable can be set programmatically using any of the AWS SDKs and tools. It can also be set in the AWS Console, and in Visual Studio using the AWS Toolkit for Visual Studio.

Conclusion

If your .NET applications are experiencing memory issues we recommend tweaking the GC with container hard memory limits or using the DOTNET_GCHeapHardLimit environment variable for Lambda functions. For more information on configuration settings for the .NET GC checkout this article from MSDN.

In the future, AWS and Microsoft hope to make the .NET GC automatically understand the memory restrictions in these environments. Microsoft has opened a GitHub issue to track handling the hierarchy cgroups that ECS uses. AWS will be looking into how we can have the DOTNET_GCHeapHardLimit variable set automatically for .NET Lambda functions.

Special thanks to Maoni Stephens from the Microsoft .NET team for helping understand the .NET GC behavior and collaborating on this post.