How do I troubleshoot OutOfMemory errors in Amazon ECS?

Last updated: 2022-03-17

I want to troubleshoot memory usage issues in my Amazon Elastic Container Service (Amazon ECS) task.

-or-

The containers in my Amazon ECS task are exiting due to OutOfMemory error.

Short description

By default, a container has no resource constraints and can use as much resources as the host’s kernel scheduler allows. With Docker, you can control the amount of memory used by a container. Be sure not to allow a running container to consume most of the host machine’s memory. On Linux hosts, when the kernel detects that there isn't enough memory to perform important system functions, it throws an OutOfMemory exception and starts to end the processes to free up memory.

With Docker, you might choose to use either of the following:

  • Hard memory limits that allow the container to use no more than a certain amount of user or system memory
  • Soft limits that allow the container to use as much memory as required unless certain conditions, such as low memory or contention on the host machine, occur

When an Amazon ECS task is ended due to OutOfMemory issues, you might receive the following error message:

OutOfMemoryError: Container killed due to memory usage

You get this error when a container in your task exits because the processes in the container consume more memory than the amount that was allocated in the task definition.

Resolution

To troubleshoot OutOfMemory errors in your Amazon ECS task, do the following:

stats max(MemoryUtilized) as mem, max(MemoryReserved ) as memreserved by bin (5m) as period, TaskId, ContainerName
| sort period desc | filter ContainerName like “example-container-name” | filter TaskId = “example-task-id”

To mitigate the risk of task instability due to OutOfMemory issues, do the following:

  • Perform tests to understand the memory requirements of your application before placing the application in production. You can perform a load test on the container within a host or server, and then check the memory usage of the containers using docker stats.
  • Be sure that your application runs only on hosts with adequate resources.
  • Limit the amount of memory that your container can use. You can do this by setting appropriate values for hard limit and soft limit for your containers. Amazon ECS uses a couple of parameters for allocating memory to tasks: memoryReservation for soft limit and memory for hard limit. When you specify these values, they are subtracted from the available memory resources for the container instance on which the container is placed.
    Note: The parameter memoryReservation isn't supported for Windows containers.
  • You can turn on swap for containers with high transient memory demands. Doing so reduces the chance of OutOfMemory errors when the container is under high load.
    Note: If you're using tasks that use the AWS Fargate launch type, then parameters maxSwap and sharedMemorySize aren't supported.
    Important: Be mindful when you configure swap on your Docker hosts. Turning on swap can slow down your application and reduce the performance. However, this feature prevents your application from running out of system memory.

Did this article help?


Do you need billing or technical support?