How do I troubleshoot OutOfMemory errors in Amazon ECS?
Last updated: 2022-03-17
I want to troubleshoot memory usage issues in my Amazon Elastic Container Service (Amazon ECS) task.
The containers in my Amazon ECS task are exiting due to OutOfMemory error.
By default, a container has no resource constraints and can use as much resources as the host’s kernel scheduler allows. With Docker, you can control the amount of memory used by a container. Be sure not to allow a running container to consume most of the host machine’s memory. On Linux hosts, when the kernel detects that there isn't enough memory to perform important system functions, it throws an OutOfMemory exception and starts to end the processes to free up memory.
With Docker, you might choose to use either of the following:
- Hard memory limits that allow the container to use no more than a certain amount of user or system memory
- Soft limits that allow the container to use as much memory as required unless certain conditions, such as low memory or contention on the host machine, occur
When an Amazon ECS task is ended due to OutOfMemory issues, you might receive the following error message:
OutOfMemoryError: Container killed due to memory usage
You get this error when a container in your task exits because the processes in the container consume more memory than the amount that was allocated in the task definition.
To troubleshoot OutOfMemory errors in your Amazon ECS task, do the following:
- Check the stopped task for errors in the Amazon ECS console. Check the Stopped reason field for the error code OutOfMemory.
- Turn on Amazon CloudWatch Logs for your tasks to debug application level issues that occur due to memory usage.
- View the service’s memory use in either the Amazon ECS console or CloudWatch console.
- Use CloudWatch Container Insights to monitor memory use. You can view the memory usage of a certain container for a certain period of time with a query similar to the following:
stats max(MemoryUtilized) as mem, max(MemoryReserved ) as memreserved by bin (5m) as period, TaskId, ContainerName | sort period desc | filter ContainerName like “example-container-name” | filter TaskId = “example-task-id”
To mitigate the risk of task instability due to OutOfMemory issues, do the following:
- Perform tests to understand the memory requirements of your application before placing the application in production. You can perform a load test on the container within a host or server, and then check the memory usage of the containers using docker stats.
- Be sure that your application runs only on hosts with adequate resources.
- Limit the amount of memory that your container can use. You can do this by setting appropriate values for hard limit and soft limit for your containers. Amazon ECS uses a couple of parameters for allocating memory to tasks: memoryReservation for soft limit and memory for hard limit. When you specify these values, they are subtracted from the available memory resources for the container instance on which the container is placed.
Note: The parameter memoryReservation isn't supported for Windows containers.
- You can turn on swap for containers with high transient memory demands. Doing so reduces the chance of OutOfMemory errors when the container is under high load.
Note: If you're using tasks that use the AWS Fargate launch type, then parameters maxSwap and sharedMemorySize aren't supported.
Important: Be mindful when you configure swap on your Docker hosts. Turning on swap can slow down your application and reduce the performance. However, this feature prevents your application from running out of system memory.