Docker attempts to automatically extend its thin pool, which can cause dmeventd to spike CPU usage on an Amazon EC2 instance. How can I fix this?

Docker attempts to automatically expand its thin pool but fails with the message "Insufficient free space" or "Failed to extend thin docker-docker--pool".

This may be caused by Logical Volume Manager (LVM) auto-extending the volume. By default, LVM extends the thin pool when it reaches 100% utilization; the pool is extended by 20% of the total volume size. This can be verified with thin_pool_autoextend_threshold and thin_pool_autoextend_percent parameters in the LVM configuration file /etc/lvm/lvm.conf.

Ensure you are using the most recent version of the ECS-Optimized AMI. As of ECS container agent 1.13.0, ECS automatically cleans up old images that use the Docker thin pool. You can also use the following commands on ECS agents before version 1.13.0:

#remove all unused images

docker images –q |xargs –no-run-if-empty docker rmi

The ECS Agent can assist with cleaning up containers because it checks the duration to wait from when a task is stopped until the Docker container is removed. The setting ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION can be modified in the /etc/ecs/ecs.config file. For more information, see Amazon ECS Container Agent Configuration.

If dmeventd is stuck in a loop attempting to extend the thin pool, you can also issue a command to stop the monitoring for the Docker volume group:

# lvchange –-monitor n docker

LVM, Docker auto extend, Docker thin pool, dmeventd, Docker, ECS, EC2 Container Service, Logical Volume Manager

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2016-10-31