Docker attempts to automatically extend its thin pool, which can cause dmeventd to spike CPU usage on an Amazon Elastic Container Service (Amazon ECS) instance. How can I fix this?

Docker attempts to automatically expand its thin pool but fails with the message "Insufficient free space" or "Failed to extend thin docker-docker--pool."

This error can be caused because of the Logic Volume (docker-docker-pool) not having enough space. By default, the Logic Volume Manager (LVM) auto-extend thin pool setting is disabled, as logical volume size is equal to secondary EBS volume size associated with the instance. The parameters thin_pool_autoextend_threshold and thin_pool_autoextend_percent and their configured values are stored in /etc/lvm/lvm.conf.

Be sure you are using the most recent version of the ECS-optimized AMI. As of ECS container agent 1.13.0, Amazon ECS automatically cleans up old images that use the Docker thin pool. You can also use the following commands on ECS agents before version 1.13.0:

#remove all unused images

docker images –q |xargs –no-run-if-empty docker rmi

The ECS Agent can assist with cleaning up containers because it checks the duration to wait from when a task is stopped until the Docker container is removed. The setting ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION can be modified in the /etc/ecs/ecs.config file. For more information, see Amazon ECS Container Agent Configuration.

If dmeventd is stuck in a loop attempting to extend the thin pool, you can also issue a command to stop the monitoring for the Docker volume group:

# lvchange –-monitor n docker

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2016-10-31

Updated: 2018-03-06