I can't start tasks on Amazon Elastic Container Service (Amazon ECS) container instances that don't have enough disk space. This happens when the Docker daemon tries to pull the Docker image from my Docker image repository and then run the tasks. How can I resolve this?

Amazon ECS determines task placement by checking the CPU, memory, and port availability of Amazon ECS container instances. Amazon ECS doesn't consider the disk space availability on container instances.

Use a disk space monitoring script that monitors the disk space and then returns a value showing available disk space on Amazon ECS container instances. This value is associated with a container instance as a custom attribute. The AWS Identity and Access Management (IAM) role associated with the instance profile of the container instance has the permission to update this custom attribute value. After installing this script, use a task definition to specify a task placement constraint that restricts the placement of tasks on container instances that don't have enough disk space.

Launch your container instance with an Amazon ECS-Optimized Amazon Linux AMI

1.    To install the AWS Command Line Interface (AWS CLI) and jq packages in the user data section of the container instance, run the following command:

yum install -y aws-cli jq

Note: The user data section bootstraps Amazon EC2 instances by running commands at launch.

2.    Open the IAM console, and then create the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ecs:PutAttributes",
            "Resource": "*"
        }
    ]
}

Note: The inline policy grants permission to the IAM role associated with the IAM instance profile attached to the container instance.

3.    Associate the inline policy with the ecsInstanceRole IAM role.

Note: The IAM permission associates the SpaceLeft custom attribute with the container instance using the aws ecs put-attribute command in the AWS CLI. The SpaceLeft custom attribute is calculated by the disk space monitoring script.

Install and run the disk space monitoring script on the Amazon ECS container instance

1.    To monitor the disk space (in GB) of your cluster, install the following script on the container instance:

#! /usr/bin/env bash
set -o pipefail
set -o nounset
set -o errexit

until curl -s http://localhost:51678/v1/metadata
do
   sleep 1
done

clusterName=$(curl -s http://localhost:51678/v1/metadata | jq .Cluster | tr -d '"' )
SpaceLeft=$(docker info | grep 'Data Space Available' | cut -d':' -f2)
Unit=$(echo "$SpaceLeft" | grep -o '..$')
if [ "$Unit" != GB ]; then
    SpaceLeft=$(echo "$SpaceLeft" | sed 's/[A-Za-z]*//g' | cut -d '.' -f1)
    SpaceLeftValue=$(echo "$SpaceLeft" | awk '{$1=$1/(1000); print $1;}')
else
        SpaceLeftValue=$(echo "$SpaceLeft" | sed 's/[A-Za-z]*//g')
fi

instanceArn=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $NF}' )
region=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F: '{print $4}')

aws ecs put-attributes \
  --cluster "$clusterName" \
  --attributes name="SpaceLeft",value="$SpaceLeftValue",targetType="container-instance",targetId="$instanceArn" \
  --region "$region"

Note: You can name the script diskspaceMonitoring.sh, and then add it to the /home/ec2-user/diskspaceMonitoring.sh path on every Amazon ECS container instance in your cluster. You can also specify the disk space monitoring script as part of the user data on the container instance.

2.    On the container instance, run the following script:

$echo "* * * * * root /home/ec2-user/diskspaceMonitoring.sh" >> /etc/crontab

Note: You can run the script periodically as part of a cron job at a one-minute interval or at an interval of your choice.

3.    Open the Amazon ECS console, and then create a task placement constraint with your Amazon ECS task definition. See the following example:  

"placementConstraints": [
    {
        "expression": "attribute:SpaceLeft >= 0.1",
        "type": "memberOf"
    }
]

Note: The associated task definition now has a placement constraint that restricts the placement of tasks on container instances that don't have enough disk space.


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2019-01-22