How can I avoid placing tasks on Amazon ECS container instances with Amazon Linux 1 AMIs that don't have enough disk space?

Last updated: 2020-08-20

I can't start tasks on Amazon Elastic Container Service (Amazon ECS) container instances with Amazon Linux 1 Amazon Machine Images (AMIs) that don't have enough disk space. This happens when the Docker daemon tries to pull the Docker image from my Docker image repository and then run the tasks.

Short description

When you launch a task that uses the Amazon Elastic Compute Cloud (Amazon EC2) launch type, Amazon ECS determines where to place the task based on the requirements specified in the task definition, such as CPU, memory, and port. Amazon ECS doesn't consider the disk space availability on container instances.

To resolve this issue, you can use a disk space monitoring script that monitors the disk space, and then returns a value showing available disk space on Amazon ECS container instances. This value is associated with a container instance as a custom attribute. The AWS Identity and Access Management (IAM) role associated with the instance profile of the container instance must have the permission to update this custom attribute value. After installing this script, use a task definition to specify a task placement constraint that restricts the placement of tasks on container instances that don't have enough disk space.

Note: Amazon Linux 1 reaches the end of support on December 31, 2020. It's a best practice to migrate workloads to Amazon Linux 2 before end of support.


Launch your container instance with an Amazon ECS-optimized Amazon Linux AMI

Important: It's a best practice to launch instances with an Auto Scaling group configured with multiple Availability Zones. You can add commands to bootstrap the instances in the user data of the launch configuration.

1.    To install the AWS Command Line Interface (AWS CLI) and jq packages (from the jq website) in the user data section of the container instance launch configuration, run the following command:

yum install -y aws-cli jq

Note: The user data section bootstraps Amazon EC2 instances by running commands at launch.

2.    Open the IAM console, and then create the following inline policy:

    "Version": "2012-10-17",
    "Statement": [
            "Effect": "Allow",
            "Action": "ecs:PutAttributes",
            "Resource": "*"

Note: The inline policy grants permission to the IAM role associated with the IAM instance profile attached to the container instance.

3.    Associate the inline policy with the ecsInstanceRole IAM role.

Note: The IAM permission associates the SpaceLeft custom attribute with the container instance using the aws ecs put-attribute command in the AWS CLI. The SpaceLeft custom attribute is calculated by the disk space monitoring script.

Install and run the disk space monitoring script on the Amazon ECS container instance

1.    To monitor the disk space (in GB) of your cluster, install the following script on the container instance:

#! /usr/bin/env bash
set -o pipefail
set -o nounset
set -o errexit

until curl -s http://localhost:51678/v1/metadata
   sleep 1

clusterName=$(curl -s http://localhost:51678/v1/metadata | jq .Cluster | tr -d '"' )
SpaceLeft=$(docker info | grep 'Data Space Available' | cut -d':' -f2)
Unit=$(echo "$SpaceLeft" | grep -o '..$')
if [ "$Unit" != GB ]; then
    SpaceLeft=$(echo "$SpaceLeft" | sed 's/[A-Za-z]*//g' | cut -d '.' -f1)
    SpaceLeftValue=$(echo "$SpaceLeft" | awk '{$1=$1/(1000); print $1;}')
        SpaceLeftValue=$(echo "$SpaceLeft" | sed 's/[A-Za-z]*//g')

instanceArn=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $NF}' )
region=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F: '{print $4}')

aws ecs put-attributes \
  --cluster "$clusterName" \
  --attributes name="SpaceLeft",value="$SpaceLeftValue",targetType="container-instance",targetId="$instanceArn" \
  --region "$region"

Note: You can name the script, and then add it to the /home/ec2-user/ path on every Amazon ECS container instance in your cluster. You can also specify the disk space monitoring script as part of the user data on the container instance launch configuration.

2.    On the container instance, run the following script:

$echo "* * * * * root /home/ec2-user/" >> /etc/crontab

Note: You can run the script periodically as part of a cron job at a one-minute interval. You can also choose a different time interval. You can add the script to the user data on your launch configuration.

3.    Open the Amazon ECS console, and then create a task placement constraint with your Amazon ECS task definition. For example:

"placementConstraints": [
        "expression": "attribute:SpaceLeft >= 0.1",
        "type": "memberOf"

Note: The associated task definition now has a placement constraint that restricts the placement of tasks on container instances that don't have enough disk space.

Did this article help?

Do you need billing or technical support?