How can I avoid placing tasks on Amazon ECS container instances with Amazon Linux 1 AMIs that don't have enough disk space?

Last updated: 2021-01-28

I use Amazon Elastic Container Service (Amazon ECS) container instances with Amazon Linux 1 Amazon Machine Images (AMIs). However, I can’t start tasks on Amazon ECS container instances with Linux 1 AMIs that don’t have enough disk space. This happens when the Docker daemon tries to pull the Docker image from my Docker image repository and then run the tasks. How can I prevent this issue?

Short description

When starting a task using the Amazon Elastic Compute Cloud (Amazon EC2) launch type, Amazon ECS doesn't consider the disk space availability on container instances. Amazon ECS determines where to place the task based on the requirements specified in the task definition, such as CPU, memory, and port.

To resolve this issue, you can use a disk space monitoring script that monitors the disk space. The script returns a value showing available disk space on Amazon ECS container instances. The returned value is associated with a container instance as a custom attribute. The AWS Identity and Access Management (IAM) role associated with the container instance profile must have permission to update this custom attribute value.

Then, use a task definition to specify a task placement constraint restricting the placement of tasks on container instances that don't have enough disk space.

Note: Amazon Linux 1 reaches the end of support on December 31, 2020. It's a best practice to migrate workloads to Amazon Linux 2 before end of support.

Resolution

Launch your container instance with an Amazon ECS-optimized Amazon Linux AMI

Important: It's a best practice to launch instances with an Auto Scaling group configured with multiple Availability Zones. You can add commands to bootstrap the instances in the launch configuration user data.

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

1.    Run the following command to install the AWS CLI and jq packages in the user data section of the container instance launch configuration. For more information, see ./jq on the jq website.

yum install -y aws-cli jq

Note: The user data section bootstraps Amazon EC2 instances by running commands at launch.

2.    Open the IAM console, and then create the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ecs:PutAttributes",
            "Resource": "*"
        }
    ]
}

Important: The above policy uses a wildcard parameter, which allows the principal to call ecs:PutAttributes for any compatible resources (container instances) in the account. The example is for ease of use, but it's a best practice to grant least privilege when possible. For more information, see Amazon Elastic Container Service identity-based policy examples. The following example policy limits the call to only container instances in a specific cluster:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:PutAttributes"
            ],
            "Resource": [
                "arn:aws:ecs:<region>:<accountId>:container-instance/<clusterName>/*"
            ]
        }
    ]
}

3.    Associate the inline policy with the ecsInstanceRole IAM role.

Note: The IAM permission associates the SpaceLeft custom attribute with the container instance using the aws ecs put-attribute command in the AWS CLI. The SpaceLeft custom attribute is calculated by the disk space monitoring script.

Install and run the disk space monitoring script on the Amazon ECS container instance

1.    To monitor the disk space (in GB) of your cluster, install the following script on the container instance:

#! /usr/bin/env bash
set -o pipefail
set -o nounset
set -o errexit

until curl -s http://localhost:51678/v1/metadata
do
   sleep 1
done

clusterName=$(curl -s http://localhost:51678/v1/metadata | jq .Cluster | tr -d '"' )
SpaceLeft=$(docker info | grep 'Data Space Available' | cut -d':' -f2)
Unit=$(echo "$SpaceLeft" | grep -o '..$')
if [ "$Unit" != GB ]; then
    SpaceLeft=$(echo "$SpaceLeft" | sed 's/[A-Za-z]*//g' | cut -d '.' -f1)
    SpaceLeftValue=$(echo "$SpaceLeft" | awk '{$1=$1/(1000); print $1;}')
else
        SpaceLeftValue=$(echo "$SpaceLeft" | sed 's/[A-Za-z]*//g')
fi

instanceArn=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $NF}' )
region=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F: '{print $4}')

aws ecs put-attributes \
  --cluster "$clusterName" \
  --attributes name="SpaceLeft",value="$SpaceLeftValue",targetType="container-instance",targetId="$instanceArn" \
  --region "$region"

Note: You can name the script diskspaceMonitoring.sh, and then add it to the /home/ec2-user/diskspaceMonitoring.sh path on every Amazon ECS container instance in your cluster. You can also specify the disk space monitoring script as part of the user data on the container instance launch configuration.

2.    On the container instance, run the following script:

$echo "* * * * * root /home/ec2-user/diskspaceMonitoring.sh" >> /etc/crontab

Note: You can run the script periodically as part of a cron job at a one-minute interval. You can also choose a different time interval. You can add the script to the user data on your launch configuration.

3.    Open the Amazon ECS console, and then create a task placement constraint with your Amazon ECS task definition. For example:

"placementConstraints": [
    {
        "expression": "attribute:SpaceLeft >= 0.1",
        "type": "memberOf"
    }
]

The associated task definition now has a placement constraint that restricts the placement of tasks on container instances that don't have enough disk space.


Did this article help?


Do you need billing or technical support?