Introducing Amazon ECS Task Placement Policies

Today, Amazon ECS announced capabilities that provide granular control over how tasks are placed onto clusters. Previously, if you needed to place a task on a container instance with specific resource requirements (e.g., a specific instance type), you would have had to write custom schedulers to filter, find, and group resources.

The following diagram outlines the new task placement process:

Now, you can customize how tasks are placed without writing any code. ECS includes built-in attributes, such as instance type and Availability Zone, and supports custom attributes. For example, you can label container instances with attributes such as environment=production, use the list API operations to find those resources, and use the RunTask and CreateService API operations to place tasks on those resources.

You can also use placement strategies such as bin pack and spread to further define where tasks are placed. You can chain policies together to achieve sophisticated placement capabilities. For example, you can create a policy that places tasks only on g2.* instances, spreads the tasks across Availability Zones, and bin packs tasks within each zone based on memory.

First, look at attributes. You can use built-in attributes, such as instance type, to find container instances and place tasks on those container instances. In the following example, you can see all the t2 instances in the cluster:

aws ecs list-container-instances --filter "attribute:ecs.instance-type matches t2.*"
{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:123456789000:container-instance/40f0e62c-38cc-4cd2-a28e-770fa9796ca1",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/eb6680ac-407e-42a6-abd3-1bbf57d7401f",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/ecc03e17-6cbd-4291-bf24-870fa9796bf2",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/fbc03e17-acbd-2291-df24-4324ab342a24",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/f9a69f54-9ce7-4f1d-bc62-b8a9cfe8e8e5"
    ]
}

Then, list only the t2 instances that are in Availability Zone us-east-1a:

aws ecs list-container-instances --filter "attribute:ecs.instance-type matches t2.* and attribute:ecs.availability-zone == us-east-1a"
{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:123456789000:container-instance/40f0e62c-38cc-4cd2-a28e-770fa9796ca1",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/eb6680ac-407e-42a6-abd3-1bbf57d7401f",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/ecc03e17-6cbd-4291-bf24-870fa9796bf2"
    ]
}

Custom attributes extend the ECS data model with key-value pairs for your custom metadata. The following example adds the attribute stack=prod to a specific container instance:

aws ecs put-attributes --attributes name=stack,value=prod,targetId=40f0e62c-38cc-4cd2-a28e-770fa9796ca1,targetType=container-instance

And we can then see the container instances that do not have the attribute stack=prod:

aws ecs list-container-instances --filter "attribute:stack != prod"
{
    "containerInstanceArns": [
        "arn:aws:ecs:us-east-1:123456789000:container-instance/eb6680ac-407e-42a6-abd3-1bbf57d7401f",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/ecc03e17-6cbd-4291-bf24-870fa9796bf2",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/fbc03e17-acbd-2291-df24-4324ab342a24",
        "arn:aws:ecs:us-east-1:123456789000:container-instance/f9a69f54-9ce7-4f1d-bc62-b8a9cfe8e8e5"
    ]
}

Now, use these attributes to schedule a task. Constraints are rules based on attributes that are evaluated when ECS makes a scheduling decision. Constraints use MemberOf or DistinctInstance to create a subset of instances in the cluster to place tasks. The following example runs five tasks on instances that are of type t2.small or t2.medium and are not in Availability Zone us-east-1d:

aws ecs run-task --task-definition myapp --count 5 --placement-constraints type="memberOf",expression="(attribute:ecs.instance-type == t2.small or attribute:ecs.instance-type == t2.medium) and attribute:ecs.availability-zone != us-east-1d"

In addition to constraints, you can use a placement strategy using the RunTask API operation, and for a new service using the CreateService API operation. The strategy types are predefined but you can customize and combine them. The supported strategy types are Random, Binpack, and Spread. The following example spreads tasks across Availability Zones and—within each zone—bin packs tasks on memory.

aws ecs run-task --task-definition myapp --count 9 --placement-strategy type="spread",field="attribute:ecs.availability-zone" type="binpack",field="memory"

You can use constraints and placement strategies together. For example, you may want to spread tasks across Availability Zones and bin pack tasks on memory within each zone but only for instance type g2.*.

To define this placement policy in the ECS console, choose Create service. In the Task placement section, choose the AZ Balanced BinPack placement template, and choose Edit.

You can now create a custom policy based on that template. In the template’s Constraint section, add the following:

attribute:ecs.instance-type=~g2.*

You now have a service that places tasks only on G2 instances, spreads those tasks across Availability Zones, and bin packs the tasks onto the fewest number of instances in each zone by memory.

Conclusion

You can now add new attributes to ECS objects, query ECS resources in a more granular fashion, and direct task placement.

To learn more about task placement on ECS, see the topic in the Amazon ECS Developer Guide or the re:Invent session.

If you have questions or suggestions, please comment below.