Containers

Use CloudFormation to automate management of the Fargate profile in Amazon EKS

Organizations are embracing microservices architectures and container-based deployments to gain agility, scalability, isolation, and separation of concerns. AWS Fargate, the serverless compute engine for running containers in the AWS Cloud, improves agility by taking away the undifferentiated heavy lifting of worker node provisioning and management. With Fargate, organizations can focus on building applications and application features that matter to their business rather than performing operational tasks like patching, scaling, security, and so on.

In Amazon Elastic Kubernetes Service (Amazon EKS), the Fargate profile is used by administrators to define selectors that determine which pods are run on Fargate. The profile includes attributes for controlling networking and security and a pod execution role that allows Fargate to perform actions on the user’s behalf, such as pulling images from Amazon Elastic Container Registry (Amazon ECR).

In large organizations, the attributes of a Fargate profile might be managed by different teams. For example, networking team might be responsible for designating the subnet(s) on which Fargate pods should run while another team might be responsible for configuring the selectors.

Manually administering a large number of Fargate profiles is time-consuming and error-prone, especially in a fluid or growing environment. Accidentally deleting or modifying the wrong profile can introduce significant risk and cause deployments to become unpredictable.

These are some of the most popular approaches and tools used to automate management of the Fargate profile in Amazon EKS environment:

You can use other options like the Fargate APIs or the AWS SDK for Python (Boto3). You can also write your own operator or use an existing one. Due to the complexity and maintenance overhead, these are not preferred options. The choice depends on an organization’s preference and strategy, tooling and skills, priorities for extensibility, recoverability, security and controls, and more.

In this blog post, I explain how to use CloudFormation to manage Fargate profiles and refer to an open source CloudFormation template solution that automates the deployment of Fargate profiles using AWS CodePipeline.

Solution

A Fargate profile consists of subnets, selectors, and a pod execution role.

The following screenshot shows a sample Fargate profile.

sample Fargate profile

Because the pod execution role applies to all the pods scheduled under a profile, it’s not a good practice to use pod execution roles to assign permissions to Fargate pods. Use the IAM roles for service accounts feature, which allows you to implement granular access controls while adhering to the principle of least privilege.

Pods that match a selector are scheduled on Fargate. If a namespace selector is defined without any labels, Amazon EKS will attempt to schedule all pods that run in that namespace onto Fargate using that profile. You can specify up to five selectors in a Fargate profile. A pod only needs to match one of the selectors to run using that profile.

Using CloudFormation to manage a Fargate profile

The management of a Fargate profile or its components is handled at the profile level. The profile is operated as a single unit of work regardless of whether the change is limited to specific components or the entire profile. You can perform create and update operations against a Fargate profile.

Note: Fargate profiles are immutable. However, you can create a new updated profile by combining profile delete and create operations to replace an existing profile and then delete the original after the updated profile has been created.

Step 1: Creating the Fargate profile

For Amazon EKS to successfully schedule pods to Fargate, you must first add a matching Fargate profile to the EKS cluster.

When you use CloudFormation to create a profile, you supply a template that includes a profile definition and create a CloudFormation stack. A new Fargate profile will be created if:

  • The Fargate profile has a valid EKS cluster name and contains at least one selector with a namespace, private subnet, and execution role.
  • The logical ID for the Fargate profile resource (first blue arrow in the sample template) is unique and not linked to any other profile in the cluster.
  • The FargateProfileName property (second blue arrow in the sample template) is a user-defined name that is unique across profiles in the cluster.

The sample template contains two resources: the EKS cluster (outlined in orange) and a Fargate profile (outlined in blue). Running this stack will create an EKS cluster named eksfargate and a new profile named fargateprofile1.

It’s a good practice to use a DependsOn clause in the Fargate profile resource definition to tie dependency to the EKS cluster resource.

sample template

Step 2: Updating the Fargate profile

You might need to update a Fargate profile several times during the lifecycle of a cluster. For example, you might need to add new subnet IDs for cross-AZ scaling or split profiles to support a change in ownership or governance.

When you deploy a profile change through CloudFormation, you simply update the stack by applying the updated template. CloudFormation will delete the profile with the matching logical ID and create a new profile definition from the template if:

  • The Fargate profile has a valid EKS cluster name and contains at least one selector with a namespace, private subnet, and execution role.
  • The logical ID for the Fargate profile resource (first blue arrow in the sample template) is the same as the one used in the previous deployment and is not linked to any other Fargate profile in the cluster.
  • The FargateProfileName property (second blue arrow in the sample template) is a user-defined name that is unique across profiles in the cluster.

Note: It’s good DevOps practice to version CloudFormation templates in a version control system and automate the testing and deployment of the templates into the environment.

Behind the scenes, CloudFormation synchronizes the operations to make sure that the profile is successfully deleted before a new profile is created.

Warning: Any running pods that are using the Fargate profile will be stopped and put into a pending status when the profile is deleted.

Wait for the cluster update operation to complete before you issue more profile operations to the same cluster.

The following sample contains two resource definitions: the EKS cluster (outlined in orange) and the Fargate profile (outlined in blue). Running the following stack on top of the sample in Step 1 will remove the Fargate profile named fargateprofile1 from the eksfargate cluster and create a new profile named updatedfargateprofile and the definition provided in the following template.

sample template two resource definitions

Step 3: Deleting the Fargate profile

To delete a Fargate profile, you simply remove the profile resources from the template and supply an updated template to update the stack. Behind the scenes, CloudFormation will delete the existing profile that was created by the stack.

The following sample contains only EKS cluster resource definitions. Running the following stack update on top of the sample in Step 2 will delete the Fargate profile named updatedfargateprofile from the eksfargate cluster.

sample template with only EKS cluster resource definitions

Wait for the profile delete operation to complete before you issue more profile operations to the same cluster.

Continuous delivery of the Fargate profile through CloudFormation

When you design CloudFormation templates, use features like nested stacks. Use the AWS::Include function to create a modular design that separates one profile definition from another and from the cluster itself. The AWS::Include function allows you to insert the contents of the specified file at the location of the transform in the CloudFormation template.

The AWS::Include function is helpful if you’re working in a multi-tenancy environment with strict compliance and security requirements, where developers are responsible for definition of the Fargate profile, but the profile deployment is handled by another team that manages the cluster.

Here is an approach commonly used to implement a CD pipeline for a Fargate profile:

CD pipeline flow for Fargate profile

Pipeline flow

  1. A DevOps admin commits the Fargate profile template JSON file to AWS CodeCommit. For information, see Create or add a file to an AWS CodeCommit repository in the AWS CodeCommit User Guide.
  2. Upon successful commit, the Fargate profile is uploaded to the AWS CodeCommit repository.
  3. An Amazon EventBridge rule is configured to monitor for repository state change. It triggers AWS CodePipeline to start the pipeline execution when the file is added or updated.
  4. The CodePipeline execution invokes the Source stage of the pipeline, which pulls the Fargate profile template from the repository and passes it as input to the Deploy stage. The Deploy stage is configured to use the CloudFormation provider feature. The pipeline creates a CloudFormation stack to deploy the template committed by the admin in step 1.
  5. The stack is launched to create or delete the Fargate profile. The EKS cluster is updated to reflect those changes based on the template.

The open-source solution that automates the setup of the CD pipeline and management of the Fargate profile is available on GitHub.

Note: If you have multiple pipeline executions for a Fargate profile in the same cluster, design your pipelines with appropriate waits or checks to ensure that the current operation is complete before issuing subsequent profile operations to the same cluster.

Cleaning up

To avoid incurring future charges in your account, delete the CloudFormation stack to remove EKS cluster.

Conclusion

The AWS Containers team is constantly adding capabilities to make it easier and faster for developers and operations teams to deploy workloads to Amazon EKS on Fargate.

In this blog post, I showed you how to manage and automate the deployment of Fargate profiles using AWS CloudFormation. For more information about Fargate profile constructs and CloudFormation, see AWS::EKS::FargateProfile in the AWS CloudFormation User Guide and Creating Amazon EKS resources with AWS CloudFormation in the Amazon EKS User Guide.

This information might be helpful when you design a DevOps strategy: