How do I troubleshoot issues creating an AWS Fargate profile?

Last updated: 2021-11-04

I have questions about creating an AWS Fargate profile. Or, I'm having issues creating an AWS Fargate profile. How can I troubleshoot this?

Short description

A Fargate profile is a mechanism to specify which pods should be scheduled on Fargate nodes in an Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

A Fargate profile has selectors that are matched with every incoming pod specification YAML file. If the match is successful and AWS Fargate considerations are met, then the pod is scheduled on Fargate nodes using subnets and the AWS Identity and Access Management (IAM) role specified in the Fargate profile.

Some of the pod placement rules are as follows:

  • If you configured both namespace and match labels for your pod selectors:
    The Fargate workflow considers your pod to be matched with a Fargate profile only if both the conditions (namespace and labels) match the pod specification.
  • If you have specified multiple pod selectors within a single Fargate profile:
    When a pod matches either of these selectors, it's scheduled by the fargate-scheduler on a Fargate node using data from the matched Fargate profile.
  • If a pod specification matches with multiple Fargate profiles:
    The pod is scheduled according to a random Fargate profile, unless the following annotation is specified within the pod specification: eks.amazonaws.com/fargate-profile:<fp_name>.

Note the following limits when creating a Fargate profile:

  • You can create up to ten Fargate profiles per cluster.
  • You can have up to five selectors per Fargate profile.
  • You can have up to five label pairs per selector.

Resolution

The following are common scenarios and issues encountered when creating a Fargate profile:

How can I create a Fargate profile to schedule pods on Fargate nodes?

You can use the Amazon EKS console, AWS Command Line Interface (AWS CLI), SDK, or API (Cloudformation/eksctl, and so on) to create a Fargate profile.

How can I create a Fargate profile using AWS CloudFormation?

You can use the AWS::EKS::FargateProfile CloudFormation resource type to create a Fargate profile.

However, if you're not creating Amazon Elastic Compute Cloud (Amazon EC2) node groups along with Fargate nodes, then coredns add-ons have the following annotation by default:

eks.amazonaws.com/compute-type : ec2

To change the coredns annotations, you must patch your deployment externally. You can do this from the terminal where you manage your EKS cluster. Or, you can use a CloudFormation custom resource to automate this process based on your use case.

Note: It's a best practice to use eksctl to create/update EKS clusters because it simplifies cluster resource administration.

What is the pod execution role that must be included with the Fargate profile?

The pod execution role is an IAM role that's used by the Fargate node to make AWS API calls. These include calls made to fetch Amazon Elastic Container Registry (Amazon ECR) images such as VPC CNI, CoreDNS, and so on. The AmazonEKSFargatePodExecutionRolePolicy managed policy must be attached to this role.

Kubelet on the Fargate node uses this IAM role to communicate with the API server. This role must be included in the aws-auth configmap so that kubelet can authenticate with the API server. When you create a Fargate profile, the Fargate workflow automatically adds this role to the cluster's aws-auth configmap.

If your Fargate nodes are showing as 'Not Ready', then make sure that the pod execution role is included in aws-auth.

The following is a sample aws-auth mapRoles snippet after creating a Fargate profile with a pod execution role:

mapRoles: |   
    - groups:
      - system:bootstrappers
      - system:nodes
      - system:node-proxier
      rolearn: <Pod_execution_role_ARN>
      username: system:node:{{SessionName}}

If the aws-auth configmap is altered after creating the Fargate profile, then you might receive the following warning while scheduling pods on Fargate nodes:

Pod provisioning timed out (will retry) for pod: <pod_nginx>

I'm planning to migrate workloads to EKS Fargate. How do I create subnets and security groups for usage?

EKS Fargate currently supports only private subnets. This means that there isn't a default route to the internet gateway within the route tables attached to the subnets specified within your Fargate profile. So, you can have either a NAT gateway or VPC endpoints configured for the subnets that you intend to use for the Fargate profile.

The cluster security group is by default attached to Fargate nodes. You don't need to provision a security group specifically for this purpose. Also, if using VPC endpoints for your subnets, make sure that the cluster has private endpoint access activated.

Make sure that the security group attached to the VPC endpoints has an inbound rule allowing HTTPS port 443 traffic from the cluster's VPC CIDR.

I'm creating Fargate profiles using an API-based provisioner such as Terraform or AWS CloudFormation. Why are my Fargate profiles going into the CREATE_FAILED state.

Only one Fargate profile can be created or deleted at a time. So, if you're deleting a Fargate profile, then no other Fargate profiles can be created or deleted at the same time.

While using an API-based provisioner, such as CloudFormation, make sure that the creation of a Fargate profile starts after all other Fargate profiles are created successfully. You can create a chain-like hierarchy among Fargate profiles using the DependsOn attribute so that creation and deletion is sequential. If the requests aren't sequential, then you might receive an error similar to the following:

Cannot create Fargate Profile <fp_name1> because cluster <cluster_name> currently has Fargate profile <fp_name2> in status CREATING

Can I specify the resources (CPU, memory) to be provisioned for Fargate nodes within the Fargate profile?

You can't directly specify the amount of resources to be provisioned within the Fargate profile. It's a best practice to specify resource requests within your Fargate pod specification YAML file. Doing so helps the Fargate workflow assign at least that amount of resources for the pod.

The amount of vCPU or memory you see after running the kubectl describe node <node_name> command might not be the same as the amount of CPU or memory that you requested for the pod. The amount of memory and CPU the node has depends on the available capacity in the Fargate resource allocation pool. You're billed according to the amount that you requested within your pod specification. You aren't billed for the amount of resources visible with kubectl.

CPU and memory is always provisioned and billed by AWS Fargate as discrete combinations. The billable amount includes utilization by components other than the pod running on the node, such as kubelet, kube-proxy, and so on. For example, if you request one vCPU and eight GB memory for your pod, then you're billed for the next higher combination of two vCPU and nine GB memory. This accounts for the resources used by kubelet and other Kubernetes components on the node. For more information, see Pod CPU and memory.


Did this article help?


Do you need billing or technical support?