Understanding data transfer costs for AWS container services

Overview

Data transfer costs can play a significant role in determining the overall design of a system. The Amazon Elastic Container Registry (Amazon ECR), Amazon Elastic Container Service (Amazon ECS), and Amazon Elastic Kubernetes Service (Amazon EKS) can all incur data transfer charges depending on a variety of factors. It can be difficult to visualize what that means relative to an Amazon ECS or Amazon EKS deployment. This blog illustrates common deployment patterns for AWS container services and explains the resulting data transfer charges that may be incurred. Service charges are out of scope for this blog, but should be carefully considered when designing any architecture.

Amazon ECR

An Amazon ECR private registry hosts container images in a highly available and scalable architecture. There are two types of registries in Amazon ECR—private and public—and each one has different data transfer charges.

Amazon ECR private registry

You can use an ECR private registry to manage private image repositories consisting of Docker and Open Container Initiative (OCI) images and artifacts.

Architecture diagram describing data transfer in to and out of ECR private registry. Data transfer across Regions – either for image pulls or for ECR cross-Region replication – will incur charges. Data transfer within the same Region does not incur charges.

Figure 1. ECR private registry

Data transfer into the Amazon ECR private registry incurs no charge from Amazon ECR. Data transferred between Amazon ECR and other services within the same Region is free. Data transferred between Amazon ECR and other services in different Regions will be charged at Internet Data Transfer rates on both sides of the transfer. Note that this is aggregated with other outbound data transfer across multiple services, and rate tiers will apply based on the amount of data transferred.

Amazon ECR offers built-in functionality to replicate container images to different locations. This could be useful for disaster recovery purposes, a promote-to-production pipeline, or to help reduce the image pull time and data transfer costs when running containers in different Regions. This data transfer is charged at the cross-Region data transfer charge rates described on the Amazon EC2 pricing page

Architecture diagram describing data transfer in to and out of ECR Public Registry. Data transfer in to ECR Public does not incur charges. Data transfer out exceeding 5 TB/month to non-AWS destinations will incur charges.

Figure 2. ECR public registry

Refer to the Amazon ECR pricing page and Amazon EC2 pricing page for more information.

Amazon ECR Registry Public

Amazon Elastic Container Registry Public is a managed AWS container image registry service that is secure, scalable, and reliable. Amazon ECR supports public image repositories with resource-based permissions using AWS Identity and Access Management (IAM) so that specific users can access your public repositories to push images. Developers can use their preferred CLI to push and pull container images and OCI-compatible artifacts. Images are publicly available to pull, either anonymously or using an Amazon ECR public authentication token.

All data transfer into Amazon ECR Public incurs no charge from Amazon ECR. Data transfer out is subject to charges when more than 5 TB/month is transferred out to non-AWS destinations, and you have authenticated to Amazon ECR with an AWS account. Up to 500 GB/month can be transferred out to non-AWS destinations to clients that have not authenticated (that is, anonymously). After that limit is reached, no further anonymous data transfer is allowed.

Data transfer for Amazon ECS

Three common deployment models for Amazon ECS are clusters with external network access, clusters without external network access, and ECS Anywhere.

Amazon ECS clusters with external access

Compute capacity for container instances in Amazon ECS can be deployed within VPCs that allow access to the Amazon ECS service endpoint using the internet. For example, the compute capacity for container instances can be deployed in public subnets (with an internet gateway), private subnets (with a NAT gateway, shown in Figure 3), or can route to the internet through another VPC using the AWS Transit Gateway. Both Amazon EC2 and AWS Fargate can be used as compute for this type of deployment, and there is no difference in data transfer costs based on which is chosen.

Diagram of sample ECS deployment with ECR, ECS Tasks, and an RDS database deployed across multiple Availability Zones. Internet access is provided via NAT Gateway in public subnets in each Availability Zone. Data transfer is charged for communication across Availability Zones, and for outbound communication to ECR and the ECS control plane.

Figure 3. ECS deployment with internet access

The following data transfer is not charged in the sample deployment:

Data transfer in from the Amazon ECS control plane (that is, responses to API calls from the data plane) and from Amazon ECR (that is, “image pulls”)
Communication between tasks and the Application Load Balancer
Communication between tasks and the database instance in the same Availability Zone

In this deployment, data transfer charges accrue for:

Data transfer out through the NAT gateway, including polling traffic from the ECS agent to the Amazon ECS control plane and outbound data to Amazon ECR
Cross-Availability Zone traffic to the database

It is important to note that although the NAT gateway does not charge for data transfer in, there will still be a per-GB data processing charge on data that flows through the NAT gateway, regardless of direction. The data transfer out charges are in addition to this charge. NAT gateway pricing can be found on the Amazon VPC pricing page.

Amazon ECS with no external access

Another common pattern to deploy Amazon ECS workloads is to restrict all external network access (Figure 4). Because container instances in Amazon ECS clusters need external network access to communicate with the Amazon ECS service endpoint, AWS PrivateLink must be used to communicate with the service endpoints. Again, both Amazon EC2 and AWS Fargate can be used for the compute capacity in this type of deployment. However, there is a difference in data transfer costs based on which is chosen.

Diagram of sample ECS deployment with ECR, ECS Tasks, and an RDS database deployed across multiple Availability Zones with no internet access. AWS PrivateLink is used to communicate to S3, ECR, and ECS. Data transfer is charged for communication across Availability Zones.

Figure 4. ECS deployment in private network

The following data transfer is not charged in the sample deployment:

Communication between tasks and the Application Load Balancer
Communication between tasks and the database instance in the same Availability Zone

In this deployment, data transfer charges accrue for cross-AZ traffic to the database.

For clarity, the diagram depicts the Amazon ECS and Amazon ECR PrivateLink VPC interface endpoints as single objects. However, there are actually multiple endpoints required for each. For a description of endpoint requirements, consult this AWS Compute Blog or the Amazon ECS and Amazon ECR documentation.

Although there is no data transfer fee, AWS PrivateLink imposes both a per-hour service charge and a per-GB data processing charge for data processed through each VPC endpoint, billed per Availability Zone. More details can be found on the AWS PrivateLink pricing page.

Using AWS Fargate in private Amazon ECS clusters

If AWS Fargate provides the compute capacity for the Amazon ECS cluster, the AWS PrivateLink endpoints for Amazon ECS are not required. This eliminates the service charges—hourly and data processing—for communication to the Amazon ECS service endpoint. Note that AWS PrivateLink VPC Endpoints for Amazon ECR are still required to pull images from Amazon ECR.

Amazon ECS Anywhere

Amazon Elastic Container Service Anywhere is a feature of Amazon ECS that enables you to easily run and manage container-based applications on-premises, including on your own virtual machines (VMs) and bare metal servers. With Amazon ECS Anywhere, you do not need to install or operate local container orchestration software, thus reducing operational overhead. ECS Anywhere offers a completely managed solution that enables you to standardize container management across all of your environments.

Architecture diagram describing ECS Anywhere over the internet. There is no charge for data transfer when communication between the Amazon ECS Anywhere control plane and ECS agent occurs over the open internet.

Figure 5. ECS Anywhere over the internet

Architecture diagram describing ECS Anywhere over the AWS Direct Connect. Standard data transfer fees apply to AWS Site-to-Site VPN or AWS Direct Connect for communication from the ECS Anywhere control plane to the (on-premise) ECS agent.

Figure 6. ECS Anywhere with Direct Connect

Data transfer fees are accrued based on how the customer-managed infrastructure connects to the Amazon ECS Anywhere control plane:

If connecting via the internet (Figure 5) – There is no charge for data transfer when communication between the Amazon ECS control plane and ECS agent occurs over the open internet.
If connecting via AWS Site-to-Site VPN or AWS Direct Connect (Figure 6) – Standard data transfer fees (that is, “Data Transfer Out”) apply to AWS Site-to-Site VPN or AWS Direct Connect for communication between the ECS Anywhere control plane and ECS agent that occurs through Site-to-Site VPN or AWS Direct Connect link.

Refer to the ECS Anywhere pricing page, AWS Site-to-Site VPN pricing page, and the AWS Direct Connect pricing page for more detail.

Data transfer for Amazon EKS

Similar to Amazon ECS, Amazon EKS data transfer charges follow the guidelines described on the Amazon EC2 pricing page. Figure 7 represents a sample Amazon EKS workload with two pods deployed to different Amazon EKS worker nodes in different Availability Zones.

Diagram of a sample Amazon EKS workload with two pods deployed to different Amazon EKS worker nodes in different Availability Zones, communicating with an RDS database. Data transfer is charged for communication across Availability Zones.

Figure 7. Sample application EKS deployment

The following data transfer is not charged in this example:

Traffic in/out of control plane (not shown on diagram)
Image pulls from ECR within the same Region (not shown on diagram)
Communication between pods and the Application Load Balancer
Communication between pods and the database instance in the same Availability Zone

The following data transfer charges accrue for this example:

Data transfer between the Kubernetes pod and the database in a different Availability Zone
Data out (egress) from the Application Load Balancer
Communication between pods in different Availability Zones

There are several other configuration options and deployment strategies to consider in an Amazon EKS deployment relative to data transfer costs: cluster access (public/private), pod-to-pod communication, and communication between the pods and the load balancer.

Public EKS clusters

Public EKS clusters (Figure 8) have the public endpoint access enabled and the private endpoint disabled. Communication between the control plane and worker nodes will exit the VPC based on the VPC’s routing. For worker nodes in private subnets, communication will traverse a NAT gateway and exit via an internet gateway. There is no data transfer charge associated with this. However, there is a per-GB NAT gateway data processing fee. In addition, worker nodes not in the same Availability Zone as the NAT gateway will incur a per-GB data transfer charge for cross-Availability Zone traffic. For worker nodes in public subnets, communication will exit the internet gateway and will not be charged.

Diagram of an EKS cluster with three nodes deployed across three Availability Zones. The nodes communicate with the EKS Control Plane via a NAT Gateway deployed in a single Availability Zone. Data transfer is charged for communication across Availability Zones (to the NAT gateway).

Figure 8. Public EKS cluster

Private EKS clusters

In a cluster with private Amazon EKS endpoints (Figure 9), worker nodes will communicate with private endpoints. The Kubernetes API Server Endpoint URL will resolve to elastic network interfaces (ENIs) within the (customer) VPC. Worker nodes inside the VPC will communicate with these ENIs. In this deployment model, there is a chance that a worker node communicates with an ENI in a different Availability Zone. This communication would be subject to a cross-Availability Zone data transfer charge.

Diagram of a private EKS cluster - three nodes deployed across three Availability Zones. Worker nodes inside the VPC will communicate with the Elastic Network Interfaces for the Kubernetes API Server endpoints. In this deployment model, there is a chance that a worker node communicates with an ENI in a different Availability Zone. This communication would be subject to a cross-Availability Zone data transfer charge.

Figure 9. Private EKS cluster

Pod-to-pod communication

Kubernetes applications often leverage communication between containers and between pods (Figure 10).

Figure 10. Pod-to-pod communication

Containers within the same pod are guaranteed to be on the same worker node and will communicate over the loopback device, resulting in no data transfer charges. Data transfer charges between pods depend on the pod placement:

There is no data transfer charge for pods deployed on the same node or within the same Availability Zone
Pods that communicate and are placed in different Availability Zones will incur a data transfer charge

Load balancer-to-pod communication

Amazon EKS workloads often include a load balancer to distribute the traffic evenly across pods. Pods are deployed alongside Kubernetes Service and Ingress objects. The add-on AWS Load Balancer Controller monitors these objects and automatically provisions the AWS load balancer resource. The path traffic takes (and resulting data transfer charges) depend on how the Service and Ingress objects are configured. There are two common methods of configuration.

The first method involves having a target group consisting of all worker nodes within the cluster (Figure 11). This grouping includes Service:LoadBalancer, Service:NodePort, and Ingress TargetType: Instance. Here, an ephemeral port (NodePort) is opened on each node in the cluster, and traffic is distributed evenly across all nodes. If the destination pod is on the node, no additional data transfer occurs. If the destination pod was scheduled on a different node, data transfer charges may accrue if the target pod is in a different Availability Zone.

Diagram of Application Load Balancer communicating to EKS pods using TargetType = “instance”. In this configuration, the traffic may be forwarded across Availability Zone, resulting in data transfer charges.

Figure 11. Load balancer-to-pod using TargetType “instance”

The second method involves a target group consisting of the pod IP addresses (Figure 12). In this method, the load balancer targets are the IP address of the pods. Communication bypasses the kube-proxy and targets the pod directly, keeping traffic in the same Availability Zone and avoiding data transfer charges.

Diagram of Application Load Balancer communicating to EKS pods using TargetType = “IP”. In this configuration, traffic stays within the same Availability Zone and does not incur data transfer charges.

Figure 12. Load balancer-to-pod using TargetType “IP”

Tips

Below are some tips to avoid excess data transfer charges and data processing charges in your container workloads:

Additional components in the network path, such as NAT gateways, AWS PrivateLink, or AWS Transit Gateway, may incur additional data transfer or data processing charges.
Review potential data transfer charges at both the source and the target of your communication channel. Remember that “Data Transfer In” to a destination is also “Data Transfer Out” from a source.
Limit cross-Region data transfer where possible. Use the Amazon ECR built-in cross-Region replication features to limit what is transferred across Regions.
Use cross-Region replication to replicate images in Amazon ECR into additional Regions and then pull images from the local Region.
Use container image build best practices and limit container images to only the essentials required to run your workload. Images with extraneous binaries cost more to store, transfer, and increase startup time.
Determine the most efficient network path for your traffic. For example, do your requirements indicate a need for a private cluster with no external connectivity? As more networking components are added, data transfer costs will increase.
Consider consolidating AWS PrivateLink endpoints in a central VPC connected by the Transit Gateway described in this AWS PrivateLink whitepaper.
In an Amazon ECS deployment with no external network connectivity, consider using AWS Fargate to host your containers. This will eliminate the need for the AWS PrivateLink endpoint for ECS.
When using a load balancer in your Amazon EKS workload, try to avoid NodePort services and target the pods directly using the IP-based TargetType for target groups.

Conclusion

In order to determine the most cost-efficient architecture for your container-based workloads, it’s important to understand how data transfer charges are calculated. Design decisions made related to compute capacity deployment, access to public AWS services, and general network architecture have an impact on data transfer charges.

Interested in learning more? Check out the blog post Overview of Data Transfer Costs for Common Architectures for an explanation of data transfer charges for common network architectures, and be sure to visit the AWS Containers Blog for more great container-related content!

Containers