AWS for Industries

Analyze Data Transfer and adopt cost optimized designs to realize cost savings

Programmatic advertising applications leverage large volumes of data (petabyte scale) to make decisions on delivering personalized experiences to users. Organizations need to have a granular visibility into their data transfer cost to make better decisions on the bidding process and leverage cost optimized design patterns. Amazon Web Services (AWS) offers different tools and techniques to gain visibility into different data transfer usage types along with design patterns to optimize data transfer cost.

In this blog, we’ll provide queries you can use to get insights from your cost and usage report (CUR) data and enable your networking teams to make decisions on networking design. We will examine two common use cases that we’ve seen with organizations. We’ll show you how to identify the top data transfer usage line items and what may be contributing to those line items. We will also explore cost optimized design patterns for both use cases. We note that minimizing data movement across networks is an important aspect of the sustainability pillar in the AWS Well architected Framework.

Setup purpose-built data transfer dashboard

The first step towards cost optimization is cost visibility with actionable findings. After working with a number of organizations across the globe, AWS teams have built the reusable Data Transfer dashboard. It is part of the cloud intelligence dashboards and is specifically in the Cost and Usage Dashboards Operations Solution (CUDOS) dashboard.

The Data Transfer dashboard is an interactive and customizable Amazon QuickSight dashboard to help customers gain insights into their data transfer. It uses CUR data from your AWS accounts that is published from AWS billing to your configured Amazon Simple Storage Service (Amazon S3) bucket. It analyzes data transfer usage such as outbound internet, regional, and inter region data transfers from different AWS services.

This dashboard contains a number of views such as:

Depending upon the type of application and application architecture, data transfer usage differs. However, following are some common data transfer usage line items we have seen:

  • Data transfer out from Amazon S3 to either same AWS region or another AWS region
  • Data transfer out from Amazon Elastic Compute Cloud (Amazon EC2)
  • AWS NAT gateway charges
  • Inter-AZ data transfer
  • Inter-region data transfer

Let’s start with deployment of the CUDOS dashboard in your AWS Account. For this, you will need:

  • Access to Management Account for creation of a CUR, an S3 bucket and S3 replication rules.
  • In the Data Collection Account permission to access Amazon Athena, AWS Glue, Amazon S3, and Amazon QuickSight through both the console and the Command Line Tool.
  • Permissions to run an AWS CloudFormation stack.

Then follow steps to deploy the dashboard. Once completed you will have access to the QuickSight CUDOS dashboard in which you can select the Data Transfer & Networking section.

Analyzing data transfer costs

After you have setup your Data Transfer dashboard, you will see a number of views. Following are several example dashboard views with example insights derived from them.

Data Transfer GB per Service
The Data Transfer GB per Service view shows the contribution of specific services to your total data transfer in GBs. In the example, Figure A, you can see that Amazon EC2 and Amazon Virtual Private Cloud (Amazon VPC) are the top two contributors to data transfer and how they are varying on a daily basis.

Figure A – View showing Data Transfer GB per ServiceFigure A – View showing Data Transfer GB per Service

Data Transfer Daily GB per Operations
The Data Transfer Daily GB per Operations view shows the contribution of specific operations to your total data transfer in GBs. An operation is a component that contributes to data transfer costs. In the example, Figure B, you can see that InterZone-In and InterZone-Out are the top two contributors to daily data transfer and their variation on a daily basis.

Figure B – View showing Data Transfer Daily GB per OperationsFigure B – View showing Data Transfer Daily GB per Operations

Identify Top Data Transfer Categories
In the Data Transfer Costs per Type view, Figure C, you can see that Inter AZ component is the highest contributor to total costs with transit gateway processing as the second highest.

Figure C – View showing Data Transfer Costs per typeFigure C – View showing Data Transfer Costs per type

Identify/Analyze Top Data Transfer Inter Region
The Data Transfer Usage in GB Previous Month view, Figure D, shows that the highest inter region transfer in GB last month happened between EU (Frankfurt) and EU (Ireland).

Figure D – View showing Data Transfer Inter RegionFigure D – View showing Data Transfer Inter Region

Identify Top Data Transfer resources from last 30 days
By looking at the first five rows in the Data Transfer Top 10 Resources Last 30 Days view, Figure E, you can see which specific resources (two Amazon VPCs, one Amazon Relational Database Service (Amazon RDS) and one Amazon EC2 instance) were the top contributors to costs in the last 30 days. There are further filters available in the dashboard (not shown) which allow you to filter by the type of resource (for example, ELB or AWS NAT Gateway). Using additional filters can help you fine tune your understanding of top resources contributing to the total cost.

Figure E – View showing Top Data Transfer resources from last 30 daysFigure E – View showing Top Data Transfer resources from last 30 days

Public IP Cost and projection
The Public IPv4 Cost and Projections Last 30 Days view in Figure F shows current Public IPv4 costs as well as projected increases. This new charge will go into effect February 1, 2024.

Figure F – View showing Public IPv4 Cost and projectionFigure F – View showing Public IPv4 Cost and projection

Along with the number of views in the CUDOS Dashboard’s Data Transfer & Networking section, you can also run custom queries to receive more insights from your CUR data. Following are a few example CUR queries that you can use as is or modify as per your needs. To learn more on how to craft CUR queries, refer to the CUR query library.

Top 15 AWS NAT Gateway usage and cost

SELECT "bill_billing_period_start_date",
"line_item_resource_id",
"line_item_operation",
CAST(SUM("line_item_usage_amount") AS INTEGER) total_usage,
CAST(SUM("line_item_unblended_cost") AS INTEGER) total_cost
FROM "<insert your database name from Athena>"
WHERE"line_item_resource_id" like '%natgateway%'
GROUP BY "bill_billing_period_start_date", "line_item_resource_id", "line_item_operation"
ORDER BY total_cost desc limit 15 ;

Top resources with Inter-AZ traffic

SELECT "bill_billing_period_start_date",
"line_item_resource_id","line_item_usage_type",
CAST(SUM("line_item_usage_amount") AS INTEGER) total_usage,
CAST(SUM("line_item_net_unblended_cost") AS INTEGER) total_cost
FROM "<insert your database name from Athena>"
WHERE"bill_billing_period_start_date" = (DATE_TRUNC('month', current_timestamp) - INTERVAL '1' MONTH)
AND "line_item_usage_type" like '%DataTransfer-Regional-Bytes'
GROUP BY 1, 2,3
ORDER BY total_cost DESC limit 15

Inter region usage and cost by resource

SELECT "bill_billing_period_start_date",
"line_item_resource_id",
"line_item_usage_type",
CAST(SUM("line_item_usage_amount") AS INTEGER) total_usage,
CAST(SUM("line_item_net_unblended_cost") AS INTEGER) total_cost
FROM "<insert your database name from Athena>"
WHERE "bill_billing_period_start_date" = (DATE_TRUNC('month', current_timestamp) - INTERVAL '1' MONTH)
AND "line_item_usage_type" like '%AWS-Out-Bytes'
GROUP BY 1, 2,3
ORDER BY total_cost DESC limit 15

Total data transfer out to internet usage and cost by resource

SELECT "bill_billing_period_start_date", "line_item_resource_id","line_item_usage_type", CAST(SUM("line_item_usage_amount") AS INTEGER) total_usage, CAST(SUM("line_item_net_unblended_cost") AS INTEGER) total_cost
FROM "<insert your database name from Athena>"
WHERE"bill_billing_period_start_date" = (DATE_TRUNC('month', current_timestamp) - INTERVAL '1' MONTH)
AND "line_item_usage_type" like '%DataTransfer-Out-Bytes'
GROUP BY 1, 2,3
ORDER BY total_cost DESC limit 15

Top S3 buckets with data transfer

SELECT "bill_billing_period_start_date",
"line_item_resource_id",
"product_from_location",
"product_to_location",
CAST(SUM("line_item_usage_amount") AS INTEGER) total_usage,
CAST(SUM("line_item_net_unblended_cost") AS INTEGER) total_cost
FROM "<insert your database name from Athena>"
WHERE "bill_billing_period_start_date" = (DATE_TRUNC('month', current_timestamp) - INTERVAL '1' MONTH)
AND "line_item_usage_type" like '%AWS-Out-Bytes'
AND "line_item_product_code"='AmazonS3'
GROUP BY 1,2,3,4
ORDER BY total_cost DESC limit 15

Now, you have insights into the top resources contributing to data transfer, top data transfer line items, AWS regions, AWS accounts and specific AWS service operations. You can use the information to define the use case and deploy a more cost optimized design pattern. Let’s discuss the two common use cases we have seen.

Use cases and cost-optimized design patterns

Use case #1 – Collection of real-time bidding logs into a central storage location
Let us explore an example of an open Real-Time Bidding (RTB) application running on Amazon EC2 compute frontend connected to a backend NoSQL database to store user profile information. When the bidding servers receive bid requests from other systems, they write information about the bid, the decision on the bid and outcome of the bid to logs.

Unoptimized architecture
Figure G shows an unoptimized architecture where log information is locally stored centrally in us-east-1 region. In Path A, the bidding servers in the private subnet in us-east-1 uses an AWS NAT Gateway to write the logs to the public S3 bucket endpoint in the same region. In Path B, the bidding servers in two private subnets from the us-west-2 region go through AWS NAT Gateways and then inter the region through the S3 bucket in the us-east-1 region. In both paths, AWS NAT Gateway is deployed in the same AZ as the Amazon EC2 instances so there are no inter-AZ charges.

Figure G – Unoptimized architecture showing real-time bidding logs collectionFigure G – Unoptimized architecture showing real-time bidding logs collection

Optimized architecture
Figure H shows the optimized architecture where there is a central S3 bucket in us-east-1 and a local staging bucket in us-west-2 with cross region replication enabled. The local bidder instance in us-east-1 is utilizing a private S3 gateway endpoint to write to the centralized S3 bucket in the same region. The bidder instances in the us-west-2 region are peered with the bidder instance in us-east-1, which hosts the S3 interface endpoint. So, the bidder instances from us-west-2 use Amazon VPC peering and then leverage the Amazon S3 interface endpoint in us-east-1 to write logs to the centralized S3 bucket.

Figure H – Optimized architecture showing real-time bidding logs collectionFigure H – Optimized architecture showing real-time bidding logs collection

Cost analysis of savings
For this use case, we’ve seen customers keep short lived logs in the same AWS region S3 bucket. Most of the logs are sent to a different AWS region S3 bucket for generating insights in order to optimize real-time bidding performance.

For this use case example let’s assume the real-time bidding application generates 300 TB of logs per month. We assume 30% of the logs are written to an S3 bucket in the same region (assuming, 90 TB) and 70% of the logs are written to an S3 bucket in a different region (assuming, 210 TB). The cost is based on current public pricing. (below calculations are based on pricing on blog publishing date)

  • Unoptimized design
    • Path A – Data flow happens from Amazon EC2 instances in private subnet to an S3 bucket in the same region. The estimated cost will be $0.045 * 90 TB = $4,050 per month
    • Path B – Data flow happens from Amazon EC2 instances in a private subnet to an S3 bucket in a different region. The estimated cost will be $0.065 * 210 TB = $13,650 per month
  • Cost optimized design
    • Path A – Data flow happens from Amazon EC2 instances in a private subnet to an S3 bucket in the same region. The estimated cost will be $0. There is no additional charge for using an S3 gateway endpoint.
    • Path B – Data flow happens from Amazon EC2 instances in a private subnet to an S3 bucket in a different region. The estimated cost will be (0.02$/GB for inter region + 0.01$/GB for the interface endpoint) * 210 TB = $6300 per month.
    • The total cost savings with the optimized design is around 64%.

Use case #2 – Programmatic bidding network connectivity between demand-side platform and supply-side platform with data transfer within the same region
For this use case let’s explore a demand-side platform (DSP) and supply-side platform (SSP) on AWS operating in the same region who will be charged 0.01$ per GB of data transfer in both directions.

Unoptimized architecture
Following in Figure I is the non-optimized architecture for a programmatic bidding network connectivity (between the DSP and SSP) with a data transfer between Amazon EC2 instances of two organizations within the same region.

Figure I – Unoptimized architecture showing programmatic bidding network connectivityFigure I – Unoptimized architecture showing programmatic bidding network connectivity

Cost optimized architecture – AdTech peer network using private Network Load Balancer
Figure J and Figure K shows an AWS PrivateLink network that enables supply-side and demand-side platforms to optimize data transfer cost and improve security. When you connect over the AWS PrivateLink service, real-time bidding traffic is routed on the AWS backbone network through a private endpoint.

Figure J – Optimized architecture showing programmatic bidding network connectivityFigure J – Optimized architecture showing programmatic bidding network connectivity

Figure K – Optimized architecture showing programmatic bidding network connectivity in more detailFigure K – Optimized architecture showing programmatic bidding network connectivity in more detail

Cost analysis of savings
Since a number of customers calculate their costs in terms of bid queries/ad requests per second, also called, QPS. We have made some assumptions to define the scale of real-time bidding application and used the numbers to calculate estimated data transfer cost. The cost is based on current public pricing.

Supply-side platform
For this use case example let’s assume a supply-side platform of 10 KB (avg bid request size) * 100k (bid queries per second to multiple DSPs) * 3600 (converting to hour) * 10 (taking 10 hours of traffic) * 30 (avg number of days in a month) = 1080 TB per month

  • Unoptimized design
    • Data flow happens from Amazon EC2 instances in the private subnet of the supply-side platform to Amazon EC2 instances in a private subnet of the demand-side platform. The cost of per GB is $0.045 per GB (AWS NAT Gateway processing charges) + $0.05 per GB data transfer out (DTO) equaling $0.095 per GB.
    • The total estimated cost of data transfer will be $0.095 per GB * 1080 TB = $102,600 per month
  • Cost optimized design
    • We are assuming that traffic goes across availability zones within the same AWS region. The cost of per GB is $0.01 (cross AZ charges) + $0.01 (Amazon VPC interface endpoint charges) equaling $0.02 per GB.
    • The total estimated cost will be $0.02 per GB * 1080 TB = $21,600 per month.
    • The total cost savings with the optimized design is around 79%.

Variation of use case #2 – Run auction instances in public subnets
There are use cases in which customers run auction instances in public subnets (instead of private subnets) along with web application firewalls to properly secure their application. This is to optimize the data transfer path and reduce AWS NAT Gateway charges.

In this scenario, supply-side platforms will still save on DTO cost when leveraging VPC endpoint design pattern. The data transfer cost for the supply-side platform running instances in public subnets will be $0.05 per GB (DTO). The total estimated cost of data transfer will be $0.05 per GB * 1080 TB = $55,296 per month. However, adopting an Amazon VPC endpoint design pattern will reduce the cost to $0.02 per GB. The total estimated cost will be $0.02 per GB * 1080 TB = $21,600 per month. This means a total cost savings of around 60.9%.

Demand-side platform
For this use case example let’s assume a demand-side platform of 10 KB (avg bid request size) * 100k (bid responses to multiple SSPs) * 3600 (converting to hour) * 10 (taking 10 hours of traffic) * 30 (avg number of days in a month) = 1080 TB per month

  • Unoptimized design
    • Data flow happens from Amazon EC2 instances in a private subnet on the demand-side platform to Amazon EC2 instances in a private subnet on the supply-side platform. The cost of per GB will be $0.05 per GB (DTO).
    • The total estimated cost of data transfer will be $0.05 per GB * 1080 TB = $54,000 per month
  • Cost optimized design
    • All the bid responses will be going through a Network Load Balancer (NLB) within a private subnet. There will be no data transfer cost, but there will be a cost of replacing the Application Load Balancer with a Network Load Balancer. Network Load Balancer is comparatively cheaper than Application Load Balancer for the same traffic.
    • Without factoring in the reduced cost of the load balancer, this means a 100% cost savings on data transfer charges for bid response traffic.

Best practices checklist

We have compiled a list of top best practices to follow to cost optimize the data transfer for your workload in AWS.

General:

  • Enable CUR and use a Cost Intelligence dashboard to analyze data transfer components—cost and usage.
  • Enable Amazon VPC flow logs to receive metadata about the actual traffic flow. This will provide an extra level of details compared to just the CUR.
  • For high amounts of data transfer consider private pricing agreements.
  • Use a Network Access Analyzer to detect unintended paths of traffic going through an AWS NAT Gateway.
  • AWS will start charging for all public IPv4 addresses effective Feb 1 2024. Refer to the announcement for more details and steps you can take to stay cost optimized.

Cost optimize Inter-AZ/regional data transfer:

  • Design applications to be AZ aware. Use AZ affinity type of functionality available in the various infrastructure services you are using. As an example, consider an Aerospike NoSQL database in which the rack awareness feature can be used to reduce cross AZ traffic when possible.
  • Use Application Load balancer and Classic Load balancer where possible as they have no cross AZ charges. This applies to traffic going to targets as well as traffic sourced from the same Amazon VPC going to the listener.
  • Note that NLB charges for cross-AZ traffic to targets. Disable NLB cross-zone load balancing where not needed.
  • Use an S3 AWS PrivateLink gateway endpoints (removing AWS NAT Gateway) for private subnet in your Amazon VPC to S3 bucket traffic within the same AWS Region.
  • Amazon Route 53 Resolver Domain Name System resolution for Amazon Elastic File System endpoint returns the endpoint IP based on the AZ the query came from (AZ affinity), avoiding cross-AZ costs for the customer.
  • Use Amazon VPC sharing, if using multiple account IDs in the same AZ ID, to reduce data transfer costs.

Cost optimize Inter region data transfer:

  • Use private link network to connect to your Ad partners in the same region.
  • Use private link interface endpoint to reduce complexity of networking between regions.

Cost optimize data transfer out:

  • Research if using AWS Direct Connect instead of the internet to talk between your AWS and on-premises environments is right for your use case. It’s usually multiple times less expensive.
  • Note that adding AWS Transit Gateway for on-premises connectivity adds a data processing fee. Avoid AWS Transit Gateway for hybrid connectivity where expected data transfer volumes are high.
  • Use edge service like Amazon CloudFront for static content delivery. Data transfer out to CloudFront is at no charge. CloudFront DTO is slightly cheaper than Amazon EC2 DTO.
  • Use Amazon VPC flow logs (Amazon S3 + Amazon Athena) to identify traffic sent to Amazon Public IP ranges. This would indicate your Amazon VPC might not be using endpoints and is sending data out through the AWS NAT Gateway/internet gateway instead. Read Analyzing Amazon VPC Flow Log data with support for Amazon S3 as a destination for guidance.
  • Use local zones and/or outposts as appropriate to further bring down the latency when talking to your partner.
  • Create an AWS NAT Gateway per AZ, as per the article How can I reduce data transfer charges for my NAT gateway? This could help save on cross-AZ costs (note: there is a per hour charge for each AWS NAT Gateway).

Conclusion

Having the correct granular visibility into your data transfer cost is important in helping you make better decisions on the bidding process while leveraging cost optimized design patterns. To this end, we showed how getting the right visibility into your data transfer charges can uncover significant cost savings opportunities. We discussed some of the dashboard views you can use and some cost and usage report queries to run in order to gain insights into your data transfer usage and costs. We also discussed the most common use case architecture design patterns in programmatic real-time bidding and illustrated how you can save using optimized design patterns.

Determine your particular use case and start saving today.

Contact an AWS Representative to know how we can help accelerate your business.

Further Reading

TAGS:
Akshay Karanth

Akshay Karanth

Akshay is a Sr Solutions Architect at AWS. He helps digital native businesses learn, build, and grow in the AWS Cloud. Before AWS, he has worked at companies such as Juniper Networks and Microsoft in various customer facing roles across networking and security domains. When not at work, Akshay enjoys hiking up a hard trail or cooking a fulfilling meal with his family.

Anuj Gupta

Anuj Gupta

Anuj Gupta is a Principal Solutions Architect working with digital-native business customers on their cloud-native journey. He is passionate about using technology to solve challenging problems and has worked with customers to build highly distributed and low-latency applications. He also contributes to open-source solutions. Outside of work, he loves traveling with his family and meeting new people.