Using AWS Cost and Usage Reports and Cost Allocation Tags to understand VPC Flow Logs data ingestion costs in Amazon S3
AWS customers enable the VPC Flow Logs feature in their accounts for security, governance, and auditing. They often have several teams who create VPC flow log subscriptions for their workloads and publish the data to the same Amazon Simple Storage Service (Amazon S3) bucket as part of a centralized logging architecture. Customers need a way to categorize their bills so they can accurately charge back those internal teams who create VPC flow log records. Customers who are considering the use of VPC flow logs often don’t know what their costs will be. Creating a few flow logs and tracking charges can help customers accurately estimate their costs before they enable the feature across their environment.
With the recent support of cost allocaton tags for VPC flow logs, customers can now tag their VPC flow logs to track charges from publishing flow logs to S3 buckets. To track charges from publishing flow logs to Amazon CloudWatch Logs, customers can apply cost allocation tags to their destination log group.
In this post, we will provide steps for using cost allocation tags to analyze VPC flow log costs and charge back the responsible business unit.
The setup consists of the following configuration steps on the AWS console:
- Setting up VPC flow logs with S3 as the destination.
- Adding tags to the VPC flow logs.
- Activating the user defined cost allocation tags.
- Analyzing the VPC flow logs in the AWS Cost & Usage Report.
Create a VPC flow log
1. In the Amazon VPC console, choose your VPC.
2. On the Flow logs tab, choose Create flow log with S3 as the destination. Repeat these steps for as many VPCs as you like. To create a flow log and setting the destination to an Amazon S3 bucket you can follow the steps in our documentation.
Add tags to VPC flow logs
1. Select the flow log and then from Actions, choose Manage tags.
2. For chargeback, add tags to indicate the Line of business unit (LBU) that owns this VPC. Keep the tag name consistent across all VPC flow logs. In Figure 2, the tag name is cost-center.
Activate user-defined cost allocation tags
1. Open the Billing and Cost Management console and from the left navigation pane, choose Cost allocation tags.
2. Under User-defined cost allocation tags, search for your tag name and then activate the tag. This ensures that Cost Explorer and your AWS Cost and Usage Reports will include these tags.
Analyze VPC flow log costs in the AWS Cost and Usage Report
1. Follow the steps in Querying Cost and Usage Reports using Amazon Athena.
2. Use the following SQL query to analyze the cost of VPC flow logs per cost center.
SELECT SUM(line_item_usage_amount) AS total_usage_amount, SUM(line_item_unblended_cost) AS total_cost, line_item_product_code, line_item_operation, product_usagetype, line_item_line_item_description, resource_tags_user_cost_center FROM "cur_db"."cur_data" WHERE line_item_usage_type like '%S3-Egress%' AND line_item_product_code like '%AmazonCloudWatch%' AND line_item_operation like '%LogDelivery%' GROUP BY resource_tags_user_cost_center, line_item_product_code, line_item_operation, product_usagetype, line_item_line_item_description
AWS Cost Explorer view
Customers can also use the filters in AWS Cost Explorer to view VPC flow log charges grouped by cost allocation tags.
In this blog post, I’ve shown how customers can analyze VPC flow log data ingestion costs using cost allocation tags and AWS Cost and Usage Reports. This will help customers to chargeback business units for the VPC Flow Logs that they are generate. While implementing tagging also be sure to follow AWS tagging best practices.
About the authors
Kalyan Janaki is Senior Technical Account Manager with AWS. Kalyan enjoys working with customers and helping them migrate their workloads to the cloud. In his spare time, he tries to keep up with his 3-year-old.
Abhishek Lahiri is Senior Technical Account Manager with AWS. Abhishek helps large enterprises build secure and scalable solutions on the AWS cloud, helping them with cloud transformation initiatives.