Announcing AWS CloudTrail Lake one-year extendable retention pricing option

In 2022 Amazon Web Services (AWS) released AWS CloudTrail Lake, a managed audit and security lake that allows you to aggregate, immutably store, visualize, and query your activity logs for auditing, security investigation, and operational troubleshooting. Working backwards from our customers we have added capabilities to CloudTrail Lake such as the ability to copy CloudTrail events into your event data store, curated dashboards for visualizing top trends, and deeper integrations with other services like AWS Config and AWS Audit Manager to name a few.

Today, we are excited to announce the availability of a new one-year extendable retention pricing option for CloudTrail Lake which gives you a more cost effective way to use CloudTrail Lake if your monthly usage is less than 25 TB. This new pricing option will also provide a lower cost when importing your historical CloudTrail data into CloudTrail Lake. With this new pricing option, your first year of retention is included with your ingestion cost. You can also choose to extend your retention period, at a cost, for up to 10 years. You will still be able to choose the seven-year retention pricing option, which is recommended if your monthly usage exceeds 25 TB.

In this post, we will take a closer look at the new pricing option and demonstrate how you can estimate your monthly cost based on your current CloudTrail usage. We will then show you how to update an existing CloudTrail Lake event data store to the new one-year extendable retention pricing option. Last, we will review some common examples that can provide guidance on which pricing option is more cost effective, based on your usage and retention.

A closer look at the new one-year extendable retention pricing option

With our one-year extendable retention pricing, the first year of retention is included with your CloudTrail Lake ingestion cost. You can also extend your retention period to a maximum of 10 years by paying extended retention charges beyond the first year. Your cost for ingestion would depend on the type of events that will be stored in CloudTrail Lake. For example ingestion of events from CloudTrail management and data events would be $0.75 per GB and events from AWS data sources like AWS Config, importing historical CloudTrail data from Amazon S3, and non-AWS auditable data sources would be $0.50 per GB, providing a lower cost than the entry tier seven-year retention pricing of $2.50 per GB. As mentioned previously your first year of retention is provided at no additional cost to ingestion. However, if you choose to extend your retention period beyond a year, extended data retention is available for $0.023 per GB per month of retention. For additional information on CloudTrail Lake pricing please take a look at our pricing page. With these two pricing options, you might be wondering: how can you estimate your cost and choose the most cost effective option?

How to estimate CloudTrail Lake cost with the new pricing option

In this section, we will take a look at how to estimate the cost of using CloudTrail Lake using your current CloudTrail Trails usage. In the following example we will show you how to gather the amount of CloudTrail recorded management and data events from a previous AWS bill to then estimate the cost of delivering these types of events to an event data store for CloudTrail Lake using our new one-year extendable retention pricing.

Open the AWS Billing and Cost Management console. Then, choose Bills.
Choose the Date of a month from your previous bill.
In AWS Services Charges, expand CloudTrail.
Expand the AWS Region to view the event cost record details. Then, sum up the total of FreeEventsRecorded and DataEventsRecorded to identify the total amount of events recorded.

Getting event data store size using Amazon CloudWatch Figure 1 – Use FreeEventsRecorded & DataEventsRecorded to estimate CloudTrail Lake cost.

Example of FreeEventsRecorded & DataEventsRecorded
Total Sum	4,409,980.00

Use the following expression to convert the total number of FreeEventsRecorded & DataEventsRecorded to the amount of storage in GB for CloudTrail Lake

([Number of Total Events] x 1500 (Bytes))/ 1000000000 (Bytes)

Note: 1500 Bytes is the average size of a CloudTrail event.

For instance if we use the total amount from the example above we will have the following expression

(4,409,980.00 x 1500)/ 1000000000 = 6.61 GB)

We can then calculate an estimated cost of using CloudTrail Lake for ingesting CloudTrail management and data events. Below is the estimated cost from our example:

6.61 GB x $0.75 per GB = $4.96

Note: Ingestion from AWS and non-AWS auditable data sources, excluding CloudTrail management and data events: $0.50/GB.

Below is the estimated cost for data ingestion using the one-year extendable retention pricing vs the seven-year retention pricing, showing the cost effectiveness of the one-year extendable retention pricing option.

One-year extendable retention pricing	Seven-year retention pricing
6.61 GB x $0.75 per GB = $4.96 (data ingestion)	6.61 GB x $2.5 per GB = $16.53

In this section we demonstrated how to estimate your ingestion cost for CloudTrail Lake using the new one-year extendable retention pricing. In the next section, you’ll see how to also estimate your monthly extended retention cost, beyond the first year of retention included in your ingestion cost.

How to estimate your CloudTrail Lake extended retention cost

When creating an event data store with a retention period extending beyond one year there will be an additional monthly cost for extended retention along with the ingestion cost for new CloudTrail events being delivered to CloudTrail Lake. Let’s take a look at how to calculate your current event data store storage size to then create an estimate on what the monthly cost for your extended retention would be. In the following example we will demonstrate how you can view the storage size of your event data store using Amazon CloudWatch. Then, we will show you how you can calculate the extended retention cost for CloudTrail Lake.

Metric math for CloudWatch allows you to query multiple CloudWatch metrics and use math expressions to evaluate the values for these metrics. To create a CloudWatch metric to show the total storage usage for your CloudTrail Lake event data store, use the following steps:

Navigate to the CloudWatch console.
In the left navigation menu, select Dashboards.
Choose Create dashboard.
Name the dashboard aws-cloudtrail-lake-dashboard and choose Create dashboard.
Select Number graph and then choose by Next.
Select Metrics and choose Next.
In the right hand section, select Add math and then select Start with empty expression.
Enter in the below math expression and choose Apply.

SORT(SEARCH('{AWS/CloudTrail,"Event data store ID","Lake Metrics"} MetricName="TotalPaidStorageBytes" NOT "Lake Metrics"="IngestionMetrics"',"Sum"),SUM, DESC)

In the Period column, change the value to 1 day.
Under the Label column, change the name to Event Data Store.
In the upper left hand corner, Rename the graph to CloudTrail Lake Event Data Store Size.
Choose Create widget.
Choose Save, to save the dashboard.

CloudTrail Lake event data store size in CloudWatch dashboard
Figure 2 – CloudWatch metric to show the total storage usage for an event data store.

Use the following expression to convert the total size in MB to GB for CloudTrail Lake

([Total Size] (MB))/ 1000 (MB)

We can then calculate an estimated cost of the extended retention for CloudTrail Lake. Below is the estimated cost for the month from our example:

0.512 GB x $0.023 per GB per month = $0.011776

If we include our initial cost estimate of data ingestion together with the extended retention cost for events past the first year, we can then compare it with the cost of the seven-year retention pricing. This helps us see the cost effectiveness of using the new pricing option.

One-year extendable retention pricing

Seven-year retention pricing

6.61 GB x $0.75 per GB = $4.96 (data ingestion)

6.61 GB x $2.5 per GB = $16.53

0.512 GB x $0.023 per GB per month = $0.011776

(extended retention after first year)

Total: $4.18

Note: This doesn’t include any data scanned running CloudTrail Lake queries.

In this section we were able to see how we can estimate the extended retention cost past the first year within our event data store for CloudTrail Lake. Now that we have a better understanding on how the new pricing option works, let’s see how we can update an existing event data store over to the new one-year extendable retention pricing option.

How to update an existing CloudTrail Lake event data store to the new pricing option

If you are currently using CloudTrail Lake, you can choose to update your existing event data store configuration to the new one-year extendable retention pricing option. We will walk through the steps in updating your event data store for CloudTrail Lake.

Navigate to the CloudTrail console .
In the left-hand navigation menu, choose Lake.
Choose Event data stores.
Choose your event data store.
Choose Edit under the General details section.
Under the Pricing Options section you can choose the “one-year extendable retention pricing”.
Under the Retention period section, the default duration is set to the value that was initially created for the event data store.

Event data store configuration screen Figure 3 – Configuration screen to update an existing event data store.

Note: You cannot switch an existing event data store from the one-year extendable retention pricing to the seven-year retention pricing.

Choose Save changes.

If you would like to learn how to create a new event data store for CloudTrail Lake using the one-year extendable retention pricing take a look at Create an event data store – AWS CloudTrail. Additionally, you have the option to copy trail events to CloudTrail Lake. However, the retention period for both pricing options will apply to the date of the event when it was recorded and not when it was copied. For example, if you were to copy a trail event from 6 months ago to a newly created event data store with a 3 year retention period, you would only have 2 1/2 years remaining. Now that we have seen how to update an existing event data store over to the new pricing option, let’s take a look at a couple of examples where using the new one-year extendable retention pricing will provide flexibility and be more cost effective.

Investigating security, compliance, and audit events

The addition of the new pricing option will provide you the flexibility to right-size for each use case you may have. For monthly ingestion usage needs below 25 TB, the one-year extendable retention pricing is recommended. Where higher amounts of CloudTrail event data is expected you could use the seven-year retention pricing option. For this example, let’s say you are centrally collecting event logs for all AWS accounts and regions and are expecting to ingest 50 TB of data. When you create an event data store in AWS CloudTrail Lake you can choose all of the accounts in your AWS Organization and all activity in all AWS regions. You can use CloudTrail Lake to centrally analyze data events for security, compliance, and audits by collecting aggregated event activity which you can query using SQL. To help you get started quickly you can take advantage of the provided queries in the CloudTrail lake console or from the sample queries in our repository. You may be preforming investigations to identify activities on S3 resources using data events for Amazon S3 objects. With CloudTrail Lake, event activity can be queried without the need for extra processing or sending logs downstream to another service or application. In this example, where we are ingesting 50 TB of management and data events into an event data store, the seven-year retention pricing option would result in 31% savings.

You may also want to investigate other data such as AWS Config configuration items, historical CloudTrail event logs imported from S3 to CloudTrail Lake, or non-AWS auditable sources. You can create these with the one-year extendable retention pricing option ($0.50 per GB). We recommend using this option when your monthly usage is less then 25 TB. Let’s say that you are expecting 200 GB of configuration items from AWS Config; this would result in about $100 of ingestion cost for one year. We know that different teams often have different requirements and need to be provided with different retention periods to meet compliance regulations or organizational policy. The cost is the same for importing historical event logs from CloudTrail or from non-AWS sources and the event data stores can be created either centrally or at the individual account level.

CloudTrail Lake can be a cost effective approach when you need to give individual teams access to CloudTrail event data. In order to maintain security and compliance best practices, as well as free tier eligibility for AWS CloudTrail, you may opt to configure an organizational trail that logs all events for all AWS accounts in that AWS Organization. While this design allows you to centralize logs into a security or audit account, it doesn’t provide the user of each aggregated account access to the CloudTrail event data. Since CloudTrail Lake is a managed service, it removes complexity when access to CloudTrail event data is needed outside of the centralized log location. By using Identity and Access Management (IAM), you can scope the right permissions for each team to meet your unique business requirements.

One common use case for account level event data stores is retaining logs for 1 year to satisfy PCI-DSS requirements or to perform quick investigations. In this scenario we will assume that you will be expecting 1 TB of CloudTrail data which you estimated by using the method we highlighted earlier in this post. For the ingestion and storage of 1 TB (1024 GB) of CloudTrail events using the one-year extendable retention pricing option ($0.75 per GB) we can approximate a spend of $770, see the CloudTrail pricing page for more details. Logs can be stored as dictated by your compliance requirements or organizational policies. If you have longer term requirements, you can opt for the seven-year retention pricing option of $2.5 per GB up to 5 TB, next 20 TB at $1 per GB, and next 25 TB at $0.5 per GB. For example, if you use the seven-year retention pricing option for 3000 GB, you can estimate a cost of approximately $7500.

Clean up

Navigate to the CloudWatch console.
Select the aws-cloudtrail-lake-dashboard dashboard and choose Delete.

Conclusion

In this blog post, we dove deep into the new one-year extendable pricing option, provided methods to estimate your CloudTrail Lake cost, and gave you examples of when to use one of our two pricing options. With the new pricing option offered by AWS CloudTrail Lake, you can have more flexibility to choose the most cost effective approach for storing your data in CloudTrail Lake. To learn more about CloudTrail Lake and its capabilities see some of the content below:

AWS Cloud Operations & Migrations Blog