AWS Cloud Operations Blog

Unlock deeper insights and faster investigations with AWS CloudTrail Lake

AWS recently introduced several new enhancements to AWS CloudTrail Lake, a managed data lake that helps customers capture, immutably store, access, and analyze their AWS and non-AWS activity logs, as well as AWS Config configuration items. These latest updates provide you with powerful new tools to streamline analysis, uncover anomalies, and accelerate investigations of your data stored within your event data stores.

The enhancements include:

  • New enhanced filtering options for CloudTrail events ingested into event data stores
  • AI-powered natural language query generation
  • AI-powered query result summarization (preview)
  • Cross-account sharing of event data stores
  • Additional comprehensive dashboard capabilities, including:
    • A new dashboard that provides an overview of your AWS activity logs captured within your event data stores, with AI-powered insights (the AI-powered insights feature is currently in preview)
    • A suite of 14 pre-curated dashboards for different use cases such as security, compliance, and operational monitoring
    • The ability to create your own custom dashboards with scheduled refreshes

Enhanced event filtering

AWS CloudTrail Lake is enhancing its event filtering capabilities, giving you greater control over which CloudTrail events are ingested into your event data stores. This update increases the efficiency and precision of your security, compliance, and operational investigations while reducing analysis workflow costs by allowing you to ingest only the most relevant event data for your use case.

You can now filter both AWS CloudTrail management and data events by attributes such as event name, event type, event source, the IAM entity that made the request (identified by the userIdentity.arn field), or if the event originated from an AWS Management Console session. The event source filter allows you to include or exclude any AWS services. The userIdentity.arn field contains the full Amazon Resource Name (ARN) of the IAM principal that made the request, including the user, role or service involved. For each of these attributes, you can choose to either include or exclude specific values.

These new filters empower you to be more selective in the data that you capture. For example, you can now filter CloudTrail events based on the userIdentity.arn field to exclude events generated by specific IAM roles or users. You can exclude a dedicated IAM role used by a service that performs frequent API calls for monitoring purposes. This allows you to significantly reduce the volume of CloudTrail events ingested into CloudTrail Lake, lowering costs while maintaining visibility into relevant user and system activities.

Enhanced event filtering is available in all AWS Regions where AWS CloudTrail Lake is supported, at no additional charge.

AI-powered natural language querying

AWS CloudTrail Lake now offers powerful new capabilities that greatly simplify the exploration of your AWS activity data. This feature empowers you to ask questions in plain English about your AWS API and user activity, without having to write complex SQL queries. For instance, you can inquire “What S3 bucket policy changes were made in the last week?” or “Show me all events where users were given administrative access in the last 24 hours”. CloudTrail Lake will then automatically generate the corresponding SQL queries, streamlining the analysis of your AWS activity data.

Please note that running queries will incur CloudTrail Lake query charges. Refer to AWS CloudTrail pricing for details.

AI-powered query result summarization (preview)

Building on the natural language query generation, we are further simplifying your experience of analyzing your AWS account activity. We are introducing an AI-powered query result summarization capability in preview. This feature significantly reduces the time and effort required to extract meaningful information from your AWS activity logs, leading to quicker and more informed decisions.

With this feature, you can simply click the Summarize results button for your lake query results, and CloudTrail will automatically analyze the data and provide a natural language summary of the key insights. For example, if you ran a query to find all access denied events that occurred yesterday, you may receive a result with hundreds of events. Instead of reviewing each event individually, you can get a quick summary of the output by leveraging the AI-powered summarization feature. It would analyze the results and provide a natural language summary such as:

“The query results show multiple instances of access denied errors for various Amazon S3 operations. The majority of these events involve an assumed role “Role1” attempting to perform actions on the “examplebucket-1a1b2c” bucket. These actions include PutObject, HeadBucket, and GetBucketEncryption, all of which were denied due to lack of permissions. There were also access denied events for anonymous AWS accounts trying to list objects in different buckets. Additionally, there was an instance of an assumed role “Role2″ being denied access to perform GetObject on a bucket related to AWS QuickSetup patch policies. These events indicate potential misconfigurations in IAM policies or bucket permissions that need to be addressed.”

This AI-powered summarization capability is being introduced in preview at no additional cost.

Figure 1: CloudTrail Lake AI-powered natural language querying and query result summarization

Cross-account sharing of event data stores

AWS CloudTrail Lake now enables you to securely share your event data stores with selected principals (such as AWS accounts, IAM users, or roles) using Resource-Based Policies (RBP). Authorized principals can then query the shared event data store within the same AWS Region where the event data store was created. This capability facilitates analysis across different accounts while maintaining control over who can access your event data stores through fine-grained permissions. Additionally, it offers cost efficiency by reducing duplication of data across accounts, potentially lowering storage costs.

This feature is available in all AWS Regions where AWS CloudTrail Lake is supported, at no additional cost.

Comprehensive dashboard capabilities

We are introducing a new Highlights dashboard in AWS CloudTrail Lake that offers an overview of your AWS activity logs (management and data events) for the last 24 hours. This dashboard is refreshed every 6 hours, keeping you up to date with recent data.

It includes an Activity Overview section displaying key metrics such as Regions accessed and failed API calls. This dashboard includes generative AI-powered insights, currently in preview, which provide a natural language summary of account activities, helping you quickly understand patterns and anomalies. Additionally, the dashboard presents interactive visualizations for critical security and operational metrics, like failed console login attempts and throttled API calls. Each visualization allows you to view and edit the underlying query, enabling deeper investigation when needed.

In addition, AWS CloudTrail Lake now offers a suite of 14 pre-built dashboards designed for different personas and use cases including those tailored for security, enabling you to track and analyze key security indicators, such as top access denied events, users who disabled multi-factor authentication, and more. These pre-built dashboards also include AWS service-specific dashboards for services like Amazon EC2 and DynamoDB, allowing you to identify security risks or operational problems within those environments. These dashboards give your teams a streamlined starting point to analyze trends, detect anomalies, and conduct more efficient investigations across your AWS environments.

You are not limited to just the pre-built dashboards. AWS CloudTrail Lake also allows you to create your own custom dashboards tailored to your specific monitoring and investigation needs. When adding visualizations to these dashboards, you can either select from a library of pre-curated visualizations or write your own query and choose how to present the results.

This dual approach allows you to rapidly build insightful dashboards using pre-curated options while also providing the flexibility to create highly specific, customized views of your data. You can choose from presentation options including bar charts, line charts, pie charts, or tables. Additionally, you can set refresh schedules for these dashboards, ensuring your visualizations stay up-to-date.

These new dashboard capabilities are available in all AWS Regions where AWS CloudTrail Lake is supported, except for the AI-powered insights feature on the Highlights dashboard, which is currently available in preview in select regions. While these enhancements are available at no additional cost, standard CloudTrail Lake query charges apply when running queries for visualizations in the CloudTrail Lake dashboards. For more information on pricing, visit AWS CloudTrail pricing.

Figure 2: CloudTrail Lake Security monitoring dashboard

Conclusion

These powerful new features in AWS CloudTrail Lake represent a significant step forward in delivering a comprehensive audit log and analysis solution by enabling deeper insights and quicker investigations for proactive monitoring and faster incident response across your AWS environments.

Getting started

For detailed instructions on how to set up and access these new AWS CloudTrail Lake features, I encourage you to check out our news blog Introducing new capabilities to AWS CloudTrail Lake to enhance your cloud visibility and investigations. This step-by-step guide provides practical examples and screenshots to help you quickly leverage these enhancements in your own environment.

For more information on these new AWS CloudTrail Lake features, please refer to the following resources:

About the author

Darpa Sehgal

Darpa Sehgal

Darpa Sehgal is a Senior Product Manager (Technical) at AWS. She specializes in cloud operations and governance, helping customers strengthen their security and compliance posture, and achieve operational excellence in the cloud.