Introducing AWS Security Analytics Bootstrap

Organizations running workloads in Amazon Web Services (AWS) often must search and analyze logs to troubleshoot or investigate operations, governance, or security events. Amazon Athena enables AWS customers to search and analyze log data directly from in Amazon Simple Storage Service (Amazon S3) using standard SQL queries. Additionally, we understand that customers need a common environment for security investigation use cases that is easy to set up, deploy, and maintain.

In this article, we provide an overview of AWS Security Analytics Bootstrap, which lets customers perform security investigations on AWS service logs through an Amazon Athena analysis environment.

Overview

AWS Security Analytics Bootstrap is an open source framework designed for AWS customers who need a quick method to set up Athena and perform investigations on AWS service logs archived in Amazon S3 buckets. AWS Security Analytics Bootstrap was designed to provide AWS customers with common requirements for security use cases, such as:

Table schemas for logs commonly used in security use cases (for example, AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) flow logs, Amazon Route 53 DNS Resolver Logs).
Support for dynamic partitioning by account, region, resource, or date.
Athena workgroup configuration to manage Athena configuration and output storage.
Example SQL queries to get started quickly.
AWS Identity and Access Management (IAM) roles that follow the principle of least privilege.
The ability to deploy everything as infrastructure as code (for example, AWS CloudFormation).
No code to maintain.

The AWS Security Analytics Bootstrap tool provides AWS CloudFormation templates for creating a fully configured Athena analysis environment, including an Amazon Athena workgroup, AWS Glue databases, AWS Glue tables, and demo Athena queries. Each table schema supports common security investigation requirements, including partitioning and searches across multiple accounts, regions, and dates.

AWS Security Analytics Bootstrap uses partition projection with Amazon Athena to provide dynamic partitioning across accounts, regions, and dates without any additional infrastructure, code, or frequent maintenance. Partitioning AWS service log data by account, region, and date allows AWS customers to create targeted queries and reduce their cost and query times.

Common use cases for AWS Security Analytics Bootstrap include situations when AWS customers:

Want to search AWS service logs natively in AWS.
Currently have no security information and event management (SIEM) available or accessible.
Need to search logs beyond the SIEM retention period.
Don’t have the required AWS service logs indexed in the SIEM.
Need to investigate an AWS account that hasn’t been centralizing its logs.
Need quick and lightweight log investigation environment when starting their security journey.

AWS Security Analytics Bootstrap currently supports the AWS service logs shown in the following table:

	AWS service log	Description
1	AWS CloudTrail	AWS API events (management and data events)
2	Amazon Virtual Private Cloud (VPC) Flow Logs	Network events
3	Amazon Route 53 DNS resolver query logs	DNS resolution events

The following architecture diagrams show the two ways in which AWS Security Analytics Bootstrap can be deployed. AWS Security Analytics Bootstrap supports both a single account deployment, where it is deployed in the same account as the Amazon S3 logging buckets, and a cross-account deployment, where it is deployed in a different account than the Amazon S3 logging buckets.

Deploying AWS Security Analytics Bootstrap

Prerequisites

AWS service logs (for example, AWS CloudTrail, Amazon VPC Flow Logs, Amazon Route 53 resolver query logs) must be delivered to Amazon S3 buckets unmodified in their native format.
For cross-account deployments, bucket policies must be in place, and objects must be owned by the bucket account owner to allow cross-account access.
For logs encrypted via AWS Key Management Service (KMS), the AWS Identity and Access Management (IAM) principals that will be used to submit Athena queries must have permissions for kms:Decrypt and kms:DescribeKey in their IAM policy. The KMS key policy will need to grant them the same access.

Getting started

The Athena Infrastructure CloudFormation template will deploy a fully functional security analytics environment including the resources listed in the following table:

	Resource	Notes
1	Athena workgroup	Configured to provide encrypted output to a specified Amazon S3 location. Includes preconfigured demo queries as named queries.
2	Glue database	Contains associated Glue tables.
3	Glue tables	Standardized table schemas with dynamic partitions for account, region, and date for: CloudTrail logs VPC Flow Logs Route 53 DNS Resolver Logs

Deployment time is approximately 10 minutes.

Comments are provided in the CloudFormation parameters section to assist with the parameters required for deployment, and a detailed walkthrough of the deployment process is provided in the AWS Security Analytics Bootstrap Deployment Guide. AWS Security Analytics Bootstrap has been designed to allow rapid deployment; however, we recommend deploying it and getting experience using it so you’re prepared in case there is an urgent need.

AWS Security Analytics Bootstrap resources

The Athena Infrastructure CloudFormation template can be deployed by itself or in combination with any of the additional resources depending on customers’ use cases and requirements.

	Resource type	Resource	Resource provides	Cleanup and removal notes
1	AWS CloudFormation template	Athena Infrastructure CloudFormation template	Creates the ready-to-use Athena security analytics environment including: Athena workgroup, Glue database, Glue tables, and demo named queries. Comments are provided in the CloudFormation parameters section to walk customers through deployment, or customers can review the AWS Security Analytics Bootstrap Deployment Guide for more detail.	All resources created by this template will be deleted when the CloudFormation Stack is deleted. This will not affect the source log data.
2	AWS CloudFormation template	IAM roles and policies for Athena Admin and Athena Analyst	Creates IAM roles and policies for a Athena Admin and Athena Analyst Roles designed according to least-privilege principals.	All resources created by this template will be deleted when the CloudFormation Stack is deleted.
3	AWS CloudFormation template	Enable flow logs	Enables VPC Flow Logs for the specified VPC, subnet, or ENI with all fields through v5 in the order expected by Athena Bootstrap.	The VPC Flow Log configuration will be deleted when the CloudFormation stack is deleted. Any logs created will need to be deleted (if desired) separately from the target S3 bucket.
4	CREATE TABLE SQL statement	AWS CloudTrail table schema	Creates a Glue table for CloudTrail logs partitioned by account, region, and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation template; this SQL statement can be used to create a table in an existing Athena environment for ad hoc deployment use cases. “TODO” comments are included above sections must be updated with customers’ environment details.	The table can be deleted with the Athena query statement DROP TABLE <table name> (for example, DROP TABLE cloudtrail).
5	CREATE TABLE SQL statement	Amazon VPC Flow Logs table schema	Creates a Glue table for VPC Flow Logs partitioned by account, region, and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation template; this SQL statement can be used to create a table in an existing Athena environment for ad hoc deployment use cases. “TODO” comments are included above sections must be updated with customers’ environment details.	The table can be deleted with the Athena query statement DROP TABLE <table name> (for example, DROP TABLE vpcflowlogs).
6	CREATE TABLE SQL Statement	Amazon Route 53 Resolver Query Logs table schema	Creates a Glue table for Route 53 DNS Resolver Logs partitioned by account, VPC ID, and date via Athena SQL query statement. This table is also created by the Athena Infrastructure CloudFormation template; this SQL statement can be used to create a table in an existing Athena environment for ad hoc deployment use cases. “TODO” comments are included above sections that must be updated with customers’ environment details.	The table can be deleted with the Athena query statement DROP TABLE <table name> (for example, DROP TABLE r53dns).
7	Demo Athena queries	AWS CloudTrail demo queries	Demo Athena queries for CloudTrail logs. These queries area also created in the Athena workgroup as named queries by the Athena Infrastructure CloudFormation template.	N/A: No resources created
8	Demo Athena queries	Amazon VPC Flow Log demo queries	Demo Athena queries for VPC Flow Logs. These queries area also created in the Athena Workgroup as named queries by the Athena Infrastructure CloudFormation template.	N/A: No resources created
9	Demo Athena queries	Amazon Route 53 Resolver Query Log demo queries	Demo Athena queries for Route 53 DNS Resolver Logs. These queries area also created in the Athena workgroup as named queries by the Athena Infrastructure CloudFormation template.	N/A: No resources created

As development continues on this tool, the following items are currently out of scope:

How data is provided to Amazon S3 buckets (for example, configuration of logs).
Optimization of the underlying data sources (for example, merging small files, converting to Parquet/ORC columnar formats).
Bucket policy updates for cross-account deployments.
AWS KMS policy updates.

By installing AWS Security Analytics Bootstrap, AWS customers may incur charges from the following services:

Conclusion

In this post, we’ve explained how AWS Security Analytics Bootstrap framework helps AWS customers investigate common operational, governance, or security issues. The project is licensed under the Apache 2.0 License, and you can find more information about its use and how to contribute on GitHub.

AWS Open Source Blog