AWS Cloud Operations & Migrations Blog

Visualizing AWS CloudTrail Events using Kibana

In this blog post you learn how to visualize AWS CloudTrail events, near real time, using Kibana. This solution is useful if you use an ELK (Elasticsearch, Logstash, Kibana) stack to aggregate logs from all your systems and applications, analyze these logs, and create visualizations for application and infrastructure monitoring. This solution is also useful for faster troubleshooting and security analytics.

In this solution, we replace the Logstash portion of the ELK stack with AWS native solutions to stream CloudTrail events to an Amazon Elasticsearch (Amazon ES) domain. Because the cost of your Amazon ES cluster increases as log data grows, you may want to use cheaper storage tiers within the Amazon ES leveraging the UltraWarm feature. For more information about Amazon ES cluster costs, see Amazon ES pricing.

CloudTrail enables governance, compliance, operational auditing, and risk auditing of your AWS account. You can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. Kibana is a popular open-source visualization tool designed to work with Elasticsearch. Amazon ES provides an installation of Kibana with every Amazon ES domain. Offering a Kibana dashboard to your security organization to continuously monitor the CloudTrail logs helps simplify operational analysis and troubleshooting compliance issues.

Solution Overview 
In this solution, we send the CloudTrail events to Amazon CloudWatch Logs. Then we stream the logs to an Amazon ES cluster in near-real time, through a CloudWatch Logs subscription. When the logs are streaming to the Amazon ES cluster, you can access the Kibana endpoint to visualize the data.

The following diagram illustrates the architecture of this solution.

CloudTrail Events sent to CloudWatch Logs. CloudWatch Logs trigger Lambda function to send the Trail Events to an Amazon Elasticsearch Index. Finally, Kibana to create the near real-time dashboard

 

 

 

 

Prerequisites
For this tutorial, you should have the following prerequisites:

  • An AWS account
  • An IAM user with access to AWS resources used in this solution

High-level approach
CloudTrail is enabled on your AWS account when you create it. You can use the Event history page on the CloudTrail console to view, search, download, archive, analyze, and respond to account activity across your AWS infrastructure for the past 90 days. This includes activity made through the AWS Management Console, AWS CLI, AWS SDKs, and programmatically.

To implement this visualization solution using Kibana, you complete the following high-level steps:

  1. Create a CloudTrail trail for an ongoing record of events in your AWS account.
  2. Send the CloudTrail events to an CloudWatch Logs log group.
  3. Configure your trail to send events to an Amazon ES domain in near-real time.
  4. Create an Amazon ES domain to store the CloudTrail logs, which contain trail events to Amazon ES
  5. Visualize the CloudTrail events using Kibana

Creating a CloudTrail trail
You can create, update, or delete your trails on the CloudTrail console or using the AWS CLI. For this post, you use the CloudTrail console. For instructions, see Creating a Trail. You can create up to five trails for each Region. After you create a trail, CloudTrail automatically starts logging API calls and related events in your account to the Amazon S3 bucket that you specify. To stop logging, you can turn off logging for the trail or delete it.  You can set up a trail that delivers a single copy of management events in each Region free of charge. Once a CloudTrail trail is set up, Amazon S3 charges apply based on your usage.

When creating your trail, provide the following information:

  • Trail name – myblog-all-events (or name of your choice)
  • S3 bucket for storing logs – blog-cloudtrail-events (or S3 bucket of your choice)
  • SSE-KMS encryption – Use an existing key or create one based on your needs

You also send the CloudTrail events to a CloudWatch log. For instructions, see Sending Events to CloudWatch Logs. Specify a log group name, which you use on the CloudWatch console later in this post to send the trail events to Amazon ES. We use the name CloudTrail/MyBlogLogGroup.

By default, AWS CloudTrail trails log all management events, and don’t include data or CloudTrail Insights events. Management events capture management operations, data events show the resource operations performed on or within a resource, and CloudTrail Insights helps you identify and respond to unusual activity associated with write API calls. You can select the event options based on your use case. For this post, we use the default management events. Choosing the additional data and CloudTrail Insights events increases your overall cost.

Creating an Amazon ES domain
Placing your Amazon ES domain within a VPC provides inherent strong layer of security and is recommended for production clusters. Because we want to search the Elasticsearch index using Kibana, we configure an SSH tunnel to access Kibana from outside the VPC. You can also use an NGINX proxy or client VPN to access Kibana from outside a VPC, along with Amazon Cognito authentication. For more information, see How can I access Kibana from outside of a VPC using Amazon Cognito authentication? The main steps involved to access Kibana from outside VPC includes:

  1. Create an Amazon Cognito user pool and identity pool.
  2. Create an Amazon EC2 (Amazon EC2) instance in a public subnet in the same VPC that the Elasticsearch domain is in.
  3. Use a browser add-on, such as FoxyProxy, to configure a SOCKS proxy.
  4. Create an SSH tunnel from your local machine to the EC2 instance.
  5. Optionally, use the elasticsearch-in-vpc-only AWS Config Config Rule to determine if Elasticsearch is mistakenly accessible from outside the VPC.

Creating an Amazon Cognito user pool and identity pool
You can create the Amazon ES production cluster beforehand and modify the access policy, or you can configure the Amazon Cognito user pools, identity pools, and later create the Amazon ES domain. In this post, I create the Amazon Cognito identity pool (vpcidentitypool) and user pool (vpcuser) beforehand and then create a new Amazon ES domain. For instructions, see How can I use an SSH tunnel to access Kibana from outside of a VPC with Amazon Cognito authentication?

When creating your Amazon ES domain, complete the following steps:

  1. Enter a name for your Amazon ES domain.
  2. Select three Availability Zones.
  3. Choose your instance types.
  4. Choose your number of nodes (a multiple of your selected Availability Zones).
  5. Provide your storage requirements.
  6. Provide your dedicated primary node (instance type and number).
  7. Keep UltraWarm unselected.
  8. In Network configuration:
    1. Select VPC access.
    2. Select your VPC where you want to create the cluster and associated subnets.
    3. Select the security group to use for your Amazon ES domain
  9. Optionally, you may want to use fine-grained access control, powered by Open Distro for Elasticsearch. For instructions, see Fine-Grained Access Control in Amazon Elasticsearch Service.
  10. Enable Amazon Cognito authentication and choose the user pool and identity pool you created earlier. Your configuration looks similar to the following screenshot.
  11. For the Amazon ES domain access policy, use an access policy similar to the following and update the placeholders: 
    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:sts::<AWS Account ID>:assumed-role/Cognito_<Cognito Identity Pool Name>Auth_Role/CognitoIdentityCredentials"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2: <AWS Account ID>:domain/<ES Cluster Name>/*"
    }
  ]
}
  12. Select all the encryption options.
  13. Confirm all the configurations and create your domain.

Creating an EC2 instance
As a prerequisite to the SSH tunnel, you create an EC2 instance in the same VPC, where you created your Amazon ES domain.  Then, configure the security group rules for the EC2 instance. For instructions, see the Create an EC2 instance and configure security group rules section in How can I use an SSH tunnel to access Kibana from outside of a VPC with Amazon Cognito authentication?

Configuring a proxy
To access the Kibana dashboard, you first configure a proxy. For instructions, see the Configure the SOCKS proxy section in How can I use an SSH tunnel to access Kibana from outside of a VPC with Amazon Cognito authentication? You can also use an NGINX proxy setting or a Client VPN to establish this secure connection.

Creating an SSH tunnel
After you complete these steps, create an SSH tunnel to access the Kibana dashboard from your local machine (outside VPC).

  1. Run the following command from the local machine that you use to access the Kibana dashboard. Replace mykeypair.pem with the key pair for your EC2 instance and replace change public_dns_name with the public DNS of your tunnel_ec2 EC2 instance.                         ssh -i "mykeypair.pem" ec2-user@public_dns_name -ND 8157
  2. Enter the Kibana endpoint in your browser. The Amazon Cognito login page for Kibana appears.
  3. Use your Amazon Cognito user ID and password to log in to the dashboard.

Streaming CloudWatch Logs data to Amazon ES
Now that your Amazon ES cluster is ready to use, you can configure a CloudWatch Logs log group to stream the data it receives to your Amazon ES cluster in near-real time through a CloudWatch Logs subscription. For instructions, see Streaming CloudWatch Logs Data to Amazon Elasticsearch Service. For Log format, choose AWS CloudTrail.

When the sample testing is complete, start streaming the events to Amazon ES.

You get a notification like the following screenshot when you successfully start streaming.

When the subscription filter starts streaming the data to the Amazon ES domain, you get a confirmation like the following screenshot.

On your Amazon ES console, after a few minutes, you can see activities in the Key performance indicators section. The following screenshot shows an increase in Indexing rate.

 

 

You can also see an increase in the Searchable documents counts.

 

Visualizing the CloudTrail events using Kibana
In this step, you access the Kibana endpoint shown in your Amazon ES cluster overview and create a dashboard.

  1. On the Amazon ES console, choose your domain.
  2. On the Overview page, copy your Kibana endpoint.
  3. In your web browser, choose Use proxy Kibana Proxy for all URLs to enable FoxyProxy.
  4. When prompted, enter your Amazon Cognito user name and password to log in to Kibana.
  5. On the Add Data to Kibana page, choose Use Elasticsearch data, and connect to your Amazon ES index.
  6. When you’re connected, enter cwl-* as your index pattern.
  7. Enter eventTime as the time filter field.

You can now go to the Discover tab to add specific fields as filters and search for them. In the following screenshot, I selected fields specific to error events logged in CloudTrail to find the issues.

 

You can create other visualizations and dashboards based on your requirements from Kibana.

Checking the number of API events
In this first use case, you want to quickly check the number of API events logged by CloudTrail on your account in the last hour.

  1. Create a bar chart.
  2. For aggregation type in the Y-axis, choose Count.
  3. For Aggregation type in the X-axis, choose Terms.
  4. For Field, search for and choose eventName.keyword.
  5. For Order by, choose Metric: Number of Events.

Run a visualization to get a graph like in the following screenshot.

Checking for types of errorCode errors
In this use case, you create a time series graph to check for the different errorCode errors that CloudTrail detects in your AWS account.

  1. Select Visualize.
  2. Select Create new visualization.
  3. From the list of visualization options, choose TSVP.
  4. For Group by, choose Terms.
  5. For By, choose errorCode.keyword.

You can change the calendar at the top right of the visualization to choose a specific window, such as the last 60 minutes.

The following screenshot shows a graph with the occurrences of ResourceNotFound errors in the last hour.

Cleaning up
To avoid incurring future charges, delete the resources you created in this tutorial.  These include your Amazon ES domain, S3 bucket, CloudTrail events, CloudWatch Logs, and other associated resources.

Conclusion
In this post, we demonstrated how you can easily send your CloudTrail trail events to an Amazon ES cluster and then use Kibana to quickly search for any issues with access to your AWS resources.  To learn more about the CloudTrail and Elasticsearch integration works, visit the Logging and Monitoring in Amazon Elasticsearch Service documentation.

Author bio

Avijit Goswami is a Principal Solutions Architect at AWS, helping startup customers become tomorrow’s enterprises. When not at work, Avijit likes to cook, travel, watch sports, and listen to music.