How do I retrieve log data from CloudWatch Logs?

Last updated: 2019-05-08

I want to find log data for my Amazon Web Services (AWS) account. What are the best practices for getting log data from Amazon CloudWatch Logs?

Short Description

There are four recommended methods for retrieving log data from CloudWatch Logs:

  • Use subscription filters to stream log data to another receiving source in real time
  • Run a query with CloudWatch Logs Insights
  • Export log data to Amazon Simple Storage Service (Amazon S3) for batch use cases
  • Call GetLogEvents or FilterLogEvents in the CloudWatch API

Resolution

Use subscription filters

To immediately retrieve log data from CloudWatch Logs in real time, you can use subscription filters. For more information, see Real-time Processing of Log Data with Subscriptions and Using CloudWatch Logs Subscription Filters.

Filtering for log events is performed internally, which prevents CloudWatch API throttling. Amazon Kinesis Data Streams automatically retries throttled service API calls.

Run a query in CloudWatch Logs Insights

To quickly search and analyze your log data, run a query in CloudWatch Logs Insights.

Export log data to Amazon S3 (batch use cases)

To move log data from CloudWatch Logs to Amazon S3 in batch use cases, see Exporting Log Data to Amazon S3. Log data can take up to twelve hours to become available for export from CloudWatch Logs. For real-time analysis and processing, use subscription filters.

Important: An AWS account can have only one export task for log data in the state PENDING, PENDING_CANCEL, or RUNNING.

Call GetLogEvents or FilterLogEvents

To find log data manually, use GetLogEvents or FilterLogEvents in the CloudWatch API.

Important: This method isn’t scalable due to the transactions per second (TPS) limits for GetLogEvents and FilterLogEvents. For current limits, see CloudWatch Logs Limits. If you experience throttling when performing these actions, use subscription filters instead.


Did this article help you?

Anything we could improve?


Need more help?