Which Log Group is causing a sudden increase in my CloudWatch Logs bill?

Last updated: 2019-05-20

My Amazon CloudWatch Logs bill is unusually high. How can I determine which log group is causing the increase in my CloudWatch Logs bill?

Short Description

Sudden increases in CloudWatch Logs bills are often caused by an increase in ingested or storage data in a particular log group. Check data usage using CloudWatch Logs Metrics and review your Amazon Web Services (AWS) bill to identify the log group responsible for bill increases.

Resolution

Check how much data you're ingesting

The IncomingBytes metric shows you how much data is being ingested in your CloudWatch log groups in near-real time. This metric can help you to determine:

  • Which log group is the highest contributor towards your bill
  • Whether there's been a spike in the incoming data to your log groups or a gradual increase due to new applications
  • How much data was pushed in a particular period

To query a small set of log groups:

  1. Open the Amazon CloudWatch console.
  2. In the navigation pane, choose Metrics.
  3. For each of your log groups, select the IncomingBytes metric, and then choose the Graphed metrics tab.
  4. For Statistic, choose Sum.
  5. For Period, choose 30 Days.
  6. Choose the Graph options tab and choose Number.
  7. At the top right of the graph, choose custom, and then choose Absolute. Select a start and end date that corresponds with the last 30 days.

To query hundreds of log groups:

Note: Before running the API calls below, be sure to review costs associated with making API calls. Also note that the ListMetrics call should be distributed to avoid throttling. The default limit for ListMetrics is 25 transactions per second. However, you can request a limit increase if necessary.

  1. Make a ListMetrics call. Use this call to find all log group names that have ingested data in the past 14 days. Use the following parameters:
    Namespace: AWS/Logs
    MetricName: IncomingBytes
  2. Make a GetMetricData call. Use this call to find the sum of all incoming bytes in a month for every log group name you get from the ListMetrics call. Use the following parameters:
    Namespace: AWS/Logs
    MetricName: IncomingBytes
    Dimensions: As received from the ListMetrics call
    StartTime: [Date and time 14 days prior to the current date]
    EndTime: [Current date and time]
    Period: [EndTime - StartTime, in seconds]
    Statistics: Sum
  3. Sort the resulting data points in descending order to display the log group names with the highest Ingested data amounts.

To be sure that ingested data charges don't exceed a specified limit in the future, you can create a CloudWatch alarm.

Review your storage data usage

Check your most recent AWS bill to see how much storage data you used in the previous billing cycle.


Did this article help you?

Anything we could improve?


Need more help?