How can I retrieve the number of configuration items per month recorded by AWS Config?

Last updated: 2019-08-05

I want the number of configuration items recorded by AWS Config.

Resolution

To help you identify previous and current billing trends, use Amazon Athena to identify the number of configuration items (CI) per month for your account.

Verify your Amazon S3 bucket contains configuration files

Note: Be sure the AWS Config service is able to deliver configuration history files successfully to your Amazon Simple Storage Service (Amazon S3) bucket. Configuration history files are available only for the last 90 days. For more information, see Viewing Configuration Details.

  1. Open the AWS Config console, and then choose Settings from the navigation pane.
  2. In the Amazon S3 bucket section, note the Bucket name.
  3. Open the Amazon S3 console, and then for Bucket name, choose your Amazon S3 bucket. Verify that the Amazon S3 bucket you choose contains configuration files.
    Note: If there are no configuration files, your role can be missing permissions. For more information, see Setting Bucket and Object Access Permissions.

Create a table in Athena

  1. Sign in to the Athena console, and then follow the instructions at To create a table using the wizard.
  2. Use the following table syntax.
    Note: For LOCATION, use the location and Region for the AWS Config items stored in your Amazon S3 bucket.
CREATE EXTERNAL TABLE awsconfig (
         fileversion string,
         configSnapshotId string,
         configurationitems ARRAY < STRUCT < configurationItemVersion : STRING,
         configurationItemCaptureTime : STRING,
         configurationStateId : BIGINT,
         awsAccountId : STRING,
         configurationItemStatus : STRING,
         resourceType : STRING,
         resourceId : STRING,
         resourceName : STRING,
         ARN : STRING,
         awsRegion : STRING,
         availabilityZone : STRING,
         configurationStateMd5Hash : STRING,
         resourceCreationTime : STRING > > 
) 
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' LOCATION 's3://<BUCKET-NAME>/AWSLogs/<ACCOUNT-ID>/Config/<REGION>/';

Example Athena queries

The following example query retrieves the number of configuration items per day in February 2018:

SELECT result.configurationitemcapturetime,
         count(result.configurationitemcapturetime) AS NumberOfChanges
FROM 
    (SELECT regexp_replace(configurationItem.configurationItemCaptureTime,
         '(.+)(T.+)', '$1') AS configurationitemcapturetime
    FROM default.awsconfig
    CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
    WHERE "$path" LIKE '%ConfigHistory%'
            AND configurationItem.configurationItemCaptureTime >= '2018-02-01T%'
            AND configurationItem.configurationItemCaptureTime <= '2018-02-28T%') result
GROUP BY  result.configurationitemcapturetime
ORDER BY  result.configurationitemcapturetime

The results are similar to this:  

configurationitemcapturetime    NumberOfChanges
2018-02-02    7
2018-02-03    3
2018-02-07   11
...

The following example query retrieves the number of changes per resource in February 2018, sorted by most frequently changed:

SELECT configurationItem.resourceType,
         configurationItem.resourceId,
         COUNT(configurationItem.resourceId) AS NumberOfChanges
FROM default.awsconfig
CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
WHERE "$path" LIKE '%ConfigHistory%'
        AND configurationItem.configurationItemCaptureTime >= '2018-02-01T%'
        AND configurationItem.configurationItemCaptureTime <= '2018-02-28T%'
GROUP BY  configurationItem.resourceType, configurationItem.resourceId
ORDER BY  NumberOfChanges DESC

The results are similar to this:  

resourcetype              resourceid        NumberOfChanges
AWS::EC2::VPC             vpc-9ed00bfa        7
AWS::EC2::Subnet          subnet-4472e248     5
AWS::EC2::SecurityGroup   sg-450c6531         4

Note: When comparing the total number of CIs between Athena query results and AWS billing data for the same month and Region, a discrepancy can occur. The data queried by Athena can cross day boundaries and also include CIs billed in adjacent months. AWS Config CIs are metered based on when the time configurationItemCaptureTime was initiated.

As a best practice, increment the specified end day from end of month by one.

For example, change this query:

AND configurationItem.configurationItemCaptureTime <= '2018-02-28T%') result

To this query:

AND configurationItem.configurationItemCaptureTime <= '2018-03-01T%') result

Note: The last day of the month is incremented by one day.


Did this article help you?

Anything we could improve?


Need more help?