Understanding the differences between configuration history and configuration snapshot files in AWS Config

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

When you run your applications on AWS, you often use AWS resources, which you must create and manage collectively. As the demand for your application keeps growing, so does your need to keep track of your AWS resources. AWS Config tracks changes made to these supported AWS resources and records their changes as configuration items (CIs), which are JSON files delivered to an Amazon Simple Storage Service (Amazon S3) bucket. These files are accessed through AWS Config APIs and optionally sent through Amazon Simple Notification Service (Amazon SNS).

AWS Config delivers three types of configuration files to the S3 bucket:

• Configuration history
• Configuration snapshot
• OversizedChangeNotification

In this blog post, we discuss the difference between configuration history and configuration snapshot files and the use cases for which they qualify.

Configuration history file

Figure 1: Configuration history file delivery

1. An AWS resource undergoes configuration changes.
2. The configuration recorder captures those changes.
3. Configuration items (CIs) are generated in JSON format.
4. The CIs are delivered as configuration history files to a designated S3 bucket every 6 hours.

Example:

123456789012_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20201007T123229Z_20201007T123229Z_1.json.gz

Configuration snapshot file

Figure 2: Configuration snapshot file delivery

1. A user manually performs the DeliverConfigSnapshot API call or the API is invoked according to a periodic setting for the snapshot in the configuration recorder.
2. AWS Config performs a point-in-time capture of the configurations of all the resources.
3. The CIs are generated in JSON format.
4. The CIs are delivered as configuration snapshot files to a designated S3 bucket.

"configHistoryDeliveryInfo": 
{
  "lastSuccessfulTime": 1450317838.412,
  "lastStatus": "SUCCESS",
  "lastAttemptTime": 1450317838.412
}

"configSnapshotDeliveryInfo": 
{
  "lastSuccessfulTime": 1452185597.094,
  "lastStatus": "SUCCESS",
  "lastAttemptTime": 1452185597.094
}

123456789012_Config_eu-west-1
_ConfigHistory_AWS::EC2::SecurityGroup_
20180816T155344Z_20180816T155350Z_1.json.gz

123456789012_Config_eu-west-1
_ConfigSnapshot_20180816T155344Z
_4f0d4257-61fe-46a3-9573-2afe04d3c8e9.json.gz

Note: The S3 bucket also contains an empty file named ConfigWritabilityCheckFile. AWS Config creates this file to verify that the service has permissions to successfully write to the S3 bucket.

Popular use cases

Because AWS Config delivers Configuration history and snapshot files to the S3 bucket, you can use the service’s integration with Amazon Athena to query either file type.

• Querying the configuration history file
• Querying the configuration snapshot file

You can also customize your query and use Amazon QuickSight to visualize the data according to your use case. For more information, see the Visualizing AWS Config data using Amazon Athena and Amazon QuickSight blog post.

FAQs

Q. Can I use the console to enable delivery of configuration snapshot files?

No. Currently, you must use the AWS CLI. Use the PutDeliveryChannel API to create a channel to deliver configuration information and then configure the delivery frequency.

Q. Can I have snapshot files delivered on demand or is the delivery only periodic?

Yes. Use the deliver-config-snapshot command to deliver the snapshot file on demand.

Q. Do configuration history and configuration snapshot record CI in the same way?

Yes. The configuration recorder records all changes in the Region and delivers the same CI in the same format to the S3 bucket in two different folders (ConfigHistory and ConfigSnapshot). 

Q. How can I view these files?

You can use the Amazon S3 console and the AWS CLI to view the files.

If you use the console:

There are two folders (ConfigHistory and ConfigSnapshot) in the following path:

config-bucket-XXXXXXXX/AWSLogs/<AccountID>/Config/<Region>/YYYY/MM/DD

Figure 3: Configuration history and snapshot files in the Amazon S3 console

ConfigHistory

Figure 4: Configuration history files in the Amazon S3 console

ConfigSnapshot

Figure 5: Configuration snapshot file in the Amazon S3 console

If you’re using the AWS CLI:

To retrieve configuration history files, use the get-resource-config-history command and specify the resource type and the resource ID.

Note: You cannot use the AWS CLI to retrieve configuration snapshot files.

Command

$ aws configservice get-resource-config-history --resource-type AWS::EC2::SecurityGroup --resource-id sg-123456ab

Q. If the files in the S3 bucket are deleted, will it affect the timeline view in the AWS Config console?

No. The configuration history and snapshot files delivered to an S3 bucket are for your auditing and analysis only. The configuration information displayed in the AWS Config console is from its data store.

Q. Does the retention period in the configuration recorder setting affect S3 files?

No. This retention period does not apply to the files in the S3 bucket. The files in the AWS Config data store depend on the retention period setting in the Config. By default, this value is 7 years. 

Q. If I’m using an aggregator, are the CIs of member accounts delivered to the master account?

Only current state CIs that represent the configuration and compliance state of AWS resources can be aggregated into an AWS Config aggregator. Historical state CIs of member accounts are delivered in the history files for each account. You can designate a central S3 bucket to deliver the history files of all member accounts into a central bucket.

Q. Can I use advanced querying to query the history and snapshot files?

No. Advanced querying can only be used to query the current configuration and compliance state of your resources natively in the console or through APIs. You can use Amazon Athena to query the history and snapshot files in your S3 bucket.

Conclusion

In this post, we showed you the differences between configuration history and configuration snapshot files. With this information, you can now start monitoring your account for recorded changes for billing and other reasons required by your use case. You can take advantage of the AWS Config integration with Amazon Athena, Amazon ES, and Kibana to extract the details in these files.

About the authors

Attalla is a Sr. Cloud Support Engineer. He specializes in AWS Config, Logging and Monitoring AWS services. Attalla is passionate about building secure and effective solutions for AWS customers. He enjoys deep dive into compliance related challenges, writing articles and building tutorials. Outside of work, Attalla’s favorite activities include snorkeling and reading. He loves spending time with his daughters.

Sagara is a Cloud Support Engineer. She has a passion for providing customers with technical guidance of AWS Security services and coming up with innovative solutions to customer issues. Outside of work, she enjoys spending time with her family and going on hikes.

Sushma is Cloud Support Engineer at AWS. She is a multi-domain enthusiast with a vast experience in the field of cyber-security and is passionate about coding solutions to automate processes. She is motivated to innovate ideas which simplifies solutions and develop articles with gotchas for the customers. Outside of work she is a painter, dancer, enjoys a good Netflix binge but can also be found on long bike rides on hilly country roads.