Why is my EC2 AMI or EBS snapshot creation slow?
Last updated: 2021-05-25
I'm trying to create a backup of my Amazon Elastic Compute Cloud (Amazon EC2) instance or Amazon Elastic Block Store (Amazon EBS) volume by creating an Amazon Machine Image (AMI) or snapshot. However, this process is slow or appears to be stuck in the Pending state.
Amazon EBS-backed AMIs include one or more Amazon EBS snapshots. The creation of EBS-backed AMIs or EBS snapshots might be slow because of the large amount of data that must be copied to Amazon Simple Storage Service (Amazon S3). Many factors, such as Write activity on the EBS volume, can impact the creation time. Therefore, creation times for the snapshots might vary widely.
The most common cause for slow creation of AMIs or snapshots is the amount of dirty data that must be copied to Amazon S3. This dirty data is measured by the number of blocks. The following factors might result in a large number of dirty blocks:
- Size of the EBS volume
- Time since last snapshot
- Write activity on the volume
Snapshots are designed to be incremental. This means that Amazon EBS copies only the blocks that changed since the last snapshot was created. An EBS volume might not have an existing snapshot because no snapshots were created before, or previous snapshots were deleted. In such cases, the blocks can't be compared against any snapshot. Therefore, all the blocks are considered dirty.
Also, if a snapshot was created a long time ago, or the EBS volume is very active, you might have a large number of blocks that must be copied as part of the new snapshot.
To avoid this issue, it's a best practice to create snapshots frequently. With this practice, the number of blocks to be copied for each snapshot is smaller. You can use Amazon Data Lifecycle Manager to automate the creation, retention, and deletion of snapshots for your EBS volumes. You can set the frequency of snapshots based on your Recovery Point Objective (RPO). EBS snapshots aren't charged based on the number of snapshots, but rather for the incremental data saved in S3. For more information, see Amazon EBS pricing.
Multiple volumes queued for snapshot creation
Snapshot creation is a shared bandwidth operation. This means that Amazon EBS uses a shared bandwidth to send data to S3. Therefore, you might encounter delays if there are multiple volumes queued for snapshot creation. This typically happens if several snapshots are created at the top of the hour. For example, if you have automated processes to create snapshots exactly at midnight, then the snapshot creation might be delayed.
To avoid this issue, it's a best practice to create snapshots at various times past the hour. This practice might help reduce the time taken to create a snapshot. You can use Amazon Data Lifecycle Manager to automatically create snapshots within an hour after the scheduled start time, instead of creating snapshots immediately.
When you create multiple snapshots for the same volume within a short period of time, then the first snapshot is created, and the other snapshots are moved to the Pending state. The creation of the snapshots in the Pending state doesn't progress until the first snapshot creation is completed. Deleting a snapshot that's in the Pending state doesn't stop the creation process. If you try to delete the snapshot that's in the Pending state, the snapshot is created first before it's deleted.
To resolve this issue, avoid creating snapshots when a snapshot creation for a volume is in progress, unless it's required to do so.