New – EBS Direct APIs – Programmatic Access to EBS Snapshot Content

EBS Snapshots are really cool! You can create them interactively from the AWS Management Console:

You can create them from the Command Line (create-snapshot) or by making a call to the CreateSnapshot function, and you can use the Data Lifecycle Manager (DLM) to set up automated snapshot management.

All About Snapshots
The snapshots are stored in Amazon Simple Storage Service (Amazon S3), and can be used to quickly create fresh EBS volumes as needed. The first snapshot of a volume contains a copy of every written block on the volume. Subsequent snapshots contain only the blocks that have changed since the previous snapshot. The incremental nature of the snapshots makes them very cost-effective, since (statistically speaking) many of the blocks on an EBS volume do not change all that often.

Let’s look at a quick example. Suppose that I create and format an EBS volume with 8 blocks (this is smaller than the allowable minimum size, but bear with me), copy some files to it, and then create my first snapshot (Snap1). The snapshot contains all of the blocks, and looks like this:

Then I add a few more files, delete one, and create my second snapshot (Snap2). The snapshot contains only the blocks that were modified after I created the first one, and looks like this:

I make a few more changes, and create a third snapshot (Snap3):

Keep in mind that the relationship between directories, files, and the underlying blocks is controlled by the file system, and is generally quite complex in real-world situations.

Ok, so now I have three snapshots, and want to use them to create a new volume. Each time I create a snapshot of an EBS volume, an internal reference to the previous snapshot is created. This allows CreateVolume to find the most recent copy of each block, like this:

EBS manages all of the details for me behind the scenes. For example, if I delete Snap2, the copy of Block 0 in the snapshot also deleted since the copy in Snap3 is newer, but the copy of Block 4 in Snap2 becomes part of Snap3:

By the way, the chain of backward references (Snap3 to Snap1, or Snap3 to Snap2 to Snap1) is referred to as the lineage of the set of snapshots.

Now that I have explained all this, I should also tell you that you generally don’t need to know this, and can focus on creating, using, and deleting snapshots!

However…

Access to Snapshot Content
Today we are introducing EBS direct APIs that provide you with access to the snapshot content, as described above. These APIs are designed for developers of backup/recovery, disaster recovery, and data management products & services, and will allow them to make their offerings faster and more cost-effective.

The new APIs use a block index (0, 1, 2, and so forth), to identify a particular 512K block within a snapshot. The index is returned in the form of an encrypted token, which is meaningful only to the GetSnapshotBlock API. I have represented these tokens as T0, T1, and so forth below. The APIs currently work on blocks of 512K bytes, with plans to support more block sizes in the future.

Here are the APIs :

ListSnapshotBlocks – Identifies all of the blocks in a given snapshot as encrypted tokens. For Snap1, it would return [T0, T1, T2, T3, T4, T5, T6, T7]. For Snap2, it would return [T0, T1, T2, T3, T4, T5, T6, T7] with updated values for T0 and T4.

GetSnapshotBlock – Returns the content of a block. If the block is part of an encrypted snapshot, it will be returned in decrypted form.

ListChangedBlocks – Returns the list of blocks that have changed between two snapshots in a lineage, again as encrypted tokens. For changes between Snap2 and Snap1, it would return [T0, T4] and for changes between Snap3 and Snap2, it would return [T0, T5]. For changes between Snap3 and Snap1, it would return [T0, T4, T5].

Like I said, these APIs were built to address one specialized yet very important use case. Having said that, I am now highly confident that new and unexpected ones will pop up within 48 hours (feel free to share them with me)!

Available Now
The EBS direct APIs are available now and you can start using them today in the US East (N. Virginia), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Singapore), and Asia Pacific (Tokyo) Regions; they will become available in the remaining regions in the next few weeks. There is a charge for calls to the List and Get APIs, and the usual KMS charges will apply when you call GetSnapshotBlock to access a block that is part of an encrypted snapshot.

— Jeff;