Output Pagination with AWS Tools for PowerShell

Amongst the changes to the cmdlets in version 1.1 of the AWS Tools for Windows PowerShell are added support for both automatic and manual pagination of the output from services to the pipeline. Most of the time, you’ll probably want to use automatic paging to get all the data from a cmdlet, but on occasion you may prefer to control this yourself.

Automatic Pagination

Making use of automatic output pagination is simple—run the cmdlet with no paging parameters:

"mybucket" | Get-S3Object

The output from this to the pipeline will be zero or more S3Object instances. By default Amazon S3 returns a maximum of 1000 items per call, so the cmdlet keeps calling on your behalf and emitting each batch of results to the pipe until S3 signals that there is no more content.

Here’s another example of using automatic pagination in conjunction with the improved pipeline support, in this case to get the size (in GB in this example) of a bucket:

((Get-S3Object -BucketName mybucket).Size | Measure-Object -Sum).Sum / 1024 / 1024 / 1024

Automatic pagination doesn’t just work with Amazon S3:

Get-CFNStack | Get-CFNStackResources

In this example, Get-CFNStack enumerates all of your AWS CloudFormation stacks (in the current region set for the shell) and emits each to the downstream Get-CFNStackResources cmdlet to get additional resource information on the stack.

If you’re curious, or need to diagnose a problem, you can see the repeated calls that the cmdlets make to obtain a full set of results by inspecting the entries in the $AWSHistory shell variable (this is also new with version 1.1). You can also it to see how the cmdlets track the ‘next page’ marker for each service call on your behalf.

Controlling Pagination Yourself

Most of the time, automatic pagination can be used without further thought. However, there may be occasions when you are dealing with huge data sets in a memory-constrained environment. In these cases, you may elect to control the pagination yourself. The names of the parameters to use to control paging do vary by service but are usually called NextMarker, Marker, NextToken, and so on.

The cmdlets for S3 use parameters called MaxKeys and Marker to control the iteration (and NextMarker and IsTruncated fields in the service response to indicate where the next page of results can be fetched, if any). The first example in this post can be rewritten like this to fetch all the S3Object instances in batches of 50 at a time:

$nextMarker = $null
$keysPerPage = 50
do
{
	$objects = Get-S3Object "mybucketname" -KeyPrefix / -MaxKeys $keysPerPage -Marker $nextMarker
	$nextMarker = $AWSHistory.LastServiceResponse.NextMarker
        # do something with the batch of results in $objects here
} while ($nextMarker)

When you handle paging yourself, be sure to capture the ‘next marker’ token from the response straight away and before you make another call to AWS; otherwise, the response that $AWSHistory.LastServiceResponse points to may not be what you think it is.

Similar patterns can be followed for other services that support paginated result sets.

Parameter Aliases

As mentioned above, Amazon S3 cmdlets use MaxKeys and Marker parameters to expose pagination, mirroring the underlying service API. Other services use different names (NextToken for example, or MaxRecords). This can be difficult to remember, especially if you’re unfamiliar with the service. To help, we add aliases to the parameters so that you can expect a consistent naming scheme, regardless of the service. These aliases are NextToken and MaxItems. MaxItems, by the way, is the maximum number of items to emit to the pipeline, which may be more or less than the underlying service’s maximum per call.

Using aliases, these two commands are the same:

Get-S3Object mybucket -KeyPrefix / -Marker "token2" -MaxKeys 200
Get-S3Object mybucket -KeyPrefix / -NextToken "token2" -MaxItems 200

Summary

In this post, we covered how to make use of the automatic results pagination now available in the AWS PowerShell cmdlets and how to take control and perform the pagination yourself using parameters and aliases for a uniform experience across services. It also showed how to access additional information from the service responses being logged by the new $AWSHistory shell variable.