Introducing Paginators in AWS SDK for .NET v3.5

We are happy to introduce Paginators in the latest release of AWS SDK for .NET v3.5! Many AWS operations return paginated results when the response object is too large to return in a single response. Paginators can help you navigate through paginated responses from services.

Basic Usage

Typically, a service will return a truncated response when the response length is greater than a certain limit, containing a next token to use in a subsequent request. In order to consume all of the data, you could create a loop that makes multiple requests, replacing the next token in each iteration. However, with paginators this extra loop isn’t necessary because the SDK handles all of that logic for you; in other words, you can retrieve all of the data with less code.

For example, here is how you would list the keys of the items in your S3 bucket to the console without using paginators.

var request = new ListObjectsV2Request
{
    BucketName = "paginators-example"
};
do
{
    var response = client.ListObjectsV2(request);
    foreach(var s3Object in response.S3Objects)
    {
        Console.WriteLine(s3Object.Key);
    }
    request.ContinuationToken = response.NextContinuationToken;
}
while (request.ContinuationToken != null);

Now let’s see how to list these items using a paginator for ListObjectV2. The code looks so much cleaner!

var listObjectsV2Paginator = client.Paginators.ListObjectsV2(new ListObjectsV2Request
{
    BucketName = "paginators-example"
});
foreach (var s3Object in listObjectsV2Paginator.S3Objects)
{
    Console.WriteLine(s3Object.Key);
}

Usage Details

Some services have APIs that require pagination to access all of the data returned from the service. In this case the service client for the service will have a Paginators property. This can be found by using your IDE’s IntelliSense or by looking at the .NET SDK API reference for that service (for example, the Properties of AmazonS3Client). The operations supported by the service’s paginator can be found in the same way.

Here is an example of initializing a paginator for the S3 ListObjectsV2 operation:

var client = new AmazonS3Client();
var listObjectsV2Paginator = client.Paginators.ListObjectsV2(new ListObjectsV2Request
{
    BucketName = "paginators-example"
});

IAsyncEnumerable support

In the first example, we saw how pagination works synchronously. For .NET Core, the SDK only supports asynchronous operations. We wanted to make sure that the logic flows well between synchronous and asynchronous versions of the paginator by using Asynchronous Enumerables introduced in C# 8.

Asynchronous enumerables use the the IAsyncEnumerable interface that was introduced to .NET with .NET Core 3.1. Recently we released version 3.5 of the AWS SDK for .NET, which added an additional target platform for .NET Core 3.1 to give us access to IAsyncEnumerable. To provide access to IAsyncEnumerable for .NET Core developers who are not yet on .NET Core 3.1, the .NET Standard 2.0 variant of the AWS SDK for .NET starting with version 3.5 has a dependency on the Microsoft.Bcl.AsyncInterfaces NuGet package. This package, provided by Microsoft, adds the required types such as IAsyncEnumerables to platforms that predate .NET Core 3.1 (for example, .NET Standard 2.0).

If you are using .NET Core, you can now take advantage of this new Asynchronous Enumerables feature since it works so well for paginators. The only difference between the synchronous and asynchronous implementation is just the type of the paginator. It is an IEnumerable for customers on the .NET Framework 4.5 target and an IAsyncEnumerable for customers on the .NET Standard 2.0 and .NET Core 3.1 targets. Simply add an await keyword before the foreach loop to apply the same logic as the synchronous case to the asynchronous case.

Here is the initial paginator example in an asynchronous context:

var listObjectsV2Paginator = client.Paginators.ListObjectsV2(new ListObjectsV2Request
{
    BucketName = "paginators-example"
});
await foreach (var s3Object in listObjectsV2Paginator.S3Objects)
{
    Console.WriteLine(s3Object.Key);
}

Responses and Key Results

Paginators can access the full responses or just the key results for the AWS resource. We will explain the key results in depth later in this section.

Responses is a property on the paginator that you can use to iterate through full responses from AWS for the operation, and access any properties contained in the response.

var listObjectsV2Paginator = client.Paginators.ListObjectsV2(new ListObjectsV2Request
{
    BucketName = "paginators-example"
});
foreach (var response in listObjectsV2Paginator.Responses)
{
    Console.WriteLine(response.HttpStatusCode);
    Console.WriteLine(response.ResponseMetadata.RequestId);
    Console.WriteLine(response.KeyCount);
    foreach (var s3Object in response.S3Objects)
    {
        Console.WriteLine(s3Object.Key);
    }
}

Key results are unique to each operation and represent the properties in the response whose data is most likely to be truncated due to length. An AWS resource typically has some key properties such as items in a DynamoDB table or logs in a CloudWatch log group. We designed the paginator to give easier access to these properties whenever possible. We call these properties the key result since they are found in the service’s response. Some operations may have more than one key result. For the ListObjectsV2 paginator, the key results are S3Objects and CommonPrefixes.

Here is an example of iterating over the key results:

var paginatorForS3Objects = client.Paginators.ListObjectsV2(new ListObjectsV2Request { BucketName = "paginators-example" });
var paginatorForCommonPrefixes = client.Paginators.ListObjectsV2(new ListObjectsV2Request { BucketName = "paginators-example" });
foreach (var s3Object in paginatorForS3Objects.S3Objects)
{
    Console.WriteLine(s3Object.Key);
}
foreach (var commonPrefix in paginatorForCommonPrefixes.CommonPrefixes)
{
    Console.WriteLine(commonPrefix);
}

Note: The paginator can only be iterated once. In this example we created two separate paginator instances to iterate through the two collections.

Try Paginators Today!

We hope you are interested in trying paginators after reading this blog post. To see a full example of paginators in action, look at the Paginators topic in our Developer Guide. Please let us know what you think, and open an issue if you run into any trouble!