Pagination using Async Iterators in modular AWS SDK for JavaScript

As of December 15th, 2020, the AWS SDK for JavaScript, version 3 (v3) is generally available.

On October 19th, 2020, we published the Release Candidate (RC) of the AWS SDK for JavaScript, version 3 (v3). In v3, we are using async generator functions in paginators. In this blog post, we will cover how you can paginate using async iterators in a for await..of loop.

Motivation

Many AWS operations return paginated results when the response object is too large to return in a single response. In AWS SDK for JavaScript v2, the response contains a token you can use to retrieve the next page of results. You then need to write additional functions to process pages of results.

Pagination in JavaScript SDK v2

The ListTables operation on DynamoDB is paginated, with each page returning a maximum of 100 table names. If LastEvaluatedTableName also appears in the output, you can use this value as the ExclusiveStartTableName parameter in a subsequent ListTables request to obtain the next page of results:

const AWS = require("aws-sdk");

...
const client = new AWS.DynamoDB({ region });

// First page of results - response.TableNames contain maximum 100 entries.
const response = await client.listTables({}).promise();

if (response.LastEvaluatedTableName) {
  // pagedResponse.TablesNames contain second page of table names.
  const pagedResponse = await client.listTables({
    ExclusiveStartTableName: response.LastEvaluatedTableName
  }).promise();
}
...

The following example code calls the ListTables operation till all table names are returned, and pushes results returned in each paged response in the tableNames array.

const AWS = require("aws-sdk");

...
const client = new AWS.DynamoDB({ region });

const tableNames = [];
let hasNext = true;
let ExclusiveStartTableName = undefined;

while (hasNext) {
  const response = await client
    .listTables({ ExclusiveStartTableName })
    .promise();

  // Add tableNames returned in this paged response. 
  tableNames.push(...response.TableNames);

  ExclusiveStartTableName = response.LastEvaluatedTableName;
  hasNext = !!ExclusiveStartTableName;
}
...

Async Iterators

In AWS SDK for JavaScript v3 we’ve improved pagination using async generator functions, which are similar to generator functions, with the following differences:

• When called, async generator functions return an object, an async generator whose methods (next, throw, and return) return promises for { value, done }, instead of directly returning { value, done }. This automatically makes the returned async generator objects async iterators.
• await expressions and for-await-of statements are allowed.
• The behavior of yield* is modified to support delegation to async iterables.

The Async Iterators were added in the ES2018 iteration of JavaScript. They are supported by Node.js 10.x+ and by all modern browsers, including Chrome 63+, Firefox 57+, Safari 11.1+, and Edge 79+. If you’re using TypeScript v2.3+, you can compile Async Iterators to older versions of JavaScript.

An async iterator is much like an iterator, except that its next() method returns a promise for a { value, done } pair. As an implicit aspect of the Async Iteration protocol, the next promise is not requested until the previous one resolves. This is a simple, yet a very powerful pattern.

Pagination using Async Iterators in JavaScript SDK v3

In v3, the clients expose paginateOperationName APIs that are written using async generators, allowing you to use async iterators in a for await..of loop. You can perform the paginateListTables operation from @aws-sdk/client-dynamodb as follows:

const {
  DynamoDBClient,
  paginateListTables,
} = require("@aws-sdk/client-dynamodb");

...
const paginatorConfig = {
  client: new DynamoDBClient({}),
  pageSize: 25
};
const commandParams = {};
const paginator = paginateListTables(paginatorConfig, commandParams);

const tableNames = [];
for await (const page of paginator) {
  // page contains a single paginated output.
  tableNames.push(...page.TableNames);
}
...

If you don’t want to pass optional parameters, the above code can be simplified as follows:

...
const client = new DynamoDBClient({}); 

const tableNames = [];
for await (const page of paginateListTables({ client }, {})) {
    // page contains a single paginated output.
    tableNames.push(...page.TableNames);
}
...

Feedback

We value your feedback, so please tell us what you like and don’t like by opening an issue on GitHub.