How can I speed up the creation of a global secondary index for a DynamoDB table?

Last updated: 2020-06-24

I'm trying to create a global secondary index (GSI) for an Amazon DynamoDB table, but it's taking a long time. How can I speed up the creation process?

Short description

When you add a new global secondary index to an existing table, the IndexStatus is set to CREATING and Backfilling is true. Backfilling is the process of reading items from the table and determining whether they can be added to the index. The time required for building a global secondary index depends on multiple factors, including:

  • The size of the base table
  • The number of items in the table that qualify for inclusion in the index
  • The number of attributes projected into the index
  • The provisioned write capacity of the index
  • Write activity on the base table during index creation
  • Data distribution across index partitions

To speed up the creation process, increase the number of write capacity units (WCUs) on the index.

Resolution

Use the OnlineIndexPercentageProgress Amazon CloudWatch metric to monitor the index creation progress:

  1. Open the DynamoDB console.
  2. In the navigation pane, choose Tables, and then select your table from the list.
  3. Choose the Metrics tab.
  4. Choose View all CloudWatch metrics.
  5. In the search box, enter OnlineIndexPercentageProgress.
    Note: If the search returns no results, wait about a minute for metrics to populate. Then, try again.
  6. Choose the name of the index to see the progress.

Determine the number of additional WCUs you need by dividing the table size in kilobytes by your desired backfill time. For example, suppose that you have a 1 GiB (1,074,000 KB) table. You want the backfilling process to complete in 10 minutes (600 seconds). Here's how many WCUs you need:

1,074,000 / 600 = 1,790 WCUs

Note: This is an estimate. Creation time varies depending on how you distribute keys across partitions, how large the items are, how many attributes are projected from the table into the index, and so on.

To provision additional write capacity:

  1. Open the DynamoDB console.
  2. In the navigation pane, choose Tables, and then select your table from the list.
  3. Choose the Capacity tab.
  4. Increase the write capacity of the index, and then choose Save.
  5. After about a minute, check the OnlineIndexPercentageProgress metric to see if the creation speed is improved.