Why is my Amazon DynamoDB table being throttled?
Last updated: 2021-10-12
Read or write operations on my Amazon DynamoDB table are being throttled.
Throttling is one of the most common performance issues that you might encounter with your DynamoDB tables. Throttling could be caused by either DynamoDB or the applications that read or write into your DynamoDB table.
Some of the common throttling issues that you might face are the following:
- Your DynamoDB table has adequate provisioned capacity, but most of the requests are being throttled.
- You enabled AWS Application Auto Scaling for DynamoDB, but your DynamoDB table is being throttled.
- Your DynamoDB table is in on-demand capacity mode, but the table is being throttled.
- You have a hot partition in your table.
For information on DynamoDB metrics that must be monitored during throttling events, see DynamoDB metrics and dimensions. These metrics might help you to locate the operations creating throttled requests and identify the cause for throttling. Use one or more of the following troubleshooting options based on your use case.
Your DynamoDB table has adequate provisioned capacity, but most of the requests are being throttled
DynamoDB reports minute-level metrics to Amazon CloudWatch. The metrics are calculated as the sum for a minute, and then are averaged. However, the DynamoDB rate limits are applied per second.
For example, if you provisioned 60 write capacity units for your DynamoDB table, then you can perform 3600 writes in one minute. However, driving all 3600 requests in one second with no requests for the rest of the minute might result in throttling. The total number of read capacity units or write capacity units per minute might be lower than the provisioned throughput for the table. However, if all the workload falls within a couple of seconds, then the requests might be throttled.
To resolve this issue, add jitter and exponential backoff to your API calls. For more information, see Exponential backoff and jitter.
You enabled AWS Application Auto Scaling, but your table is still being throttled
AWS Application Auto Scaling is not a suitable solution to address sudden spikes in traffic with DynamoDB tables. Application Auto Scaling initiates a scale up only when two consecutive data points for consumed capacity units exceed the configured target utilization value within a one-minute span. Application Auto Scaling automatically scales the provisioned capacity only when the consumed capacity is higher than target utilization for two consistent minutes. Also, a scale down event is initiated when 15 consecutive data points for consumed capacity in CloudWatch are lower than the target utilization. After Application Auto Scaling is initiated, an UpdateTable API call is invoked that might take a couple of minutes to update the provisioned capacity for your DynamoDB table or index. Application Auto Scaling requires consecutive data points with higher target utilization values to scale up the provisioned capacity of the DynamoDB table. During this period, any requests that exceed the provisioned capacity of the table are throttled. Therefore, it's not a best practice to use Application Auto Scaling to handle spiked workloads in DynamoDB. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.
Your table uses the on-demand capacity mode, but the table is still being throttled
When the table uses the on-demand capacity mode, the table doesn't throttle as long as the following conditions are true:
- The access pattern is distributed evenly across partitions to avoid issues related to hot partition.
- The table doesn't exceed double its previous peak traffic.
For on-demand tables, DynamoDB automatically allocates more capacity as your traffic volume increases to make sure that your workload doesn't experience throttling. However, throttling can occur if the traffic volume is more than double the previous peak within a span of 30 minutes. For more information, see Peak traffic and scaling properties.
You have a hot partition in your table
In DynamoDB, a partition key that doesn't have a high cardinality can result in many requests targeting only a few partitions and resulting in a hot partition. A hot partition can cause throttling if the partition limits of 3000 RCU or 1000 WCU (or a combination of both) per second are exceeded.
To find the most accessed and throttled items in your table, use the Amazon CloudWatch Contributor Insights. Amazon CloudWatch Contributor Insights is a diagnostic tool that provides a summarized view of your DynamoDB tables traffic trends and helps you identify the most frequently accessed partition keys. With this tool, you can continuously monitor the graphs for your table’s item access patterns. A hot partition can degrade the overall performance of your table. To avoid this poor performance, distribute the read and write operations as evenly as possible across your table. For more information, see Designing partition keys to distribute your workload evenly. For information on choosing the right partition key, see Choosing the right DynamoDB partition key.
Note: Using the CloudWatch Contributor Insights tool for DynamoDB incurs additional charges. For more information, see CloudWatch Contributor Insights for DynamoDB billing.