How can I troubleshoot high latency on a DynamoDB table?
Last updated: 2020-06-16
I'm seeing an increase in the response time for Amazon DynamoDB requests. Why is this happening and how can I fix it?
When analyzing the Amazon CloudWatch metric SuccessfulRequestLatency, it's a best practice to check the average latency. Occasional spikes in latency are not a cause for concern. However, if average latency is high, there might be an underlying issue that you need to resolve.
To get the latency value for all DynamoDB calls, enable latency metric logging for the AWS SDK. These metrics can help you identify the source of the increased latency. Keep in mind that DynamoDB latency metrics measure activity within DynamoDB or Amazon DynamoDB Streams only. The latency metrics don't take network latency or client-side activity into account.
Note: For most atomic operations, such as GetItem and PutItem, you can expect an average latency in single-digit milliseconds. Latency for non-atomic operations, such as Query and Scan, depends on many factors, including the size of the result set and the complexity of the query conditions and filters.
Consider one or more of the following strategies to reduce latency:
- Reduce the request timeout settings: Tune the client SDK parameters requestTimeOut and clientExecutionTimeout to timeout and fail much faster (for example, after 50 ms). This causes the client to abandon high latency requests after the specified time period and then send a second request that usually completes much faster than the first. For more information about timeout settings, see Tuning AWS Java SDK HTTP request settings for latency-aware Amazon DynamoDB applications.
- Reduce the distance between the client and the DynamoDB endpoint: If you have globally dispersed users, consider using global tables. With global tables you can specify the AWS Regions where you want the table to be available. This can significantly reduce latency for your users.
- Use caching: If your traffic is read heavy, consider using a caching service such as DynamoDB Accelerator (DAX). DAX is a fully managed, highly available, in-memory cache for DynamoDB that delivers up to a 10x performance improvement—from milliseconds to microseconds—even at millions of requests per second.
- Send constant traffic or reuse connections: When you're not making requests, consider having the client send dummy traffic to a DynamoDB table. Alternatively, you can reuse client connections or use connection pooling. All of these techniques keep internal caches warm, which helps keep latency low.
- Use eventually consistent reads: If your application doesn't require strongly consistent reads, consider using eventually consistent reads. Eventually consistent reads are cheaper and are less likely to experience high latency. For more information, see Read Consistency.