How do I troubleshoot a “No endpoints available” error in the DAX client?

Short description

The "No endpoints available" error occurs when the client doesn't have a network route to any nodes of the DAX cluster. This error might also occur when a cluster is down or experiencing high CPU utilization.

Here are some example cases that might result in this error:

• A failed health check to DAX nodes: The client can't perform a health check on cluster nodes because of high CPU load from the nodes.
• Multiple consecutive I/O Exceptions to each node: The DAX client counts the number of I/O Exceptions to determine if a server node is down. The client retrieves a list of server nodes from the configuration endpoint. If it receives more than five failures from a server node, then the client removes the node from the list of active server nodes. If all server nodes are removed in this way, then the client ends up with an empty list of active nodes. In response, the client shows a "No endpoints available" error.

Resolution

To resolve the error, use the following troubleshooting steps:

• In the DAX client, there's a thread to refresh the list of active server nodes. To restore your active nodes in the list, use the refresh option.
• Increase the RequestTimeout based on the client's traffic pattern. For more information, see Tuning AWS Java SDK HTTP request settings for latency-aware Amazon DynamoDB applications.
• Increase the health check interval. By default, the health-check-interval is 5 seconds.
  Note: Increasing this interval puts the endpoint back into the client-side endpoint set only when it's removed because of a socket-timeout-exception.
• To reduce the reoccurrence of this error, scale up to a larger instance type. Or, add more nodes to the cluster to scale out.
• This error is more likely to occur with single node clusters. If your cluster has only one node, then scale out the cluster. Place each node in a different Availability Zone.

Related information

ClientConfig (DAX)