Can I use different Amazon Kinesis Client Library applications with the same Amazon DynamoDB table?

Last updated: 2020-05-08

I'm trying to use different Amazon Kinesis Client Library (KCL) applications with the same Amazon DynamoDB table. How can I do this? What if I want to use something other than DynamoDB as a checkpointing table?

Short Description

There are two requirements for using the KCL:

  • Each KCL application must use its own DynamoDB table.
  • A DynamoDB table must be used for KCL checkpointing.

Resolution

Can I use different KCL applications using the same DynamoDB table?

Users can't use different KCL applications with the same DynamoDB table for the following reasons:

  • Scan operations are used to obtain leases from a DynamoDB table. Therefore, if a table contains leases of different KCL applications, each application could receive a lease that isn't related to the application itself. For more information about DynamoDB lease exceptions, see DynamoDBLeaseRefresher.java on the Amazon Web Services - Labs GitHub website.
  • Shard IDs in streams are used as primary keys in DynamoDB tables during checkpointing. When different KCL applications use the same DynamoDB table and the same shard IDs are used in the streams, inconsistencies in checkpoints can occur. For more information about DynamoDB shard IDs, see Application State Data.

Instead, you can use an application name as the default name for a DynamoDB table, and then manually set up a table name on the KCL. Here's an example of how to set up a table name on KCL version 1.x:

KinesisClientLibConfiguration kinesisClientLibConfiguration = // KCL 1.x
        new KinesisClientLibConfiguration(...)
        .withTableName(<table_name>);

Note: For more information about this KinesisClientLibConfiguration code example, see KinesisClientLibConfiguration.java on the Amazon Web Services - Labs GitHub website.

You can also manually set up a table name on KCL version 2.x:

ConfigsBuilder configsBuilder = new ConfigsBuilder(...).tableName(<table_name>); // KCL 2.x

Note: For more information about this ConfigsBuilder code example, see ConfigsBuilder.java on the Amazon Web Services - Labs GitHub website.

Can I use something other than DynamoDB as a checkpointing table?

Users can only use DynamoDB as a checkpointing table for the KCL. DynamoDB is required as a checkpointing table for the KCL, because the KCL behavior and implementation are interconnected with DynamoDB in the following ways:

  • The KCL includes ShardSyncTask.java, which guarantees that shard leases in a stream are included in the DynamoDB table. This check is conducted periodically in the KCL.
  • The KCL includes DynamoDBLeaseTaker.java and DynamoDBLeaseRenewer.java, which are components that manage and update leases in the KCL. DynamoDBLeaseTaker.java and DynamoDBLeaseRenewer.java work with DynamoDBLeaseRefresher.java to make frequent API requests to DynamoDB.
  • When the KCL makes checkpoints, requests from DynamoDBCheckpointer.java and DynamoDBLeaseCoordinator.java are made to DynamoDB. These requests are made through DynamoDBLeaseCoordinator.java, DynamoDBLeaseRenewer.java, or DynamoDBLeaseRefresher.java.

Did this article help you?

Anything we could improve?


Need more help?