Amazon Kinesis Producer Library (KPL) now supports ListShards API for efficient scaling of producer applications
Amazon Kinesis Producer Library (KPL) now supports the ListShards API to make it easier for your producer applications to publish data into Amazon Kinesis Data Streams at any scale. Amazon Kinesis Data Streams is a massively scalable and durable real-time data streaming service. The Kinesis Producer Library (KPL) simplifies producer application development, enabling developers to achieve high write throughput to a Kinesis data stream.
Amazon Kinesis Data Streams enables you to efficiently scale your data stream capacity without requiring any changes to producers and consumers. A producer application typically needs to discover the new shards in the stream after it has scaled up or down. Amazon KPL Version 0.14.0 replaces the DescribeStream with the ListShards API for shard discovery. The ListShards API supports 100TPS per stream compared to DescribeStream that supports 10TPS per account. For an account with 10 streams, using KPL v0.14.0 will provide a 100X higher call rate for shard discovery, eliminating the need for a DescribeStream API limit increase for scaling. Support for ListShards API is available with the Kinesis Client Library (KCL) v1.9.0 and above for default consumers and v2.0 and above for enhanced fan out (EFO) consumers. You can find more information on the ListShards API in the Amazon Kinesis Data Streams documentation.
To benefit from this additional API call rate limit, you will need to upgrade your KPL applications to version 0.14.0 or higher, and upgrade KCL to version v1.9 and above or v2.0 and above for EFO consumers.