Amazon Kinesis Data Streams now supports scaling up to 10,000 MB/s throughput with a single API call
Starting today, you can use the UpdateShardCount API in Amazon Kinesis Data Streams to scale up to 10,000 shards for a single data stream. You can use Amazon Kinesis Data Streams to capture, store, and analyze terabytes of data per hour from clickstreams, financial transactions, social media feeds, and more. The UpdateShardCount API enables you to rapidly scale the capacity of data streams up or down to handle changes in data volumes without any disruption to the producing or the consuming applications.
A Kinesis shard provides 1 MB/s ingress and 2 MB/s egress throughput. You can increase stream throughput by adding more shards. Previously, the UpdateShardCount API could scale up to 500 shards. Today’s announcement enables you to rapidly scale your steam capacity up to 10,000 shards, a 20X increase. As an example, if you had 10 shards delivering 10 MB/s throughput, you could previously scale up to 500 shards to ingest 500 MB/s. Now, you can scale up to 10,000 shards or 10,000 MB/s in response to traffic increase with a single API call or a click in the console. You can then scale down capacity after a reduction in traffic.
Updating the shard count is an asynchronous operation. Kinesis Data Streams performs splits or merges on individual shards to update the shard count. You can continue to read and write data to your stream while the scaling operation is in progress. We recommend using KPL version 0.14.0 or higher, and upgrade KCL to version 1.9 or above for using this capability. Please refer to the API Reference documentation for more details.
The 10,000 shard increase for the UpdateShardCount API is available in all AWS regions.