AWS Developer Tools Blog
Announcing Amazon Kinesis SubscribeToShard API Support in the AWS SDK for Ruby
Amazon Kinesis launched two significant performance-improving features for Amazon Kinesis Data Streams: enhanced fan-out and an HTTP/2 data retrieval API (“SubscribeToShard”). This API allows data to be delivered from producers to consumers in 70 milliseconds or better. Today, we’re excited to announce the support for Kinesis SubscribeToShard API in the AWS SDK for Ruby.
Before calling the #subscribe_to_shard API
The #subscribe_to_shard
API is available in the aws-sdk-kinesis
gem version 1.10.0 and later for Ruby version 2.1 and later. This API is built on the HTTP/2 protocol for streaming back shard events, instead of the normal API call based on the HTTP1.1 protocol. You need to have http-2 gem available when using this API.
This API is available at AsyncClient
instead of Client
, and an AsyncResponse
object would be available after making the call instead of waiting for a complete sync Response
.
Introduction to AsyncClient
Due to the nature of the HTTP/2 protocol, the AWS SDK for Ruby introduced AsyncClient
for streaming APIs over HTTP/2. This is different from the Client
object, which you might already be familiar with, for APIs over HTTP1.1.
You can create an AsyncClient
for Kinesis as follows.
In the next section, we walk through the async unidirectional streaming API usage pattern.
For general information about the SubscribeToShardAPI, see the Kinesis SubscribeToShard API documentation
#subscribe_to_shard API usage pattern
In this section, we will go through key parts for making a unidirectional async API call.
Intro to :output_event_stream_handler
Waiting for async responses to be synchronized takes minutes and misses all of the benefits of streaming the processing of events. We’ve introduced event stream objects that enable you to register callbacks for specific (or every) event that arrives.
Taking Kinesis as an example, all available EventStreamobjects
would be under Aws::Kinesis::EventStreams
. When looking at the #subscribe_to_shard
API, we have Aws::Kinesis::EventStreams::SubscribeToShardEventStream
available.
To use those event streams, you just need to provide them in the :output_event_stream_handler
option when making the request. The following sections provide a complete example.
Prepare the consumer
Before calling the API, you would need to have a consumer using enhanced fan-out. If you already have this consumer living in Kinesis data streams, you may skip this section and jump to following section “Calling the API”.
To create a consumer using enhanced fan-out, you can simply call #register_stream_consumer
available at Aws::Kinesis::Client
like:
Make sure check until the consumer is active before using it for subscribing shard events:
Then, let pick a shard that we will subscribe to, (you would need the shardId)
If you already have some fresh data in the stream shard, you can skip the rest of this section and jump to “Calling the API”, else, you can use following code snippets to put some data for testing
Calling the API
Let’s bring all the pieces together and make the API call.
For full documentation of request parameters, see here. For more information about preparing those parameter values, see the Kinesis documentation RegisterStreamConsumer.
When you run the above code, you get output like the following.
Additional notes
In addition to providing an Aws::Kinesis::EventStreams::SubscribeToShardEventStream
object with callbacks that are registered (the most recommended usage pattern), you can also simply provide a Ruby Proc object with those callbacks, such as the following.
You can also use a block when making the request to access the stream. In this pattern, you don’t have to provide :output_event_stream_handler
.
A hybrid usage pattern is also supported.
One more note for the hybrid usage pattern: When extra callbacks are registered from the block, they’re essentially tracked (appended) at :output_stream_handler
still.
Final thoughts
We walked through a #subscribe_to_shard
API usage example in Ruby to give you an introduction to AsyncClient
and AsyncResponse
. Feel free to try out the usage pattern to process events with callbacks, and let us know if you have any questions.
Feedback
Share your questions, comments, and issues with us on GitHub. You can also catch us in our Gitter channel.