How can I put records into a Kinesis data stream using the KPL?

Issue

I want to put records into a Kinesis data stream using the Amazon Kinesis Producer Library (KPL). 

Resolution

Before completing the steps below, be sure that:

  • You have a running Amazon Elastic Compute Cloud (Amazon EC2) Linux instance.
  • An AWS Identity and Access Management (IAM) role is attached to your instance.
  • The KinesisFullAccess policy is attached to the instance's IAM role.

1.    Connect to your EC2 instance.

2.    Install the latest version of the OpenJDK 8 developer package:

sudo yum install java-1.8.0-openjdk-devel

3.    Confirm that Java is installed:

java -version

You should get an output like this: 

java version "1.7.0_181"
OpenJDK Runtime Environment (amzn-2.6.14.8.80.amzn1-x86_64 u181-b00)
OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)

4.    Run the following commands to set Java 1.8 as the default java and javac providers:

sudo /usr/sbin/alternatives --config java 
sudo /usr/sbin/alternatives --config javac

5.    Add a repository with an Apache Maven package:

sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

6.    Set the version number for the Maven packages:

sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo

7.    Use yum to install Maven:

sudo yum install -y apache-maven

8.    Confirm that Maven is installed properly:

mvn -version

You should get an output like this: 

Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
Maven home: /usr/share/apache-maven
Java version: 1.7.0_181, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.181.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.14.33-51.37.amzn1.x86_64", arch: "amd64", family: "unix"

9.    Install git, and then download the KPL from AWS Labs.

sudo yum install git
git clone https://github.com/awslabs/amazon-kinesis-producer

10.    Open the amazon-kinesis-producer/java/amazon-kinesis-producer-sample/ directory and then list the files.

cd amazon-kinesis-producer/java/amazon-kinesis-producer-sample/
ls
default_config.properties  pom.xml  README.md  src  target

11.    Run a command similar to the following to create a Kinesis data stream. If you're not sure how many shards you need, see Additional Information About Shards.

aws kinesis create-stream --stream-name kinesis-kpl-demo --shard-count 2

12.    Run list-streams to confirm that the stream was created:

aws kinesis list-streams

13.    Open the SampleProducer.java file and modify the following fields:
For public static final String STREAM_NAME, enter the name of the stream that you created in step 11.
For public static final String REGION, enter the Region that you're using.

cd src/com/amazonaws/services/kinesis/producer/sample
vi SampleProducer.java

public static final String STREAM_NAME = "kinesis-kpl-demo";
public static final String REGION = "us-east-1";

14.    Run the following command in the amazon-kinesis-producer-sample directory so that Maven can download all dependencies.

mvn clean package

15.    Run the following command in the amazon-kinesis-producer-sample directory to execute the producer and put data into the stream:

mvn exec:java -Dexec.mainClass="com.amazonaws.services.kinesis.producer.sample.SampleProducer"

16.    To see how many records are put into the stream, check the Incoming Data (Count) graph on the Monitoring tab of the Kinesis console.
Note: The record count might be lower than the number of records that you put into the data stream. This is because the KPL uses Aggregation.

Writing to your Kinesis Data Stream Using the KPL 

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2019-02-11