I want to put records into a Kinesis data stream using the Amazon Kinesis Producer Library (KPL).
Before completing the steps below, be sure that:
- You have a running Amazon Elastic Compute Cloud (Amazon EC2) Linux instance.
- An AWS Identity and Access Management (IAM) role is attached to your instance.
- The KinesisFullAccess policy is attached to the instance's IAM role.
1. Connect to your EC2 instance.
2. Install the latest version of the OpenJDK 8 developer package:
sudo yum install java-1.8.0-openjdk-devel
3. Confirm that Java is installed:
java -version
You should get an output like this:
java version "1.7.0_181" OpenJDK Runtime Environment (amzn-2.6.14.8.80.amzn1-x86_64 u181-b00) OpenJDK 64-Bit Server VM (build 24.181-b00, mixed mode)
4. Run the following commands to set Java 1.8 as the default java and javac providers:
sudo /usr/sbin/alternatives --config java sudo /usr/sbin/alternatives --config javac
5. Add a repository with an Apache Maven package:
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
6. Set the version number for the Maven packages:
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
7. Use yum to install Maven:
sudo yum install -y apache-maven
8. Confirm that Maven is installed properly:
mvn -version
You should get an output like this:
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) Maven home: /usr/share/apache-maven Java version: 1.7.0_181, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.181.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "4.14.33-51.37.amzn1.x86_64", arch: "amd64", family: "unix"
9. Install git, and then download the KPL from AWS Labs.
sudo yum install git git clone https://github.com/awslabs/amazon-kinesis-producer
10. Open the amazon-kinesis-producer/java/amazon-kinesis-producer-sample/ directory and then list the files.
cd amazon-kinesis-producer/java/amazon-kinesis-producer-sample/ ls default_config.properties pom.xml README.md src target
11. Run a command similar to the following to create a Kinesis data stream. If you're not sure how many shards you need, see Additional Information About Shards.
aws kinesis create-stream --stream-name kinesis-kpl-demo --shard-count 2
12. Run list-streams to confirm that the stream was created:
aws kinesis list-streams
13. Open the SampleProducer.java file and modify the following fields:
For public static final String STREAM_NAME, enter the name of the stream that you created in step 11.
For public static final String REGION, enter the Region that you're using.
cd src/com/amazonaws/services/kinesis/producer/sample vi SampleProducer.java public static final String STREAM_NAME = "kinesis-kpl-demo"; public static final String REGION = "us-east-1";
14. Run the following command in the amazon-kinesis-producer-sample directory so that Maven can download all dependencies.
mvn clean package
15. Run the following command in the amazon-kinesis-producer-sample directory to execute the producer and put data into the stream:
mvn exec:java -Dexec.mainClass="com.amazonaws.services.kinesis.producer.sample.SampleProducer"
16. To see how many records are put into the stream, check the Incoming Data (Count) graph on the Monitoring tab of the Kinesis console.
Note: The record count might be lower than the number of records that you put into the data stream. This is because the KPL uses Aggregation.
Did this page help you? Yes | No
Back to the AWS Support Knowledge Center
Need help? Visit the AWS Support Center
Published: 2019-02-11