Getting Started / Hands-on / ...

Projects on AWS:

Build a serverless Real-Time Data Processing App

with AWS Lambda, Amazon Kinesis, Amazon S3, Amazon DynamoDB, Amazon Cognito, and Amazon Athena

Intro

Build a data stream

Aggregate data

Process streaming data

Store & query data

Extra credit & clean up

Real-time Streaming Data

In this module, you’ll create an Amazon Kinesis stream to collect and store sensor data from our unicorn fleet. Using the provided command-line clients, you’ll produce sensor data from a unicorn on a Wild Ryde and read from the stream. Lastly, you’ll use the unicorn dashboard to plot our unicorns on a map and watch their status in real-time. In subsequent modules you’ll add functionality to analyze and persist this data using Amazon Kinesis Data Analytics, AWS Lambda, and Amazon DynamoDB.

The architecture for this module involves an Amazon Kinesis stream, a producer, and a consumer.

Our producer is a sensor attached to a unicorn currently taking a passenger on a ride. This sensor emits data every second including the unicorn’s current location, distance traveled in the previous second, and magic points and hit points so that our operations team can monitor the health of the unicorn fleet from Wild Rydes headquarters.

The Amazon Kinesis stream stores data sent by the producer and provides an interface to allow consumers to process and analyze those data. Our consumer is a simple command-line utility that tails the stream and outputs the data points from the stream in effectively real-time so we can see what data is being stored in the stream. Once we send and receive data from the stream, we can use the unicorn dashboard to view the current position and vitals of our unicorn fleet in real-time.

Time to complete module: 25 Minutes

Services used:
• Amazon Kinesis Data Streams
• Amazon Cognito
• AWS IAM

Application Architecture

  • Step 1. Create an Amazon Kinesis stream

    Step 1. Create an Amazon Kinesis stream

    Use the Amazon Kinesis Data Streams console to create a new stream named wildrydes with 1 shard.

    a. Go to the AWS Management Console, select Services then select Kinesis under Analytics.

    b. Select Get started if prompted with an introductory screen.

    c. Select Create data stream.

    d. Enter  wildrydes  into Kinesis stream name and 1 into Number of shards, then select Create Kinesis stream.

    e. Within 60 seconds, your Kinesis stream will be ACTIVE and ready to store real-time streaming data. 

  • Step 2. Produce messages into the stream

    Step 2. Produce messages into the stream

    Use the command-line producer to produce messages into the stream.

    a. Switch to the tab where you have your Cloud9 environment opened.

    b. In the terminal, run the producer to start emiting sensor data to the stream.

    ./producer

    c. The producer emits a message a second to the stream and prints a period to the screen.

    d. In the Amazon Kinesis Streams console, click on wildrydes  and click on the Monitoring tab.

    e. After several minutes, you will see the Put Record (Bytes) — Sum graph begin to record several thousand bytes put per minute.

  • Step 3. Read messages from the stream

    Step 3. Read messages from the stream

    a. Switch to the tab where you have your Cloud9 environment opened.

    b. Hit the (+) button and click New Terminal to open a new terminal tab.

    c. Run the consumer to start reading sensor data from the stream:

    ./consumer

    d. The consumer will print the messages being sent by the producer:

    {
  "Name": "Shadowfax",
  "StatusTime": "2017-06-05 09:17:08.189",
  "Latitude": 42.264444250051326,
  "Longitude": -71.97582884770408,
  "Distance": 175,
  "MagicPoints": 110,
  "HealthPoints": 150
}
{
  "Name": "Shadowfax",
  "StatusTime": "2017-06-05 09:17:09.191",
  "Latitude": 42.265486935100476,
  "Longitude": -71.97442977859625,
  "Distance": 163,
  "MagicPoints": 110,
  "HealthPoints": 151
}

  • Step 4. Create an identity pool for the unicorn dashboard

    Step 4. Create an identity pool for the unicorn dashboard

    Create an Amazon Cognito identity pool to grant unauthenticated users access to read from your Kinesis stream. Note the identity pool ID for use in the next step.

    a. Go to the AWS Management Console, select Services then select Cognito under Security, Identity & Compliance.

    b. Select Manage Identity Pools.

    c. Select Create new identity pool.

    d. Enter wildrydes  into Identity pool name.

    e. Tick the Enable access to unauthenticated identities checkbox.

    f. Click Create Pool.

    g. Click Allow which will create authenticated and unauthenticated roles for your identity pool.

    h. Select Go to Dashboard.

    i. Select Edit identity pool in the upper right hand corner.

    j. Note the Identity pool ID for use in a later step.

    k. Click Cancel.

  • Step 5. Grant the unauthenticated role access to the stream

    Step 5. Grant the unauthenticated role access to the stream

    Add a new policy to the unauthenticated role to allow the dashboard to read from the stream to plot the unicorns on the map.

    a. Go to the AWS Management Console, services Services then select IAM under Security, Identity & Compliance.

    b. Select Roles in the left-hand navigation.

    c. Select the Cognito_wildrydesUnauth_Role.

    d. Select Add inline policy.

    e. Select Choose a service and select Kinesis.

    f. Tick the Read and List permissions checkboxes in the Actions section.

    g. Select Resources to limit the role to the wildrydes  stream.

    h. Select Add ARN next to stream .

    i. Enter the region you're using in Region (e.g. us-east-1), your Account ID in Account, and wildrydes  in Stream name.

    To determine the Region you're using, see the prefix of the Amazon Cognito identity pool ARN you copied.   

    To find your twelve digit AWS Accout ID number in the AWS Management Console, select Support in the navigation bar in the upper-right, and then select Support Center. Your currently signed ID appears in the upper-right corner below the Support menu.

    j. Select Add.

    k. Select Review policy.

    l. Enter wildrydesDashboardPolicy in Name.

    m. Select Create policy.

    (click to zoom)

  • Step 6. View unicorn status on the dashboard

    Step 6. View unicorn status on the dashboard

    Use the Unicorn Dashboard to see the unicorn on a real-time map.

    a. Open the Unicorn Dashboard.

    b. Enter the Cognito Identity Pool ID you noted in step 4 and select Start.

    c. Validate that you can see the unicorn on the map.

    d. Click on the unicorn to see more details from the stream.

    (click to zoom)

  • Step 7. Experiment with the producer

    Step 7. Experiment with the producer

    Stop and start the producer while watching the dashboard and the consumer. Start multiple producers with different unicorn names.

    a. Stop the producer by pressing Control + C and notice the messages stop and the unicorn disappears after 30 seconds.

    b. Start the producer again and notice the messages resume and the unicorn reappears.

    c. Hit the (+) button and click New Terminal to open a new terminal tab.

    d. Start another instance of the producer in the new tab. Provide a specific unicorn name and notice data points for both unicorns in consumer’s output:

    ./producer -name Bucephalus

    e. Check the dashboard and verify you see multiple unicorns.

  • Recap and Tips

    Recap and Tips

    🔑 Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information.

    🔧 In this module, you’ve created an Amazon Kinesis stream and used it to store and visualize data from a simulated fleet of unicorns.

In the next module, you’ll create an Amazon Kinesis Data Analytics application to aggregate sensor data from the unicorn fleet in real-time.

Next: Module 2