Monitor IoT device health at scale with Amazon Managed Grafana­­

Businesses today employ IoT devices to monitor the health of their equipment, ranging from machines on a factory floor to inventory tracking sensor locations. Insights from these IoT device fleets make them part of critical business infrastructure, however deriving meaningful insights from these IoT device fleets at scale is a common challenge customers face. IT teams that are responsible for managing the uptime, data latency, and security of these devices need the tools to help them identify and pinpoint issues, and to resolve them without delays. Amazon Managed Grafana is a fully managed, cross-project service with rich, interactive data visualizations to help customers analyze, monitor, and alarm on metrics, logs, and traces across multiple data sources.

In this post, we build a sample IoT health dashboard using Amazon Managed Grafana, that can scale to thousands of devices.

Solution Overview

Customers using IoT sensors in production environments rely on visibility of sensor data. When these devices fail, users often need to perform actions such as power cycling the device or changing the batteries to restore connectivity. How does one keep track of the health of devices without continually checking their individual status? Using Amazon Timestream and Amazon Managed Grafana, customers can create interactive dashboards to visualize their device status in near real-time.

IoT devices in the field send data securely to AWS IoT Core in the cloud. After the data reaches AWS IoT Core, we use the Rules for AWS IoT to invoke an AWS Lambda function. With Lambda, we normalize the data from different IoT devices and write them into Amazon Timestream database. We use Amazon Managed Grafana to query the data in Timestream, and to build the required visualizations. The following diagram illustrates the architecture.

Figure 1: IoT Device Monitoring Architecture.

We use the following AWS services:

1. AWS IoT Core – Connect to your devices through many different protocols with mutual authentication and end to end encryption to manage and scale your device fleets efficiently.
2. AWS Lambda – A serverless, event driven compute service that scales to your needs so you can run code for application or backend service.
3. Amazon Timestream – A fast, scalable, and serverless time-series database service that stores and analyzes trillions of events each day.
4. Amazon Managed Grafana – A fully managed service for Grafana & a popular open-source analytics platform to query, visualize, and alert on metrics, logs, and traces.
5. AWS CloudFormation – Lets you model, provision, and manage AWS and third-party resources by treating infrastructure as code.

We’ve published the AWS Cloud Development Kit (AWS CDK) project for this architecture as part of this blog. The next sections walk you through the steps to build this architecture in your AWS account.

Prerequisites

Before you get started, complete the following prerequisites:

• An AWS account with permissions to create AWS Identity and Access Management (IAM) roles, and user profiles.
• The AWS Command Line Interface (AWS CLI) version 2 installed and configured with credentials to access your account.
• NodeJS 14.15.0 or later and NPM
• The AWS Cloud Development Kit (AWS CDK) v2 installed. For more information, refer to Getting started with the AWS CDK.
• Python 3+ and the CDK libraries for Python.

Implementation time and cost estimation

Clone the GitHub repository

First, clone the GitHub repository. We’ve built a classic AWS CDK project with the directory monitor-iot-device-health-with-amazon-managed-grafana, which contains the construct and resources required to create the architecture for this solution.

git clone https://github.com/aws-samples/monitor-iot-device-health-with-amazon-managed-grafana
cd monitor-iot-device-health-with-amazon-managed-grafana

Next, run the following commands from your CDK project.

Note: if you are running python 3 under the alias python instead of python3, you need to adjust the following commands accordingly. Also, open cdk.json and change the first line from "app": "python3 app.py" to "app": "python app.py"

1. Create a virtual environment:
   1. For macOS/Linux, use python3 -m venv .venv.
   2. For Windows, use python3 -m venv .venv.
2. Activate the virtual environment:
   1. For macOS/Linux, use source .venv/bin/activate.
   2. For Windows, use .venv\Scripts\activate.bat.
   3. For PowerShell, use .venv\Scripts\activate.ps1.
3. Install the required dependencies:
   1. python3 -m pip install -r requirements.txt
4. Bootstrap your AWS environment with the following command:
   1. cdk bootstrap
5. Check your setup by running the following command which outputs the CloudFormation template:
   1. cdk synth

Check your setup with cdk synth, which outputs the cloud formation template for this solution

Bootstrap your account for the CDK using cdk bootstrap.

Install and setup the device simulator in your account

Follow the instructions to install AWS Solutions Library IoT Device Simulator in your account. You’ll use this to simulate your IoT devices. You need to enter an email address to initiate the AWS CloudFormation template. Use this to sign into the IoT Device Simulator site. To install the solution with the aws-cli, replace YOUR-EMAIL-ADDRESS field with your email address and run the following commands.

aws cloudformation create-stack \
--stack-name IoT-Simulator \
--template-url https://s3.amazonaws.com/solutions-reference/iot-device-simulator/latest/iot-device-simulator.template \
--capabilities CAPABILITY_IAM \
--parameters ParameterKey=UserEmail,ParameterValue=<<YOUR-EMAIL-ADDRESS>>

Setting up IAM Identity Center

Amazon Managed Grafana uses AWS IAM Identity Center or SAML to authenticate. If you aren’t using IAM Identity center in your account already, you must enable AWS IAM Identity Center before you can deploy the solution. Next, add a user in AWS IAM Identity Center to manage your Grafana dashboard.

NOTE: AWS IAM Identity Center requires AWS Organizations to be setup. If you don’t have this setup, follow the prompts to create a new Organizations when you enable IAM Identity Center. For more information refer to Creating and managing an organization.

Deploy the solution into your AWS account with the following command:

cdk deploy

Setup the IoT Device in the IoT Simulator

When the AWS CloudFormation template is completed, you will receive an email with instructions to sign in to your simulator demo. After you sign in, choose Device Types → Add Device Type → Import.

Locate the iot-device.json file in your /monitor-iot-device-health-with-amazon-managed-grafana/resources/simulator folder and upload it.

Figure2: Setting up the IoT Device in IoT Device Simulator.

Choose Save.

Setup the IoT Simulation

At this stage, the data schema for the IoT devices is setup in the simulator. The next step is to create a simulation with the number of devices, data transmission interval, and duration. From home, choose Add Simulation.

Figure 3: Adding a simulation in IoT Device Simulator

In the form use the following values

• Name your simulation: iot-telemetry
• Simulation type: User created
• Choose a device type: sample-telemetry-payload
• Number of devices: 10
• Data transmission interval: 1
• Data transmission duration: 600

Choose Save to save the simulation configuration, as the following screenshot shows.

Figure 4: Simulation configuration in IoT Device Simulator

Complete the Amazon Managed Grafana Setup

Sign in to your AWS console and navigate to Amazon Managed Grafana. Choose your workspace; in this example, we use IoT-Health-Workspace.

Figure 5: Amazon Managed Grafana workspace.

Before you can use the Grafana dashboard, you need to grant admin access the users that require access. Choose Assign new user or group under the Authentication tab.

Figure 6: Assigning new user or group through AWS IAM Identity Center

Choose an appropriate user. Then, choose Assign user and groups.

From your list of users choose a user that you want to provide administrative permissions. Under the Action dropdown, choose Make admin.

Figure 7: Providing administrator access to your admin users.

For more details on managing users and permissions in Amazon Managed Grafana, refer to the user guide.

Start the simulator and explore the dashboard

Navigate to Amazon Managed Grafana from your AWS Management Console, choose the Workspace URL to sign in to your Grafana dashboard. The Grafana workspace is pre-loaded with a data source connection to Amazon Timestream, and a pre-built dashboard. Once logged in, navigate to the IoT Device Dashboard.

Figure 8: Amazon Managed Grafana Dashboard.

Your dashboard is empty on first load, as the device simulator is not sending any data yet. Now, let’s start the simulator to watch the data flow in real-time. Return to your device simulator console, choose the iot-telemetry simulation you created, and choose Start simulation(s).

Figure 9: Start simulation in IoT Device Simulator.

Take a look the results from the device simulator showing up on your Grafana dashboard, similar to the following screenshot.

Figure 10: Amazon Managed Grafana Dashboard showing the IoT Devices’ health.

Congratulations, you have successfully setup a dashboard for your IoT device fleet! Explore the IoT Core Rules and Amazon Timestream database to see how the data is flowing from AWS IoT Core to the Grafana dashboard.

Cleaning up

To decommission all the resources deployed during this walkthrough, use the following steps:

• Delete the stacks by running the following commands

aws cloudformation delete-stack --stack-name IoT-Simulator
cdk destroy

• If you aren’t using the CDK in your account for other work, delete this stack. Sign in to your AWS console and go to CloudFormation. Choose the stack named CDKToolkit and choose Delete
• If you enabled IAM Identity Center you can delete that configuration
• Delete any AWS Organizations created as part of AWS IAM Identity Center setup

If you run into any errors, you might have to delete resources manually, depending on your account configuration.

Conclusion

In this post, we showed you how to build an IoT health dashboard using Amazon Managed Grafana to provide visibility across your IoT device fleet. This allows your teams to effectively manage these devices at scale, and achieve critical business outcomes. Are you working on a similar use case, and have ideas to augment this dashboard? Let us know about your implementation!

About the authors: