Derive Insights from IoT in Minutes using AWS IoT, Amazon Kinesis Firehose, Amazon Athena, and Amazon QuickSight
Ben Snively is a Solutions Architect with AWS
Speed and agility are essential with today’s analytics tools. The quicker you can get from idea to first results, the more you can experiment and innovate with your data, perform ad-hoc analysis, and drive answers to new business questions.
Serverless architectures help in this respect by taking care of the non-differentiated heavy lifting in your environment―tasks such as managing servers, clusters, and device endpoints – allowing you to focus on assembling your IoT system, analyzing data, and building meaningful reports quickly and efficiently.
In this post, I show how you can build a business intelligence capability for streaming IoT device data using AWS serverless and managed services. You can be up and running in minutes―starting small, but able to easily grow to millions of devices and billions of messages.
AWS serverless services
AWS services offer a quicker time to intelligence. The following is a high-level diagram showing how the services in this post are configured:
Veronika Megler, Ph.D., is a Senior Consultant with AWS Professional Services
“We have a fleet of vehicles, with GPS and a bunch of other sensors,” said Bob, the VP at a delivery company. “Today they send their update ‘breadcrumbs’ to another IoT service. We’re planning to have them send their breadcrumbs to AWS IoT instead; that’ll let us remove the third-party service and integrate the breadcrumbs into our AWS infrastructure. Then, we’d like to store the breadcrumbs, do both near-realtime and batch analytics on them, and see the current locations on a map. How can we do that?”
This post shows you how.
AWS IoT is a managed cloud platform that lets connected devices communicate easily and securely. It also makes it easy to interact with cloud applications and other devices. With AWS IoT, you can filter, transform, and act upon device data on the fly, based on business rules that you define. You can update your rules to implement new device and application features at any time.
AWS IoT makes it easy to integrate and control your devices from other AWS services for even more powerful IoT applications. In particular, IoT provides tight integration with AWS Lambda, Amazon Kinesis, Amazon S3, Amazon Machine Learning, Amazon DynamoDB, Amazon CloudWatch, and Amazon Elasticsearch Service.
In this post, you’ll explore two of these integrations: Amazon S3 and Amazon Kinesis Firehose. The architecture you’ll design is shown in the figure below. You’ll set up your breadcrumbs to flow into S3 via two paths: directly, and also via Firehose. From there, you’ll access them in Amazon EMR using Apache Hive, and show your vehicles on a map using Hue. You’ll also use CloudWatch Logs for debugging purposes only.