Developing a Serverless, Fully-Managed Connected Vehicle Platform with Real-Time Analytics on AWS

By Sanchit Jain, Lead Solution Architect, AWS – Quantiphi

Quantiphi

In a recent customer engagement, Quantiphi developed a serverless, fully-managed connected vehicle platform that enabled real-time analytics on telematics data for one of the world’s leading car-as-a-service companies.

This organization has 1.8 million vehicles spread across 30 countries and specializes in automobile leasing and fleet management. The proposed solution saved 50% of the monthly costs and resulted in 70% of the production release for the client.

In this post, we’ll understand the significance of Internet of Things (IoT)-based telematics solutions in transforming the fleet management industry.

I will also highlight the critical aspects of Quantiphi’s serverless, fully-managed connected vehicle solution built on Amazon Web Services (AWS) and describe how it helped a U.S.-based fleet management organization make better decisions and improve customer and asset service.

Quantiphi is an AWS Competency Partner and leading artificial intelligence-first company driven by the desire to solve transformational problems at the heart of the business.

What is Telematics Data and Connected Vehicle Technology?

Innovation in the automotive industry has resulted in vehicles evolving into even more powerful manifestations of the internet. Connected vehicle technology promises a potential for a reduction in speeding, improvement in fuel economy, and reduction in accidents of vehicles.

In a novice way, any vehicle or car which can connect to the internet is called a “connected vehicle.” Normally, such vehicles connect to the internet via Wireless Local Area Network (WLAN).

A connected vehicle can also receive the internet with devices inside and outside the car, and at the same time can also share data with any external device or services.

Fleet managers shifted their gear to connected vehicle technologies in the early 2010s partly because telematics data can help them remotely monitor every aspect of driver and vehicle behaviors.

Maintenance schedules and mechanical repairs can also integrate into telematics systems to minimize vehicle downtime and reduce instances of mechanical failure. This results in savings on lifetime vehicle repair costs, tow bills, and driver downtime.

A telematics system is only valuable, however, if fleet owners can identify and act on issues as they emerge in real-time to make decisions at the right time. This is not possible in the current setup which lacks real-time insight and notifications.

Connected Vehicle Platform Solution on AWS

Quantiphi’s client had an existing platform that ran on-premises in a rudimental way and was built using the monolithic framework, leading to the some pain points:

• No real-time analytics due to lack of on-demand scalability, in terms of storage and compute power, and the existing support capability to summarize the data in a weekly batch mode.
• Lack of dynamic scalability because existing systems fail to provide elasticity in terms of storage and compute, which makes the existing solution cost infective.
• Difficult troubleshooting because the system was designed using monolithic patterns and fails to provide the capability to isolate issues and troubleshoot the problem.

Quantiphi developed a novel solution to set up a connected vehicle platform that adjusted to the client’s existing on-premises setup and design using AWS-native services.

The connected vehicle platform answers queries on a vehicle and driver behavior such as:

• Has the vehicle fastened safety belts while en route?
• Has the vehicle reached the maximum speed limit?
• Is the driver idling or hard braking?
• Where is the vehicle?
• What is the battery life percent?

Understanding the driver’s behavior from raw telematics data was a key focus for Quantiphi, as it offers extensive data points to the client to efficiently identify and improve the field workforce’s driving skills and safety.

There are several benefits to utilizing driver telematics data, including:

• Improved safety: Drivers can instantly recognize risky driving practices and modify their practices to enhance their on-road performance.
• Better insurance pricing: Fleets use telematics data to help facilitate usage-based insurance programs by following vehicle driving data. These programs offer better incentives and lower prices to fleets who adopt safer driving habits or reduce their mileage.
• Better claims ratios and fraud detection: With the aid of telematics data, fleets claims ratios can be improved and claims costs reduced.
• Vehicle tracking: Telematics data includes GPS data, which is used to locate fleet vehicles. It’s often used to reroute vehicles, as well as to assist drivers in discovering the economical routes.
• Maintenance improvements: Telematics data assists in predictive maintenance, meaning fleets can maintain and repair issues with less downtime.

Connected Vehicle Platform Overview

Quantiphi devised a serverless and fully managed pull-push mechanism to fetch raw data and ingest processed data to Amazon Redshift, a cloud data warehouse system, to provide analytics and visualization capabilities around the telematics data with ​​the best price-performance at any scale.

Figure 1 – Solution overview.

Data Collection

• AWS Lambda is leveraged to pull raw telematics data seamlessly from Internet of Things (IoT) devices installed in the fleet to AWS in real-time.
• With every pull, a pagination token is captured and stored in Amazon DynamoDB. In case of any temporary platform failure, the Lambda function always pulls data from the token’s last read point, resulting in no real-time data loss.

Data Pre-Processing and Data Storage

• Telematics data is extracted from IoT devices and buffered into Amazon Kinesis Data Firehose, containing diagnostic-level data such as oil pressure and battery life percent.
• A Lambda function invoked via Kinesis Data Firehose enriches the data with the ‘OpenStreetMap’ public dataset via Amazon Athena.
• Amazon Kinesis Data Analytics analyzes the events in the combined raw data by parsing them, and then generates event logs data. This data is persisted to Amazon Simple Storage Service (Amazon S3) via Kinesis Data Firehose in an Apache Parquet format.

Event Generation: Real-Time Analytics and Summary Generation

• Real-time event generation:
  • Amazon Kinesis Data Analytics analyzes the events in the combined raw data by parsing them, and then generates event logs data in a near real-time mode.
  • Different data points extracted from raw data by Kinesis Data Analytics are hard braking, harsh acceleration, max speed, and posted speed.
• Daily summary:
  • AWS Glue jobs are leveraged to generate and load daily summary data of vehicles and event summary data into Amazon Redshift.
  • Different data points extracted from the processed data for summarization are tire pressure, average miles per hour, average miles per gallon, and total trip distance.

Data Consumption

• Processed actual events and summarized data are loaded into Amazon Redshift and then integrated with Amazon QuickSight to present different visualization.
• Rest APIs are also designed on Amazon Redshift to provide data extraction capabilities to the end client and different internal business tools.

Connected Vehicle Platform Architecture

Quantiphi’s solution is designed to provide a framework for connected vehicle services, focusing on extending the solution’s functionality rather than managing the underlying infrastructure operations.

Figure 2 – Solution architecture of event generation pipeline.

Event Generation Pipeline

1. The Data Collection Lambda function is triggered via an Amazon EventBridge Rule every 15 minutes. It establishes a secured connection with the IoT device, and the time to live for auth-token is two hours.
   .
   After the connection is established, a post request is executed to pull the data buffered in the IoT device for which client-id and auth-token passed as request body. The response from the IoT device is the telematics data in a JSON format and the pagination token. Telematics data is buffered to Amazon Kinesis Data Firehose, and the pagination token is preserved in the DynamoDB for a subsequent call.
   .
2. Data from Kinesis Data Firehose is persisted to the S3 bucket if the data size in the buffer is less than 5 MB, or the duration of data in the buffer is less than 300 seconds.
   .
3. As data is persisted to S3 by Kinesis Data Firehose, an upload event is generated and captured in Amazon Simple Queue Service (Amazon SQS), which triggered Enrich Lambda.
   .
4. Enrich Lambda processes the raw data, executes the Amazon Athena query against the ‘OpenStreetMap’ data set, and retrieves the max speed value for the provided latitude and longitude. This max speed value assists in identifying the speeding event. Finally, the enriched data is sent to Amazon Kinesis Data Firehose.
   .
5. Amazon Kinesis Data Firehose, which holds enriched data, is connected with Amazon Kinesis Data Analytics which helps extract near-real-time events like speeding, idling, harsh acceleration, and hard acceleration braking.
   .
   Simultaneously, enriched data from Kinesis Data Firehose is persisted to an S3 bucket under a processed zone in either of the situations if the data size buffered is less than five MB, or the duration of data in the buffer is less than 300 seconds.
   .
6. Events generated by Amazon Kinesis Data Analytics are stored in Amazon Redshift for QuickSight visualization, and in S3 with the help of Kinesis Data Firehose.
   .
7. Amazon Athena is deployed on the output bucket for any ad-hoc analysis. Similarly, different visualization charts are created in QuickSight based on the data persisted in the warehouse. QuickSight dashboards are embedded into the web application and offered to end clients to understand the driver behavior.

Summary Generation Pipeline

Below is the architecture that summarizes the events generated daily and persisting the data into the Amazon Redshift warehouse system, which is then connected with Amazon QuickSight to provide visualization capability.

Figure 3 – Solution architecture of summary generation pipeline.

1. The Daily Summary Lambda function is scheduled to trigger via Amazon EventBridge Rule every 2 a.m. EST. On invocation, Lambda fires an AWS Glue job for different clients.
   .
2. AWS Glue job operates on the enriched data stored in the processed Amazon S3 bucket. It generates the summarized output stored in Parquet format in S3 for future reference and loads it into the Amazon Redshift warehouse for QuickSight visualization.
   .
3. Amazon Athena is deployed on the output bucket for any ad-hoc analysis. Similarly, different visualization charts are created in QuickSight based on the data persisted in the warehouse.
   .
4. Amazon QuickSight dashboards are finally embedded into the web application and offered to end clients to understand the driver behavior.

DevOps Practice for Connected Vehicle Platform

Quantiphi demonstrated the power of serverless and provided advisory support on security and deployment best practices.

The following DevOps practices resulted in a 70% increase in production release for the client:

• Continuous integration using Jenkins and AWS CloudFormation templates:
  • Quantiphi used Jenkins for easy deployment of code present in the Github repository. The workflow was specified in the form of a release cycle, starting from the dev stage to the QA stage and finally into the production stage.
  • Reusable nested CloudFormation templates were developed to plug in and out resources without much change in the parent template.
• Continuous delivery using AWS managed services:
  • The entire connected vehicle platform uses services such as AWS Lambda, AWS Glue, and Amazon DynamoDB. Amazon SQS was used for decoupling, ensuring continuous delivery in a distributed system.
• Monitoring mechanism:
  • A complete configuration monitoring mechanism was implemented with the help of Amazon CloudWatch and Amazon Simple Notification Service (Amazon SNS). AWS Config was enabled to monitor all the services.
  • AWS CloudTrail was enabled to monitor all of the API calls done on all the resources configured in the AWS environment.
• Distributed tracing:
  • Services like AWS X-Ray and Amazon CloudWatch Logs Insights were used for a performance review to get the memory usage and time consumption of Lambda functions.

Summary

Through the evolutions of telematics, commercial fleets are provided crucial learning to efficiently optimize their operations. Driver telematics data eventually contributes to long-term value and cost savings, as well as effective insights to increase workforce productivity.

By using Quantiphi’s cloud-native connected vehicle solution built on AWS, fleet companies can process up to 200,000 events per month, generate real insights, and facilitate better decision making.

The content and opinions in this blog are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

.

.

Quantiphi – AWS Partner Spotlight

Quantiphi is an AWS Competency Partner and leading artificial intelligence-first company driven by the desire to solve transformational problems at the heart of the business.

Contact Quantiphi | Partner Overview

*Already worked with Quantiphi? Rate the Partner

*To review an AWS Partner, you must be a customer that has worked with them directly on a project.