Arity Boosts Kafka Adoption, Improves Operational Efficiency by 200%, and Reduces Costs by 40% with Lenses.io and Amazon MSK

Arity, wanted to eliminate the cost, complexity, and operational burden of self-managing Apache Kafka clusters. Working with AWS Partner Lenses.io, Arity migrated their mission-critical applications to fully managed Amazon Managed Streaming for Apache Kafka (Amazon MSK). As a result, Arity has seen a 200 percent improvement in operational efficiency and a 40 percent reduction in operational overhead and personnel costs. As part of the project, Arity also implemented additional AWS solutions that improve security, performance, and cost. 

Founded by The Allstate Corporation in 2016, Arity is a mobility data and analytics company that uses telematics data to better understand and predict driving behavior. Arity collects and analyzes driving data from over 30 million unique users to help insurance companies, automobile manufacturers, retail brands, municipalities, and others manage risk and operate more safely.

To process the constant influx of real-time telemetry data and deliver new services to their customers, Arity has long relied on Apache Kafka and AWS Advanced Partner Lenses.io. Lenses.io is a leader in developing software that helps teams simplify their operations on technologies such as Kafka and Kubernetes. The Lenses.io solution offers a developer experience for managing and operating Kafka and streaming data flows, enabling engineers to explore data generated by event-driven applications, continuously monitor their environment, and identify performance issues. Lenses.io takes the complexity out of Kafka by providing an intuitive user interface with governance and security that makes it easy for nearly 150 developers at Arity to embrace streaming application development.

However, as Arity’s adoption of Kafka and real-time applications grew, it became increasingly harder for the team to handle the cost and complexity of the self-managed Kafka infrastructure. Already running on Amazon Web Services (AWS), Arity chose to migrate to Amazon Managed Streaming for Apache Kafka (Amazon MSK).

Amazon MSK is a fully managed Apache Kafka service that operates, maintains, and scales Kafka clusters. By eliminating the operational burden required to provision servers, orchestrate patches and upgrades, and plan scaling events, Amazon MSK frees up technical teams and resources to focus on application development and business results.

Amazon MSK integrates seamlessly with the rest of Arity’s AWS stack, which includes Amazon Managed Service for Apache Flink to transform and analyze streaming data and Amazon Kinesis Data Firehose for extract, transform, and load (ETL) processes. Data is pulled for processing into Amazon Simple Storage Service (Amazon S3) and accessed using Amazon EMR and Amazon Athena.

To simplify the migration of their event-driven applications to Amazon MSK, Arity relied on the visibility provided by Lenses.io. “With Lenses.io, we were able to explore the data generated by applications and see key performance metrics, like how much data was flowing per second. This gave our team confidence that Amazon MSK was the right choice and that our apps could be migrated,” said Phil Falkenholm, director of infrastructure and security at Arity.

Thanks to the support of the Lenses.io and AWS teams, Arity migrated three large Kafka clusters with hundreds of applications and topics per cluster to Amazon MSK in less than five months.

Arity’s Real-Time Traffic team was the first to complete the migration and is now successfully processing over 5.8 billion geopoints per day with a peak processing load of 200,000 messages per second. Other teams have either completed or are in the process of their Kafka migration. “Getting to a higher-level version of Kafka was made immensely easier by migrating to Amazon MSK,” said Falkenholm. “The hard work and dedication across teams to complete this journey with no downtime or data loss was great to see.”

Since migrating to Amazon MSK, Arity has seen several performance improvements, including a 200 percent increase in operational efficiency and a 40 percent reduction in operational overhead costs. Before Amazon MSK, every year 13 teams of developers would work in two-week sprints rotating certificates used for authentication, a task that is no longer required.

In addition, Arity’s engineering team shifted full-time employees required to manage their Kafka clusters to new projects that deliver greater business value. “With Amazon MSK, we can expand our capabilities, add more storage and memory, and add different brokers in minutes rather than multiple hours,” Falkenholm noted.

Layering the self-service capabilities of Lenses.io on top of Amazon MSK has further decreased the number of service tickets Arity’s DevOps team have to process by 25 percent. Lenses.io includes a SQL interface that allows application developers to find events in a Kafka topic in seconds using a language they are already familiar with. Previously, this was a job for Arity’s DevOps team using insecure tools and scripting. With fewer tickets for managing Kafka infrastructure and data operations thanks to Amazon MSK and Lenses.io, Arity’s DevOps engineers and application developers are more productive, which means more time to focus on bringing additional real-time applications to market.

As part of the migration process, Arity moved from certificate-based security to AWS Identity and Access Management (IAM). This service enables Arity to securely manage identities and access to AWS resources by creating profiles for each role, which can be revoked if needed. Because IAM integrates seamlessly with Amazon S3, Arity can be confident that the sensitive personal data it stores is secure and accessible only to those who need it.

Arity is currently adopting AWS Glue Schema Registry, a fully managed solution for managing data streams in AWS environments. The Arity team believes that by stripping schema data from Apache Avro and moving it into the registry, it will lower overhead, reduce network load, and improve performance. Because Lenses.io supports Glue Schema Registry for Avro and Protobuf, the team can use Lenses.io to deserialize AVRO data, explore the data using SQL, and manage and evolve the schemas.

In addition, Arity has another project in the works that uses Lenses.io’s new Amazon S3 Kafka connector to automate the backup/restore process of Kafka data to Amazon S3 and optimize data integration for running queries using Amazon Athena.

“Whether it’s migration, security, or a new functionality that can improve our business, Lenses.io and AWS are always there for us,” concluded Falkenholm. “They’re two very good partners.”