How Cornerstone Built a Full-Fledged Apache Kafka Replication Using Amazon MSK Connect

By Suraj Talreja, Cloud Migration and Data Architect – Cornerstone Consulting Group
By Hari Ramesh, Sr. Partner Solution Architect – AWS

Cornerstone

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Amazon Managed Streaming for Apache Kafka (Amazon MSK) makes it easy to ingest and process streaming data in real-time with fully managed Apache Kafka. It eliminates operational overhead, including the provisioning, configuration, and maintenance of highly available Apache Kafka clusters.

Amazon MSK offers a feature called Amazon MSK Connect, which allows you to run fully managed Apache Kafka Connect workloads on Amazon Web Services (AWS). This feature makes it easy to deploy, monitor, and automatically scale connectors that move data between Apache Kafka clusters and external systems such as databases, file systems, and search indices.

With these advantages, customers are looking to migrate to Amazon MSK as they replatform or modernize data streaming solutions. However, migrating self-managed Kafka clusters either on-premises or on Amazon Elastic Compute Cloud (Amazon EC2) can come with challenges. In conversations with customers, we have come across various business and technical requirements, including:

• No data loss.
• Minimal to no application code change.
• Cost savings by migrating to a managed Kafka service.
• Auto-scaling and elasticity to manage sudden spikes in data traffic without manual intervention.
• High availability in AWS by deploying a multi-Availability Zone (AZ) Kafka cluster.
• Kafka topics should remain the same between the source and target clusters.
• Configurations for the topics, such as replication factor per topic, should remain the same.
• Achieve the same or better performance and security as the on-premises Kafka clusters.

In this post, we will discuss a migration solution for Amazon MSK leveraging the MM2 Replication Plugin developed by Cornerstone Consulting Group.

Cornerstone is an AWS Specialization Partner and AWS Marketplace Seller with the Amazon MSK service delivery designation. Bringing together a wealth of experience in migrating workloads to AWS, Cornerstone’s expertise is positioned to extend greater advantages to clients beyond their cloud journey.

Solution Overview

There are various tools out there that replicate data between two Kafka clusters, but Cornerstone leveraged the MirrorMaker 2 (MM2) replication engine in performing successful Kafka migrations to Amazon MSK.

MM2 is built on the Kafka Connect framework and provides unidirectional and bi-directional replication policies, as shown in Figure 1. It automatically detects new topics and partitions, while also ensuring the topic configurations are synced between clusters.

Figure 1 – MM2 bi-directional replication.

Even though the tool provides reliable, secure, and real-time replication between Kafka clusters, installing and configuring the tool is not always straightforward. Moreover, MM2 does not provide easy integration with monitoring and notification tools, which makes incident detection and resolution difficult (the problems get compounded if your source Kafka cluster is deployed on Kubernetes).

Cornerstone’s MM2 Replication Plugin

Cornerstone listened to its customers and developed a plugin that can be configured in minutes to enable real-time replication between your Kafka cluster and Amazon MSK. The plugin is built on the Amazon MSK Connect platform, scales automatically to meet source cluster workload requirements, and natively integrates with Amazon CloudWatch and Amazon Simple Notification Service (SNS) for monitoring and notifications.

The subsequent sections of this post outline the steps you need to perform to get the plugin installed for a successful Kafka migration to Amazon MSK.

Plugin Installation

You can download the plugin from AWS Marketplace. The steps are similar for serverless MSK (with AWS Identity and Access Management authentication) and are explained in detail in the user guide you can download along with the plugin.

At a high-level, you’ll have some steps to perform via the AWS Management Console‘s MSK service page, followed by the AWS CloudFormation template which will deploy the remaining AWS services.

Figure 2 – Manual steps via AWS console and automated steps via AWS CloudFormation.

Step 1: Create a Worker Configuration Through the MSK Console

• On the Amazon MSK service page, expand MSK Connect, go to Worker configurations, and click on the Create worker configuration button.
• Give a name for the Worker configuration and accept other defaults. Note that this is a manual process, as creating a new worker configuration is not supported by CloudFormation.

Figure 3 – Worker setting and parameters.

Step 2: Create MSK Connect Customized Plugin

• Upload the plugin zip file from EC2 (which you deployed from AWS Marketplace) to one of your Amazon Simple Storage Service (Amazon S3) buckets.
• On the Amazon MSK service page, expand MSK Connect, click on Customized plugins, and enter the URI of the S3 bucket with the plugin zip file. Click on the Create customized plugin button to complete the creation process. Note that this is a manual process, as creating a customized plugin is not supported by CloudFormation.

Figure 4 – Customized plugin settings.

Step 3: Deploy the CloudFormation Template

Deploy the CloudFormation Template to create the following resources:

• Amazon MSK Connect connectors: Creates source, checkpoint, and heartbeat MSK Connect connectors.
• AWS IAM role: Gives MSK Connect connectors the permissions to replicate to the target MSK cluster.
• Amazon CloudWatch log groups: Creates two log groups for the MSK Connect connectors.
• Amazon CloudWatch custom metrics: Defining metric filters and thresholds.
• Amazon CloudWatch alarms: Creates alarms which send notification to an SNS topic.
• Amazon SNS: For notification topic subscription and one-way communication over SMTP protocol.

Figure 5 – CloudFormation parameters and settings.

The CloudFormation template takes the following parameters as inputs:

• ConnectorConfigurationSourceClusterBootstrapServer (source bootstrap server).
• ConnectorConfigurationTargetClusterBootstrapServer (comma-separated destination MSK cluster bootstrap server).
• CustomPluginARN (Amazon Resource Name of the customized plugin in the MSK Connect you created during this setup).
• DestinationMSKArn (ARN of your destination MSK cluster).
• KafkaClusterVersionParam (Apache Kafka version number).
• MSKVPCSecurityGroup1 (destination MSK cluster security group ID).
• MSKVPCSubnet1ID (destination MSK cluster first subnet ID).
• MSKVPCSubnet2ID (destination MSK cluster second subnet ID).
• WorkerConfigurationARN (ARN of the worker configuration you crated during this setup).

Step 4: Subscribe to SNS Notification

The CloudFormation stack creates an SNS topic with the name msk-mm2-health. Subscribe to this topic with the desired protocol of your choice (such as email or SMS) to get notified when the source or destination cluster metrics are in the “In alarm” state.

After performing the above steps, you’ll have a target-state architecture similar to the one shown below.

Figure 6 – CloudFormation source and destination settings.

A Special Mention on CloudWatch Metrics

Monitoring is enabled via the msk/mm2_Source_Checkpoint_Connector and msk/mm2_Heartbeat_Connector log groups. The following CloudWatch metrics are created by the CloudFormation template and can be customized by editing the CloudFormation template.

Cleanup

To clean up your AWS account, perform the following steps in order:

• Delete the CloudFormation stack.
• Delete the MSK customized plugin.

Note that AWS does not support deleting the worker configuration. No costs are incurred by not deleting the worker configuration.

Conclusion

In this post, we saw how the plugin developed by Cornerstone automates and abstracts the complex tasks for setting up replication between your source and target Amazon MSK cluster.

The source cluster can be on-premises or in AWS and remains online for the duration of the migration. During the cutover window, simply stop the source cluster and point your producer and consumer applications to the MSK broker endpoint for a successful migration to Amazon MSK.

The plugin is available for free for a 90-day migration window. Visit Cornerstone in AWS Marketplace to download the plugin and get a head start on your Kafka migration.

.

.

Cornerstone Consulting Group – AWS Partner Spotlight

Cornerstone is an AWS Specialization Partner recognized for having deep Kafka migration expertise. Its overriding goal is to provide clients with an end-to-end migration service that delivers on time, is architected the right way, and completes within budget.

Contact Cornerstone | Partner Overview | AWS Marketplace