Skip to main content

Amazon Managed Streaming for Apache Kafka

Migrate to Amazon Managed Streaming for Apache Kafka

Bring your Apache Kafka workloads to Amazon MSK Express brokers with Amazon MSK Replicator — no infrastructure overhead, no configuration complexity.

What is Amazon MSK Migration?

Amazon MSK migration refers to the process of moving Kafka workloads from other Kafka deployments including those running on-premises, on AWS, or other cloud providers, as well as Kafka-protocol-compatible services like Confluent Platform, Aiven, RedPanda, WarpStream, or AutoMQ to Express brokers on Amazon MSK. Built on the same technology that powers cross-region replication, Amazon MSK Replicator automates the migration process through managed replication of topic data and metadata, while providing bidirectional replication, consumer offset synchronization, and built-in monitoring.

Missing alt text value

Key Features

Managed Kafka Migration

Migrate topic data and metadata from other Kafka deployment including including those running on-premises, on AWS, or other cloud providers, as well as Kafka-protocol-compatible services like Confluent Platform, Aiven, RedPanda, WarpStream, or AutoMQ to MSK Express brokers without provisioning migration infrastructure such as MirrorMaker 2.

Zero Infrastructure Management

Focus on your migration strategy, not infrastructure. MSK Replicator handles all the underlying resources, eliminating weeks of setup and tuning required by traditional tools like MirrorMaker2.

Bidirectional Replication

Enable safe rollbacks and phased migrations with reliable bidirectional replication between your source cluster and MSK Express brokers, giving you confidence to migrate without disruption.

Consumer Offset Synchronization

Migrate producers and consumers at your own pace in any order without data loss. MSK Replicator automatically synchronizes consumer offsets across clusters, eliminating cross-team coordination bottlenecks.

Native CloudWatch Integration

Monitor your migration with CloudWatch metrics and logging. Track replication lag, throughput, and errors in real-time without custom instrumentation.

Enterprise-Grade Security

All data is encrypted in transit using TLS. Leverage SASL/SCRAM authentication with credentials securely managed through AWS Secrets Manager, ensuring your migration meets compliance requirements.

Benefits

    Reduce migration time from weeks to days. What traditionally takes weeks with MirrorMaker2 now takes just hours with MSK Replicator

    Move applications in any sequence without waiting for cross-team coordination. Consumer offset synchronization ensures no duplicate processing regardless of migration order.

    Confident rollbacks without operational disruption. Bidirectional replication acts as a safety net, allowing you to revert changes if needed without business impact.

    Combine the performance benefits of Express brokers with reduced migration overhead. Express brokers deliver 3x more throughput per broker while eliminating the operational costs of managing migration infrastructure.

Use Cases

    Migrate aging on-premises Kafka clusters to MSK Express brokers to eliminate hardware refresh cycles and reduce operational overhead while gaining 3x throughput improvements with Express Brokers.

    Move from self-managed Kafka clusters to fully managed MSK Express brokers, eliminating storage management, months of capacity planning and infrastructure tuning.

    Establish bidirectional replication between your existing cluster and MSK as a disaster recovery strategy before completing full migration.

How it Works

  1. Configure Source Cluster: Specify your source Kafka cluster details including bootstrap servers and SASL/SCRAM credentials stored in AWS Secrets Manager.
  2. Select Destination: Choose your MSK Express cluster as the migration destination with automatic network configuration.
  3. Define Topics and Consumer Groups: Select which topics to migrate, configure naming conventions for your destination cluster, and enable consumer group synchronization.
  4. Monitor Progress: Track migration status through CloudWatch dashboards with metrics and logging.
  5. Migrate Applications: Move your producers and consumers to MSK Express brokers in any order with enhanced automatic offset synchronization.
  6. Complete Migration: Decommission your source cluster once all applications are successfully migrated. 

Did you find what you were looking for today?

Let us know so we can improve the quality of the content on our pages