Serverless and auto-scaling Kafka Connect workloads with Amazon MSK

With Amazon MSK Connect, a feature of Amazon MSK, you can run fully managed Apache Kafka Connect workloads on AWS. This feature makes it easy to deploy, monitor, and automatically scale connectors that move data between Apache Kafka clusters and external systems such as databases, file systems, and search indices. MSK Connect is fully compatible with Kafka Connect, enabling you to lift and shift your Kafka Connect applications with zero code changes. With MSK Connect, you only pay for connectors you are running, without the need for cluster infrastructure.

Benefits

Fully managed

With MSK Connect, there is no need to provision infrastructure or Apache Kafka Connect clusters. The lifecycle of clusters, workers, and connectors is all handled by the service. Monitoring, automatic restarts of failed connectors, and patching of underlying infrastructure is all transparent to the end user, so you can focus on building your streaming data flows to and from Apache Kafka clusters.

Scales automatically

Data flows frequently change, with data volumes going up and down from different sources. MSK Connect provides a serverless experience and scales the number of workers up and down, so you don’t have to provision servers or clusters and pay only for what you need to move your streaming data to and from your Apache Kafka cluster. 

Runs any connector

MSK Connect is fully compatible with Apache Kafka Connect. This means you can run any connector compatible with Apache Kafka Connect 2.7.1 and above, whether they were developed by one of our partners, by the broad open-source community, or within your own organization. MSK connect allows you to upload any third-party connector plugins and run them on AWS. . 

FAQs for Kafka Connect

Q: What is Apache Kafka Connect?
Kafka Connect, an open source component of Apache Kafka, is a framework for connecting Apache Kafka with external systems such as databases, key-value stores, search indexes, and file systems.

Q: Where can I get connectors?
MSK Connect runs any connector that implements the Kafka Connect interfaces. There are many sources of connectors, including from our partners, open source projects like Debezium, and from commercial connector vendors like Confluent and lenses.io.

Q: Can I use MSK Connect to move data between different Apache Kafka clusters?
Yes, you can use connectors that both read from and write to Kafka. The popular Apache Kafka replication tool, MirrorMaker 2, uses Kafka Connect to replicate data between topics on different Apache Kafka clusters. This provides an easy path to synchronize or migrate data between Kafka clusters.

Q: Does MSK Connect support non-MSK Apache Kafka clusters?
MSK Connect works with Amazon MSK clusters, other Apache Kafka and compatible clusters including self-managed clusters in EC2 or non-AWS environments, subject to Amazon VPC connectivity.

Learn more about Amazon MSK features

Visit the features page
Ready to build?
Get started with Amazon MSK
Have more questions?
Contact us