Open Source

Run with native Apache Kafka

Amazon MSK supports deploys native versions of Apache Kafka so applications and tools built for Apache Kafka just work with Amazon MSK out of the box, with no application code changes.

Streamlined version availability

Amazon MSK typically makes newer versions of Apache Kafka available within seven days of public availability. 

Seamless version upgrades

You can upgrade Apache Kafka versions on Amazon MSK clusters in just a few clicks, allowing you to decide when to take advantage of features and bug fixes present in new Apache Kafka versions. Amazon MSK automates the deployment of version upgrades on running clusters to maintain client I/O availability for customers following best practices.

No servers to manage

Fully managed

With a few clicks in the console, you can create a fully managed Apache Kafka cluster that follows Apache Kafka’s deployment best practices, or you can create your own cluster using your own custom configuration. Once you create your desired configuration, Amazon MSK automatically provisions, configures, and manages the operations of your Apache Kafka cluster and Apache ZooKeeper nodes.

Apache ZooKeeper included

Apache ZooKeeper is required to run Apache Kafka, coordinate cluster tasks, and maintain state for resources interacting with the cluster. Amazon MSK manages the Apache ZooKeeper nodes for you. Each Amazon MSK cluster includes the appropriate number of Apache ZooKeeper nodes for your Apache Kafka cluster at no additional cost.

Highly available

High availability is default

All clusters are provisioned across multiple availability zones (three AZ is default), are supported by Amazon MSK’s service-level agreement, and are supported by automated systems that detect and respond to issues within cluster infrastructure and Apache Kafka software. If a component fails, Amazon MSK automatically replaces it without downtime to your applications. Amazon MSK manages the availability of your Apache ZooKeeper nodes so you don’t need to start, stop, or directly access the nodes yourself. Amazon MSK also automatically deploys software patches as needed to keep your cluster up-to-date and running smoothly.

Data replication

Amazon MSK uses multi-AZ replication for high-availability. Data replication is included at no additional cost.

Highly secure

Private connectivity

Your Apache Kafka clusters run in an Amazon VPC managed by Amazon MSK. Your clusters are available to your own Amazon VPCs, subnets, and security groups based on the configuration you specify. You have complete control of your network configuration and IP addresses from your VPCs are attached to your Amazon MSK resources through elastic network interfaces (ENIs).

Granular access control

IAM Access Control is a no-cost security option that simplifies cluster authentication and Apache Kafka API authorization using IAM roles or user policies to control access. By using IAM Access Control, you no longer need to build and run one-off access management systems to control client authentication and authorization for Apache Kafka and your clusters are secured using least-privileged permissions by default. You also can use SASL/SCRAM or mutual TLS authentication with Apache Kafka access control lists (ACLs) to control client access. 

Encryption

Amazon MSK encrypts your data at rest without special configuration or third-party tools. All data can be encrypted at rest using AWS Key Management Service (KMS) Customer Master Key (CMK) by default, or your own CMK. Amazon MSK also encrypts data in-transit via TLS between brokers and between clients and brokers on your cluster.

Lowest cost

Amazon MSK lets you get started for less than $2.50 per day. Customers typically pay between $0.05 and $0.07 per GB ingested, all-in, which can be 1/13th the cost of other managed providers. Visit the Amazon MSK Pricing page for current prices and the Amazon MSK Best Practices page to learn how right-size your Amazon MSK clusters.

Deeply integrated

No other provider offers the breadth and depth of AWS integrations in Amazon MSK. These integrations include:

Scalable

Broker scaling

You can scale your Amazon MSK clusters by changing the size or family of your Apache Kafka brokers in minutes with no downtime. Changing the size or family of your brokers is a popular way to scale Amazon MSK clusters because it gives you the flexibility to adjust your MSK cluster’s compute capacity for changes in your workloads. This method can be preferred because it does not require partition reassignment which can impact Apache Kafka availability.

Alternatively you can expand the size of your cluster by up to 100s brokers per cluster using the console or command line interface (CLI). Submit a limit increase request if you need more than 15 brokers per cluster or more than 30 brokers per account.

Automatic partition management

Amazon MSK integrates with Cruise Control, a popular open source tool for Apache Kafka that automatically manages partition assignment on your behalf.

Automatic storage scaling

You can seamlessly scale up the amount of storage provisioned per broker to match changes in storage requirements using the AWS management console or AWS CLI or you can create an auto scaling policy to automatically expand your storage to meet your streaming requirements.

Configurable

Amazon MSK deploys a best practice cluster configuration for Apache Kafka by default, and gives customers the ability to tune more than 30 different cluster configurations while supporting all dynamic and topic-level configurations. For more information, see Custom MSK Configurations in the documentation.

Visible

CloudWatch metrics by default

You can visualize and monitor important cluster, broker, topic, consumer, and partition-level metrics using Amazon CloudWatch.

Export JMX and Node metrics to a Prometheus server

Open Monitoring with Prometheus enables you to monitor Amazon MSK using solutions like Datadog, Lenses, New Relic, Sumo logic, or a Prometheus server, and easily migrate your existing monitoring dashboards to Amazon MSK. For more information, see Open Monitoring with Prometheus in the documentation. 

Get started with Amazon MSK

Calculate your costs
Calculate your costs

Visit the Amazon MSK pricing page.

Review the getting-started guide
Review the getting-started guide

Learn how to set up your Apache Kafka cluster on Amazon MSK in this step-by-step guide.

Run your Apache Kafka cluster
Run your Apache Kafka cluster

Start running your Apache Kafka cluster on Amazon MSK. Log in to the Amazon MSK console.