FAQs
- General
3
- Resources
4
- Amazon MSK Provisioned
1
- Express brokers
4
- Amazon MSK Serverless
2
- Migrating to Amazon MSK
5
- Networking
3
- Encryption
2
- Access Management
1
- Monitoring, metrics, logging, and tagging
8
- Metadata Management
3
- Integrations
1
- Replication
9
- Pricing and availability
3
- Service Level Agreement
1
General
Open allApache Kafka is an open source, high-performance, fault-tolerant, and scalable platform for building real-time streaming data pipelines and applications. Apache Kafka is a streaming data store that decouples applications producing streaming data (producers) into its data store from applications consuming streaming data (consumers) from its data store. Organizations use Apache Kafka as a data source for applications that continually analyze and react to streaming data.
Apache Kafka supports real-time applications that transform, deliver, and react to streaming data, and can be used to build real-time streaming data pipelines that reliably send data between multiple systems or applications.
MSK Express brokers support data delivery to Amazon S3 general purpose buckets in source data format and streaming tables for Apache Iceberg on Amazon S3 Tables. This capability provides the most easiest, reliable and price performant way to deliver your Apache Kafka data to Amazon S3. If you prefer to use Kafka Connect, you can use MSK Connect to configure data delivery to Amazon S3 and Amazon S3 Tables.
Resources
Open allNo, each broker you provision includes boot volume storage managed by the Amazon MSK service.
Some resources, such as elastic network interfaces (ENIs), will show up in your Amazon EC2 account. Other Amazon MSK resources will not show up in your Amazon EC2 account because they are managed by the Amazon MSK service.
Amazon MSK uses Apache Kafka’s leader-follower replication to replicate data between brokers. Amazon MSK makes it easy to deploy clusters with Multi-AZ replication. With Standard brokers, you have the option to use a custom replication strategy by topic. Express brokers guarantee higher availability by always replicating your data across three Availability Zones. Leader and follower brokers will be deployed and isolated using the broker type and replication strategy specified. For example, if you select Standard brokers with three Availability Zone broker replication strategy and with one broker per Availability Zone cluster, Amazon MSK will create a cluster of three brokers (one broker in three Availability Zones in a Region), and by default (unless you choose to override the topic replication factor), the topic replication factor will also be three. To learn more about what happens during client failover, see our client failover documentation.
Yes, Amazon MSK allows you to create custom configurations and apply them to new and existing clusters. Express brokers protect more of the configurations from suboptimal values that may affect availability and durability. Express brokers also offer a simpler experience by abstracting configurations that have to do with storage since Amazon MSK fully manages the storage layer. For more information on custom configurations, see the configuration documentation.
Amazon MSK Provisioned
Open allYes. You can configure custom domain names on all MSK Provisioned clusters (Standard and Express brokers) on either ZooKeeper or Kraft metadata management mode, at no additional cost. Add the custom.advertised.listeners property to your MSK Configuration with your custom domain name, and Amazon MSK automatically applies it to every broker in the cluster, including new brokers added during scaling. For setup guidance, see Configure custom domain names.
Express brokers
Open allExpress brokers for MSK Provisioned make Apache Kafka simpler to manage, more cost-effective to run at scale, and more elastic with the low latency you expect. Brokers include pay-as-you-go storage that scales automatically and requires no sizing, provisioning, or proactive monitoring. Depending on the instance size selected, each broker node can provide up to 3x more throughput per broker, scale up to 20x faster, and recover 90% quicker compared to standard Apache Kafka brokers. Express brokers support Intelligent rebalancing at no additional cost, delivering 180x faster performance than standard Apache Kafka brokers. Additionally, Express brokers come pre-configured with Amazon MSK’s best practice defaults and enforce client throughput quotas to minimize resource contention between clients and Kafka’s background operations.
- No storage management: Express brokers eliminate the need to provision or manage any storage resources. You get elastic, virtually unlimited, pay-as-you-go, and fully managed storage. For high throughput use cases, you do not need to reason about the interactions between compute instances and storage volumes and the associated throughput bottlenecks. These capabilities simplify cluster management and eliminate storage management operational overhead.
- Intelligent rebalancing: All new Express brokers are enabled with Intelligent rebalancing by default and at no additional cost. Intelligent rebalancing makes it effortless for customers to execute automatic partition balancing operations when scaling their Kafka clusters up or down. Intelligent Rebalancing maximizes the capacity utilization of MSK Express-based clusters by optimally rebalancing Kafka resources on them for better performance, eliminating the need for customers to manage partitions themselves or via third-party tools. Intelligent Rebalancing performs these operations up to 180 times faster compared to Standard brokers.
- Faster scaling: Express brokers allow you to scale your cluster and move partitions faster than on Standard brokers. This capability is crucial when you need to scale out your cluster to handle upcoming load spikes or scale in your cluster to reduce cost. See the sections on expanding your cluster, removing brokers, reassigning partitions, and setting up LinkedIn’s Cruise Control for rebalancing for more details on scaling your cluster.
- Higher throughput: Express brokers offer up to 3x more throughput per broker than Standard brokers. For example, you can safely write data at up to 500 MBps with each m7g.16xlarge sized Express broker compared to 153.8 MBps on the equivalent Standard broker (both numbers assume sufficient bandwidth allocation towards background operations, such as replication and rebalancing).
- Configured for high resilience: Express brokers automatically offer various best practices pre-configured to improve your cluster’s resilience. These include guardrails on critical Apache Kafka configurations, throughput quotas, and capacity reservation for background operations and unplanned repairs. These capabilities make it safer and easier to run large scale Apache Kafka applications. See the sections on Express broker configurations and Amazon MSK Express broker quota for more details.
- No Maintenance windows: There are no maintenance windows for Express brokers. Amazon MSK automatically updates your cluster hardware on an ongoing basis. See Amazon MSK Express brokers for more details.
- Data delivery to Amazon S3 and Apache Iceberg tables: Express brokers supports serverless data delivery to streaming tables for Apache Iceberg on Amazon S3 Tables. This capability can reduce the cost of ingesting and delivering Apache Kafka data to Apache Iceberg with Amazon S3 Tables by up to 60% and reduces downstream query costs by up to 30% versus self-managed Apache Kafka alternatives. In addition to S3 Tables, you can deliver Kafka data to general purpose Amazon S3 buckets in source data format.
Express brokers provide more throughput per broker, so you can create clusters with fewer brokers for the same workload. Additionally, once your cluster is up and running, you can monitor the use of your cluster resources and right-size capacity faster than with Standard brokers. You can, therefore, provision resources that are fit for the capacity you need and scale faster to meet any changes in demand.
Express brokers come preconfigured with Amazon MSK best practice defaults that optimize for availability and durability. You may customize some of these configurations to further fine-tune the performance of your clusters. Read more about Express broker configurations in the Amazon MSK Developer Guide.
Amazon MSK Serverless
Open allMSK Serverless is a cluster type for Amazon MSK that makes it easy for you to run Apache Kafka clusters without having to manage compute and storage capacity. With MSK Serverless, you can run your applications without having to provision, configure, or optimize clusters, and you pay for the data volume you stream and retain.
MSK Serverless currently supports IAM (Identity Access Management) for client authentication and authorization. Your clients can assume an IAM role for authentication, and you can enforce access control using an associated IAM policy.
Migrating to Amazon MSK
Open allNetwork setup depends on your source environment. For on-premises deployments, connect to your Kafka deployment with AWS Direct Connect, AWS VPN, or the internet. For AWS self-managed clusters, use VPC peering or Transit Gateway. For other cloud providers, establish appropriate network connectivity. You'll also need to configure security groups to allow traffic between source and destination clusters.
Update your client applications' broker endpoints to point to the MSK cluster. You can migrate producers and consumers in any order, as MSK Replicator automatically synchronizes consumer offsets between clusters.
Yes, you can create a second replicator from the target cluster back to the source cluster for safe rollback. MSK Replicator includes automatic loop prevention for identical topics to avoid circular replication.
MSK Replicator supports all core features including automatic topic creation on the destination cluster, topic configuration replication, ACL replication (except Write ACLs), enhanced consumer offset synchronization for bidirectional replication, and flexible topic selection for all topics or specific patterns.
No, stateful applications like Apache Flink and Kafka Streams need to rebuild their state or implement custom checkpoint restoration strategies after migration. MSK Replicator does not preserve internal state for these applications.
Networking
Open allYes, Amazon MSK always runs within an Amazon VPC managed by the Amazon MSK service. Amazon MSK resources will be available to your own Amazon VPC, subnet, and security group that you select when the cluster is set up. IP addresses from your VPC are attached to your Amazon MSK resources through ENIs, and all network traffic stays within the AWS network and is not accessible to the internet by default.
The brokers in your cluster will be made accessible to clients in your VPC through ENIs appearing in your account. The security groups on the ENIs will dictate the source and type of ingress and egress traffic allowed on your brokers.
Yes, Amazon MSK offers an option to securely connect to the brokers of Amazon MSK clusters running Apache Kafka 2.6.0 or later versions over the internet. By enabling public access, authorized clients external to a private Amazon VPC can stream encrypted data in and out of specific Amazon MSK clusters. You can enable public access for MSK clusters after a cluster has been created at no additional cost, but standard AWS data transfer costs for cluster ingress and egress apply. To learn more about turning on public access, see the public access documentation.
Encryption
Open allYes, by default, in-transit encryption is set to TLS only for clusters created from the AWS CLI or AWS Management Console. Additional configuration is required for clients to communicate with clusters using TLS encryption. For provisioned clusters, you can change the default encryption setting by selecting the TLS/plaintext or plaintext settings. Read more about Amazon MSK encryption.
Yes, MSK clusters support TLS in-transit encryption between Kafka brokers and metadata nodes.
Access Management
Open allMonitoring, metrics, logging, and tagging
Open allYou can monitor the performance of your clusters using the Amazon MSK console, the Amazon CloudWatch console, or through JMX and host metrics using Open Monitoring with Prometheus, an open source monitoring solution.
The cost of monitoring your cluster using CloudWatch is dependent on the monitoring level and the size of your Apache Kafka cluster. CloudWatch charges per metric per month and includes an AWS Free Tier. See Amazon CloudWatch pricing for more information. For details on the number of metrics exposed for each monitoring level, see the Amazon MSK monitoring documentation.
You can use any client-side monitoring supported by the Apache Kafka version you are using.
Yes, you can tag Amazon MSK clusters from the AWS CLI or AWS Management Console.
Topic-level consumer lag metrics are available as part of the default set of metrics that Amazon MSK publishes to CloudWatch for all clusters. No additional setup is required to get these metrics.
Topic-level metrics are included in the default set of Amazon MSK metrics, which are free of charge. Partition-level metrics are charged according to Amazon CloudWatch pricing.
Amazon MSK provides INFO-level logs for all brokers within a provisioned cluster.
Yes, if you use IAM access control, the use of Apache Kafka resource APIs is logged to AWS CloudTrail.
Metadata Management
Open allApache KRaft is the consensus protocol that shifts metadata management in Kafka clusters from external Apache ZooKeeper nodes to a group of controllers within Kafka. This change allows metadata to be stored and replicated as topics within Kafka brokers, resulting in faster propagation of metadata. To learn more, refer to our Apache KRaft documentation.
There are no API changes required to use KRaft mode on Amazon MSK. However, if your clients still use the --zookeeper connection string today, you should update your clients to use the --bootstrap-server connection string to connect to your cluster and perform admin actions. The --zookeeper flag is deprecated in Apache Kafka version 2.5 and is removed starting with Kafka 3.0. Therefore, we recommend you use recent Apache Kafka client versions and the --bootstrap-server connection string.
You should check that any tools you use are capable of using Kafka Admin APIs without ZooKeeper connections. See our updated documentation on using Cruise Control for KRaft mode clusters. Cruise Control has also published steps to follow to run Kafka without ZooKeeper connection.
Integrations
Open all- Amazon S3 using data delivery to Amazon S3 from Amazon MSK Express brokers in a no-code manner
- Amazon VPC for network isolation and security
- Amazon CloudWatch for metrics
- AWS KMS for storage volume encryption
- IAM for authentication and authorization of Apache Kafka and service APIs
- AWS Lambda for Amazon MSK event sourcing
- AWS IoT Core for IoT event sourcing
- AWS Glue Schema Registry for controlling the evolution of schemas used by Apache Kafka applications
- AWS CloudTrail for AWS API logs
- AWS Certificate Manager for private CAs used for client TLS authentication
- AWS CloudFormation for describing and provisioning Amazon MSK clusters using code
- Amazon Managed Service for Apache Flink for fully managed Apache Flink applications that process streaming data
- Amazon Managed Service for Apache Flink Studio for interactively streaming SQL on Apache Kafka
- AWS Secrets Manager for client credentials used for SASL/SCRAM authentication
- Amazon S3 using MSK Connect for delivering data to Amazon S3 from MSK serverless
- Amazon VPC for network isolation and security
- Amazon CloudWatch for metrics
- IAM for authentication and authorization of Apache Kafka and service APIs
- AWS Glue Schema Registry for controlling the evolution of schemas used by Apache Kafka applications
- AWS CloudTrail for AWS API logs
- AWS PrivateLink for private connectivity
Amazon MSK integrates with the following AWS services:
Amazon MSK Serverless integrates with the following AWS services:
Replication
Open allTo set up replication between a pair of source and target MSK clusters, you need to create a Replicator in the destination Region. To create a Replicator, you specify details that include the Amazon Resource Name (ARN) of the source and target MSK clusters and an IAM role that MSK Replicator can use to access the clusters. You will need to create the target MSK cluster if it does not already exist.
MSK Replicator supports replication across MSK clusters and between existing Apache Kafka deployments and Amazon MSK Express brokers. MSK Replicator simplifies migration from self-managed Kafka and other Kafka cloud environments to MSK Express Brokers through managed replication of topic data and metadata.
Yes, you can specify which topics you want to replicate using allow and deny lists while creating the Replicator.
No, MSK Replicator only supports replication across MSK clusters in the same AWS account.
You can use CloudWatch in the destination Region to view metrics for ReplicationLatency, MessageLag, and ReplicatorThroughput at a topic and aggregate level for each Replicator at no additional charge. Metrics are visible under ReplicatorName in the AWS/Kafka namespace. You can also see the ReplicatorFailure, AuthError, and ThrottleTime metrics to check if your Replicator is running into any issues.
MSK Replicator replicates data asynchronously. Replication latency varies based on many factors, including the network distance between the Regions of your MSK clusters, your source and target clusters’ throughput capacity, and the number of partitions on your source and target clusters.
Yes. By default, when you create a new Replicator, it starts replicating data from the tip of the stream (latest offset) on the source cluster. Alternatively, if you want to replicate existing data, you can configure a new Replicator to start replicating data from the earliest offset in the source cluster topic partitions.
Yes, you can specify your choice of compression codec while creating the Replicator amongst None, GZIP, Snappy, LZ4, and ZSTD.
Yes, you can specify your choice of compression codec while creating the Replicator amongst None, GZIP, Snappy, LZ4, and ZSTD.
Pricing and availability
Open allPricing depends on the resources you create. You can learn more by visiting Amazon MSK pricing.
No, in-cluster data transfer is included with the service at no additional charge.
No, Not at this time.
Service Level Agreement
Open allOur Amazon MSK SLA guarantees a Monthly Uptime Percentage of at least 99.9% for Amazon MSK (including MSK Serverless and MSK Connect).