How do I troubleshoot common issues when using my Amazon MSK cluster with SASL/SCRAM authentication?

Resolution

You're getting the ClusterAuthorizationException error in broker logs

You might see the following error message when you access your Amazon MSK cluster:

org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=11, connectionId=INTERNAL_IP-INTERNAL_IP-0, session=Session(User:ANONYMOUS,/INTERNAL_IP), listenerName=ListenerName(REPLICATION_SECURE), securityProtocol=SSL, buffer=null) is not authorized.

You get this error when both the following conditions are true:

• Your Amazon MSK cluster has Simple Authentication and Security Layer (SASL)/Salted Challenge Response Mechanism (SCRAM) authentication enabled.
• You've set resourceType=CLUSTER and operation=CLUSTER_ACTION in the access control lists (ACLs) for your cluster.

The Amazon MSK cluster doesn't support this setting because this setting prevents the internal Apache Kafka replication. With this setting, the identity of the brokers appear as ANONYMOUS for inter-broker communication. If you need your cluster to support these ACLs while using the SASL/SCRAM authentication, then you must grant the permissions for ALL operations to the ANONYMOUS user. This prevents the restriction of replication between the brokers. Use the following command to grant this permission to the ANONYMOUS user:

./kafka-acls.sh --authorizer-properties
zookeeper.connect=example-ZookeeperConnectString --add --allow-principal
User:ANONYMOUS --operation ALL --cluster

You changed the user name and/or password for your AWS Secrets Manager secret, but the old credentials are still active

An AWS Secrets Manager secret must be associated with a single user. If the user no longer needs access to the cluster, the secret must be dissociated, and the ACLs must be updated. When you update the user name on the secret, the old user name is not automatically dissociated. Therefore, it's a best practice to create a new secret for the new user. If you need to use the old secret, first dissociate the secret, and then update the credentials before associating the secret again. For more information, see Working with users.

Non-admin users are able to create new ACLs and modify the existing ACLs

By default, any user can create or modify ACLs. To prevent non-admin users from creating or updating ACLs using Apache ZooKeeper, be sure to restrict access to Apache ZooKeeper by placing the zooKeeper nodes in a separate security group. Also, be sure all the Apache Kafka commands and client connections are run through bootstrap brokers instead of Apache ZooKeeper. Note that all Apache Kafka operations can be performed using the bootstrap string.

Related information

Apache Kafka ACLs

Scram secrets