Most clusters we review were built inside a trusted network and never had authentication turned on. That is usually fine until an audit, a new tenant, or a move to a shared VPC makes it not fine — and then someone proposes enabling SASL and ACLs in a maintenance window. Done that way, it is the single most effective way to take down every producer and consumer in the estate simultaneously.
The work is not hard. The ordering is the whole job. Here is the sequence we use, and the details that decide whether it is uneventful.
Listeners are how you get to do this incrementally
A broker can expose several listeners at once, each with its own port, security protocol, and authentication mechanism. That is the mechanism that makes a phased rollout possible: you add a new secured listener beside the existing plaintext one, migrate clients across at their own pace, and remove the old listener when it is idle.
A cluster mid-migration typically advertises something like:
PLAINTEXT://:9092— the existing listener, still serving un-migrated clientsSASL_SSL://:9094— the new authenticated listenerCONTROLLER://:9093— controller quorum traffic, KRaft-internalBROKER://:9095— inter-broker traffic, on its own listener name
inter.broker.listener.name decides which of these the brokers use to talk to each other, and it is worth moving separately from client traffic. Replication breaking is a much worse day than one application failing to connect.
Two things bite here. First, advertised.listeners must resolve from the client's network, not from the broker's — brokers hand back the advertised address in metadata, so a client that reaches the bootstrap host fine can still fail on every subsequent connection if the advertised name is internal-only. Second, every listener needs an entry in listener.security.protocol.map; a missing mapping produces a broker that refuses to start with a message that does not immediately name the cause.
Pick the authentication mechanism deliberately
There are three realistic choices, and the trade-off is operational, not cryptographic.
mTLS authenticates the client by its certificate, so the principal is the certificate's distinguished name. Strong, and no secrets in application config beyond the keystore — but you now own certificate issuance, distribution, and rotation for every client. Teams with an existing internal PKI or a service mesh find this cheap. Teams without one discover they have signed up to build a CA practice, and the first expiry takes an outage to learn.
SASL/SCRAM (SCRAM-SHA-512) stores salted credentials in the cluster's own metadata and needs no external dependency. It is the pragmatic default for self-managed clusters: credentials are created with kafka-configs, rotation is a two-step add-then-remove, and clients need one JAAS line. Keep it over TLS (SASL_SSL) — SCRAM protects the password exchange, not the payload.
SASL/OAUTHBEARER delegates to your existing identity provider, which is the right answer when you already run one and want short-lived tokens and central revocation. It also means Kafka authentication now depends on the IdP being up, so cache and fail-over accordingly.
Avoid SASL/PLAIN with static broker-side credential files. It works, and it also means every credential change is a rolling restart.
Authorization: the KRaft-era authorizer
On ZooKeeper-based clusters, ACLs lived in ZooKeeper and the config was authorizer.class.name=kafka.security.authorizer.AclAuthorizer. Under KRaft that is gone: ACLs are stored in the cluster metadata log and the setting is authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer. If you migrated to KRaft and never revisited your security config, check this now — ACLs are one of the settings most likely to have been carried over wrong or quietly dropped during a migration.
Two configs govern the transition period:
allow.everyone.if.no.acl.found=truelets a resource with no ACLs at all stay open. This is how you enable the authorizer without denying anything on day one — but it is a temporary state, not a destination, and it must be flipped tofalsebefore you can claim the cluster is actually authorized.super.userslists principals that bypass ACL checks. Put the inter-broker principal and your operations tooling here, and nothing else. A super-user list containing application principals is an authorizer that isn't.
ACLs are deny-by-default once you turn off the open fallback, evaluated per operation on a resource, and explicit DENY beats ALLOW. Prefix-matched resource patterns (--resource-pattern-type prefixed) are what make this maintainable: grant a team payments. rather than enumerating forty topics and re-granting every time they add one.
The permission everyone forgets
Producers and consumers need more than Write and Read:
- Consumers need
Readon the group resource as well as the topic. Missing group ACLs is the most common first failure, and the error surfaces as an authorization exception on join, not on connect. - Transactional producers need
WriteandDescribeon the TransactionalId resource, plusIdempotentWriteon the cluster on older broker versions. - Kafka Connect needs access to its three internal topics and its own consumer group per connector; Kafka Streams needs to create and read internal changelog and repartition topics, which usually means a prefixed ACL on the application id.
- Clients calling
describeTopicsor using admin APIs needDescribe, and tools that auto-create topics needCreateon the cluster — which is a good reason to disable auto-creation and grant nothing.
The least-effort way to get this right is not to reason about it in advance. Enable the authorizer with the open fallback on, set the authorizer logger to log denials at INFO, and run for a week. The denial log is your ACL backlog, derived from real traffic rather than a guess about it.
An order of operations
- Add TLS on a new listener. Brokers serve both; nobody is required to move yet. Verify with
kafka-broker-api-versionsagainst the new port. - Move inter-broker traffic to its own secured listener, one rolling restart, and confirm under-replicated partitions return to zero before continuing.
- Turn on the authorizer with
allow.everyone.if.no.acl.found=trueand a minimalsuper.users. Nothing is denied; denials are now observable. - Migrate clients to the authenticated listener in waves, starting with a low-stakes consumer. Each team changes bootstrap servers plus a handful of security properties.
- Build ACLs from the denial log, prefix-matched by team or domain, applied as code in your config repository rather than by hand.
- Flip the fallback to
falseonce the denial log has been quiet for a full business cycle — including the monthly batch job nobody remembered. - Remove the plaintext listener when broker metrics show zero connections on it.
kafka-network-Processorconnection counts per listener, or an audit of the broker's connection metrics, will tell you.
Steps 3 and 6 are the ones teams collapse into one, and that is where the outage lives.
What to watch while you do it
Authentication failures and authorization denials both have broker-side metrics (failed-authentication-rate, and the authorizer's denial logging). Alert on them for the duration of the migration, then keep the alert — a sudden spike in denials afterwards is either a misconfigured deploy or something you want to know about for other reasons.
Also watch the clock. Certificate and token expiry are the failure mode that arrives months after the project is declared finished, at an hour nobody chose. Whatever mechanism you pick, the rotation procedure should be written down and rehearsed once before the first real expiry, not discovered during it.
None of this is exotic. It is a listener model, a credential decision, and a deliberately slow sequence — which is exactly why it usually goes fine when it is planned and badly when it is squeezed into a window.
If you are securing a cluster that is already carrying production traffic and would like a second pair of eyes on the rollout order, get in touch — it is a common piece of our health check and operations work.