Most Kafka dashboards are built the way most dashboards are built: someone exports every JMX metric, arranges forty panels, and nobody remembers which ones mean "page me." The result is a cluster that is heavily observed and still surprises its operators. A smaller set of signals, tied to explicit objectives and to a runbook, does more work than the full firehose.
This note covers a practical set: what to alert on at the broker, what to measure from the client's point of view, and how to turn both into SLOs you can defend.
Start with what the business is promising
Broker metrics describe the cluster. Users experience streams. Before choosing alerts, write down two or three objectives in terms of data movement:
- Freshness: 99% of records on
orders.v1are consumed by the fulfillment group within 30 seconds of being produced. - Durability: no acknowledged write is lost across a single broker or single availability-zone failure.
- Availability: producers can write to critical topics successfully 99.9% of the time, measured over 30 days.
Everything below exists to tell you, early, that one of these is at risk. A metric that cannot be connected to an objective is a candidate for removal from the paging path (it may still belong on a diagnostic dashboard).
Broker signals worth paging on
Offline partitions (OfflinePartitionsCount on the controller). Any non-zero value means a partition has no leader and is unavailable for reads and writes. This is the clearest page in Kafka. Alert immediately.
Under-replicated and under-min-ISR partitions. UnderReplicatedPartitions above zero means a follower has fallen out of the in-sync set. Brief blips during a rolling restart are expected; a sustained value is not. The more serious variant is UnderMinIsrPartitionCount: with acks=all and min.insync.replicas=2, those partitions are rejecting producer writes. Alert on under-min-ISR as an availability page, and on sustained under-replication as a warning with a duration threshold (for example, 10 minutes) so upgrades do not wake anyone.
Active controller count. Across the cluster, exactly one controller should be active (ActiveControllerCount summed equals 1). Zero or more than one indicates a quorum problem. Under KRaft, also watch the metadata quorum: leader election churn and growing metadata log lag on controllers or observers.
Request latency, by request type. TotalTimeMs for Produce and Fetch requests, at the 99th percentile, is the broker's own account of how it is doing. The sub-components (RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs) tell you where the time goes. High queue time points at saturated network or I/O threads; high remote time on produce usually means followers are slow to acknowledge under acks=all.
Thread pool idleness. RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent are the closest thing Kafka has to a CPU-headroom gauge for the request path. A sustained idle ratio under roughly 30% means you are close to the point where latency climbs steeply. Treat it as a capacity signal, not a page.
ISR shrink and expand rates. Frequent shrinks followed by expands indicate flapping followers: usually a slow disk, a network problem, or replica.lag.time.max.ms set too tightly for the workload. A single shrink is noise; a pattern is a finding.
Disk. Free space on log directories, with a forecast rather than a threshold. A broker that fills its disk stops being a broker, and recovery is slow. Alert on projected time to full (say, under 72 hours at the current growth rate), and separately on a hard floor.
Client-side signals: where freshness is measured
Broker health does not prove consumers are keeping up. The signal that maps to a freshness objective is time lag: the age of the oldest unprocessed record in a partition, per consumer group.
Offset lag alone misleads, because the same number of records is trivial on a high-volume topic and severe on a quiet one. Two common approaches to time lag:
- Run an exporter that estimates lag in seconds from committed offsets and the produce timestamps of records at those offsets (several open-source lag exporters do this).
- Emit end-to-end latency from the application: stamp the produce time in the record, and have the consumer record
now - record timestampin a histogram after processing. This is the most honest measure, since it includes your processing time.
On the producer side, watch record-error-rate, record-retry-rate, and request latency average and max. A rising retry rate is often the first visible sign of a broker problem, before any broker-side alert has fired.
Also alert on group state and rebalance frequency. A group that keeps rebalancing can look healthy on lag averages while making no progress.
Turning signals into SLOs and alerts
For the freshness objective, the SLI is the fraction of records (or of time windows) where time lag is under 30 seconds. Alert on burn rate instead of on instantaneous lag: page when the error budget is being consumed fast enough to exhaust it within a day or so, and open a ticket when it is being consumed at a rate that exhausts it within the window. This cuts out the short spikes that resolve themselves and catches the slow degradation that threshold alerts miss.
A workable split:
- Page: offline partitions; under-min-ISR on critical topics; fast burn on a freshness or availability SLO; controller count wrong.
- Ticket: sustained under-replication; low thread-pool idle; disk forecast under 72 hours; slow burn on an SLO; ISR flapping.
- Dashboard only: per-topic byte rates, request rates, GC pauses, individual partition sizes.
Common mistakes
- Alerting on every JMX metric. The page rate rises, trust in alerts falls, and the real one gets muted.
- Averages for latency. Kafka latency is long-tailed. Use percentiles, and keep the 99th and 99.9th.
- Monitoring brokers but not the topology around them. Connect workers, Schema Registry, and MirrorMaker tasks fail independently. Check that each has a health signal and an owner.
- No test of the alert. Pause a follower, or stop a consumer in a staging environment, and confirm the right alert fires and the runbook is accurate.
- Thresholds copied from a blog post (including this one). Baseline your own cluster for a few weeks first; the numbers above are starting points.
A short checklist
- Two or three SLOs written in terms of freshness, durability, and availability
- Paging alerts limited to offline partitions, under-min-ISR, controller count, and SLO burn
- Time lag per consumer group, not just offset lag
- Disk alerts based on forecast as well as free space
- A runbook link on every page-level alert
- Alerts exercised in a non-production environment at least once
Monitoring is part of the design of a streaming platform, not decoration added afterward. If you would like a second set of eyes on what your cluster is and is not telling you, a Kafka Health Check covers monitoring alongside configuration, partitioning, and replication, with findings ranked by production risk.