"How many partitions?" is the question every Kafka design review reaches within the first hour, and the honest answer is a method, not a number. Partition count sets your consumer parallelism ceiling, shapes broker load, and is awkward to change later on a keyed topic — so it deserves twenty minutes of arithmetic instead of a copied default.
What a partition actually is
A partition is three things at once: the unit of parallelism (one partition is consumed by at most one consumer in a group), the unit of ordering (order is guaranteed within a partition only), and a unit of broker load (each partition is an open set of segment files, replicated replication.factor times, participating in leader elections). Sizing is balancing those three against each other.
The method
1. Measure per-partition throughput — yours, not a benchmark's. Write throughput per partition depends on record size, batching, compression, acks and the broker's disks. Run your real producer settings against a one-partition topic in your real environment and record the MB/s and records/s where latency stays acceptable. Do the same for a consumer running your real processing. These two numbers — call them P (produce) and C (consume) — are the foundation; teams that skip this step are sizing on folklore.
2. Take peak, not average, demand. Size against the worst sustained hour you must survive — the Monday-morning surge, the month-end batch, the replay after an outage. Call it T.
3. Compute both floors. You need at least T / P partitions to absorb the write rate and at least T / C to let consumers keep up. The consumer floor usually dominates, because processing is almost always slower than producing.
4. Add replay headroom. Steady-state parity means a consumer that was down for four hours takes forever to catch up. Decide your recovery objective — "drain four hours of backlog in one hour" means consuming at 5× produce rate — and scale the consumer floor accordingly. This step is the most commonly skipped and the most commonly regretted.
5. Round up to a convenient multiple. Pick a count divisible by your typical consumer instance counts (12, 24, 30, 48…) so partitions spread evenly and one slow instance doesn't carry a remainder.
Check the key distribution
Partition count assumes keys spread evenly; a hot key breaks the math. If one customer id produces 40% of the traffic, that partition is your real throughput ceiling no matter how many others sit idle. Check the actual key histogram before committing to a keyed design, and if skew is structural, consider a composite key (key + bucket) with a merge step downstream — a trade against per-key total ordering that you should make knowingly.
Why not just go big?
Over-partitioning is not free. Every partition adds open file handles and memory on the broker, lengthens leader-election and controlled-shutdown time, and adds to producer batching overhead — many nearly-empty batches instead of a few full ones, which shows up as latency. Thousand-partition topics "for future growth" are how clusters end up with six-figure partition counts that turn every broker restart into an event. Modern brokers handle large counts far better than the ZooKeeper era did, but the costs are still real; the answer to future growth is measured headroom, not a zero on the end.
Growing later
You can add partitions to a topic, but on a keyed topic the key-to-partition mapping changes: records for a key start landing on a new partition while old records stay put, so per-key ordering is briefly — sometimes not so briefly — violated, and any Streams state keyed by partition must be rebuilt. The workable patterns are: size with deliberate headroom up front (2–3× the computed floor is sane for a topic expected to grow), or plan growth as a new topic with a re-key job and a coordinated consumer cutover.
The worksheet
For each topic: measured P and C, peak T, replay multiplier, key-skew check, resulting count rounded to a consumer-friendly multiple, and one sentence on how it grows. Ten lines that answer the design review's longest argument — and, two years later, tell the team that inherits the topic why the number is 24.