Every streaming outage post-mortem has a cousin that starts with "we added a field." Schema changes are routine; broken consumers at 2 a.m. are optional. The difference is a small set of rules about compatibility, applied consistently and enforced by machines rather than code review.
Compatibility modes decide your deploy order
Schema Registry checks each new schema version against previous ones under the subject's compatibility mode, and each mode implies a deployment order you must actually follow:
- BACKWARD (the default): new readers can read old data. Safe change set: add fields with defaults, remove fields. Deploy order: consumers first, then producers. This fits event streams, where a consumer must handle records already in the topic.
- FORWARD: old readers can read new data. Safe changes: add fields, remove fields that had defaults. Deploy order: producers first.
- FULL: both directions; changes limited to optional-with-default fields; deploy order stops mattering.
The trap is the un-suffixed modes only check against the latest version. With retention measured in weeks — or compacted topics, where records are effectively forever — your consumers meet data written under schemas many versions old. Use the transitive variants (BACKWARD_TRANSITIVE, FULL_TRANSITIVE) so compatibility is checked against every registered version, not just the last one.
The rules that keep you compatible
Every new field gets a default. In Avro, a field with a default can be added under BACKWARD compatibility because old data simply fills in the default on read. A new field without one is an immediate compatibility break. In Protobuf all fields are effectively optional with zero-values, which makes the registry happy — the discipline moves to your application code, which must treat "zero" and "absent" correctly.
Never rename; alias or add-and-migrate. A rename is a delete plus an add, and old data has no value for the new name. Avro has aliases for exactly this. In Protobuf, field numbers are the identity — renaming is safe on the wire but confuses JSON representations and generated code consumers; renumbering is never safe.
Reserve what you remove. When a Protobuf field is deleted, mark its number and name reserved so a future edit cannot reuse it and silently misinterpret old bytes. Avro needs the same discipline by convention: keep a graveyard comment of retired field names.
Treat enums as append-only, with an escape hatch. An old consumer meeting an unknown enum symbol is a runtime failure in Avro unless the reader schema declares a default symbol — so give every evolving enum an UNKNOWN default from day one, and add new symbols at the end.
Model optionality honestly. ["null","string"] with a null default is the Avro idiom for "may be absent" — use it for anything not guaranteed forever, because promoting a required field to optional later is easy, and the reverse is a break.
Compacted topics change the arithmetic
On a compacted topic the oldest surviving record for a key can be years old, so "how far back must my consumer read?" has no bounded answer. Two consequences: transitive compatibility is mandatory, not advisory; and any migration that genuinely cannot preserve compatibility — a semantic change to an existing field's meaning, not just its shape — needs a new topic and a re-key job rather than a new schema version. Version numbers cannot fix a field whose meaning changed while its type stayed the same; that is the one break the registry cannot see, and it is worth a written rule that meaning changes get new field names.
Enforce it in CI, not in review
Compatibility rules only work when they are impossible to skip:
- Keep schemas in the repository next to the code that owns them; the owning service's pipeline is the only thing allowed to register new versions.
- Run a compatibility check against the live registry as a CI gate (the registry has a check endpoint and maven/gradle plugins), so an incompatible change fails the pull request rather than the deploy.
- Pin subject naming strategy deliberately.
TopicNameStrategy(one value type per topic) is the right default; the record-name strategies exist for multi-type topics and change what "compatible" is checked against — choose once, document it.
Decide who owns the contract
Behind all the mechanics is one organizational rule: a topic's schema is a published contract owned by the producing team, with consumers as customers. Changes flow through the compatibility gate; "we'll just tell the consumers in Slack" is not a compatibility mode. Teams that internalize this stop having schema incidents almost entirely — the registry turns a coordination problem into a build failure, which is exactly where you want your 2 a.m. problems to live: at 2 p.m., in CI.