Event schema evolution: changing OrderPlaced without breaking four consumers Intermediate

The problem

An online shop publishes OrderPlaced to the Kafka topic orders.order-placed. Four services consume it: billing, inventory, email and analytics. Each belongs to a different team and deploys on its own schedule.

The checkout team needs two changes: amount must grow from int to long (large orders overflow 2,147,483,647), and a new currency field. Retention is 7 days, so the topic holds v1 records for a week after the change.

There is no moment when all five services run the same version. The question: which changes can ship, in what deploy order, and what to do with a change that breaks someone?

Which approach? (summary first)

Situation	Choose	Condition
Consumers may rewind and replay old records	BACKWARD (Schema Registry default), deploy consumers first	Every consumer upgraded before the producer writes the new schema
Producer must ship before slow consumers	FORWARD, deploy producer first	Old readers must tolerate the new shape
Teams deploy in any order	FULL / FULL_TRANSITIVE	Only add or remove fields that have defaults
Change no mode allows (rename, type narrowing, new meaning)	Expand–contract: new field or new event type, then retire the old	Contract tests confirm every consumer moved

Core concepts

Two schemas at every read

Avro and Protobuf decode with two schemas: the writer schema the producer used, and the reader schema compiled into the consumer. Compatibility is a statement about that pair.

Mode	Guarantee	Deploy first	Allowed in Avro
Backward	New reader reads data written with the old schema	Consumers	Add fields with defaults, remove fields
Forward	Old reader reads data written with the new schema	Producer	Add fields, remove fields that have defaults
Full	Both directions	Any order	Add or remove fields with defaults only

Schema Registry modes

Confluent Schema Registry checks a new version before accepting it. Modes are BACKWARD (default), BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE and NONE (checks disabled).

• Non-transitive modes check only against the latest version; _TRANSITIVE checks against all previous versions.
• Confluent picks BACKWARD as default so consumers can rewind to the start of the topic.
• For Protobuf, Confluent recommends BACKWARD_TRANSITIVE, because adding new message types is not forward compatible.

Why transitive matters: v1→v2 and v2→v3 can each pass, while v3 still cannot read a v1 record sitting in the topic.

How it works

One record’s path from checkout to billing, with the registry’s default subject naming (<topic>-value):

The record carries only a schema ID; the reader resolves it against its own schema

Enlarge diagram

checkout → Schema Registryregister v2

◀ Prev▶ PlayNext ▶↺ RestartStep 1/5

The serializer registers schema v2 under subject orders.order-placed-value (or looks up its ID).

1. The serializer registers v2 (auto-registration is on by default) and gets ID 42.
2. The registry rejects v2 if it breaks the configured mode; checkout’s deploy fails here, not in production.
3. The record holds 0, then 42 in 4 bytes, then Avro binary without field names.
4. billing fetches schema 42 once and caches it.
5. Resolution follows the Avro spec: fields match by name, writer fields unknown to the reader are skipped, missing reader fields take the reader’s default or fail.

Avro rules that decide every change

Change	New reader, old data	Old reader, new data	Verdict
Add currency with "default": "VND"	Default filled	Field skipped	Full
Add currency without default	Error	Field skipped	Forward
Remove a field that has a default	Field skipped	Default filled	Full
Rename amount → totalAmount, no alias	Error	Error	Breaking
amount: int → long	Promoted	Error	Backward
amount: long → int	Error	Promoted	Forward
Add enum symbol REFUNDED, enum has no default	Fine	Error on REFUNDED	Backward

The Avro promotions are: int → long/float/double, long → float/double, float → double, string ↔ bytes. A rename is handled with aliases on the reader field: the reader treats the writer’s amount as totalAmount.

Deploy order is part of the change

Back to amount: int → long. The registry accepts it under BACKWARD, and the team forgets one consumer:

BACKWARD change with one consumer not upgraded: the registry passes, email breaks

◀ Prev▶ PlayNext ▶↺ RestartStep 1/5

The registry accepts v2: a long reader can read old int data, so the change is BACKWARD.

• Step 0–1: the registry and billing are both fine; v1 data is still readable.
• Step 2–3: once checkout writes long, the email reader (int) has no legal resolution.
• Step 4: email is stuck on the offset; see retry topics and DLQ.

The registry checks schemas, not which consumer versions are running. Deploy order is still the team’s job.

Operational details

• Protobuf matches by field number, not name. Renaming is wire-safe; reusing a number is not. Mark deleted numbers reserved.
• Proto3 preserves unknown fields, and unknown enum values are kept (open enums). Changing between int32, uint32, int64, uint64 and bool is compatible on the wire but may truncate.
• JSON Schema compatibility in Schema Registry depends on the open or closed content model (additionalProperties). Adding a property to a closed model is a different check than to an open one.
• Avro enum defaults exist since Avro 1.9. Give every enum an UNKNOWN default in v1: an old reader can only use a default it already has.
• Set the mode per subject (PUT /config/<subject>), not only globally, and disable auto.register.schemas in production so CI registers schemas.

Real-world example

A small check with avsc 5.7.7 reproduces three rows of the Avro table (run on Node 24, output trimmed):

import avro from 'avsc';
const T = (fields) => avro.Type.forSchema({ type: 'record', name: 'OrderPlaced', namespace: 'shop', fields });
const v1 = T([{ name: 'orderId', type: 'string' }, { name: 'amount', type: 'int' }]);
const v2 = T([{ name: 'orderId', type: 'string' }, { name: 'amount', type: 'long' },
              { name: 'currency', type: 'string', default: 'VND' }]);

const old = v1.toBuffer({ orderId: 'o-101', amount: 250000 });
console.log(v2.fromBuffer(old, v2.createResolver(v1)));
// { orderId: 'o-101', amount: 250000, currency: 'VND' }

v1.createResolver(v2);
// Error: cannot read "long" as "int"

Adding currency without a default makes v2.createResolver(v1) fail with no matching field for default-less shop.OrderPlaced.currency.

Migrating amount without a stop-the-world deploy

The checkout team runs expand–contract on the event. Expand: add amountMinor: long with a default, keep writing amount too. Each of the four consumers switches to amountMinor at its own pace.

Contract: when contract tests show no consumer reads amount, give it a default, stop filling it, and remove it after retention (7 days) plus the replay window.

Tolerant reader, upcasting, versioned topics

• Tolerant reader: consumers read only fields they need and ignore the rest. It makes additions safe; it does not save a consumer from a type change.
• Upcasting: the consumer converts old event versions to the latest shape in one place before handling. Common in event-sourced stores where old events can never be rewritten.
• Versioned events (OrderPlaced v1, v2 in one topic) keep ordering per key. Versioned topics (orders.order-placed.v2) allow a true break but need dual publishing and lose cross-topic ordering.

Trade-offs

Registry + compatibility mode

• Breaking schemas are rejected at registration time, before any consumer sees them.
• Payloads stay compact: an ID instead of field names.
• Deploy order becomes an explicit rule per mode.

Costs

• The registry checks schemas, not semantics: changing amount from VND to cents passes every check.
• Non-transitive modes miss conflicts with older versions still in the topic.
• Renames and narrowing need a multi-release expand–contract.

🎮 Try it

• Schema compatibility: classify 8 Avro and Protobuf changes as backward, forward, full or breaking.

Production checklist

1. Choose the mode per subject; prefer BACKWARD_TRANSITIVE or FULL_TRANSITIVE when consumers replay old data.
2. Register schemas from CI and set auto.register.schemas=false in producers.
3. Give every new field a default and every enum an UNKNOWN default from the first version.
4. In Protobuf, mark removed field numbers and names reserved.
5. Write down the deploy order in the PR: consumers first (BACKWARD) or producer first (FORWARD).
6. Run consumer-driven contract tests (for example Pact message pacts) in the producer’s CI.
7. Alert on deserialization errors and DLQ volume per consumer group.

Common mistakes

Renaming a field in place

✕ Common

{ "name": "totalAmount", "type": "long" }

✓ Prefer

{ "name": "totalAmount", "type": "long", "aliases": ["amount"] }

Why

Without the alias, the new reader finds no totalAmount in v1 data. The alias fixes only the new reader: old readers still need amount, so producers keep writing it until they upgrade.

Reusing a Protobuf field number

Deleting string coupon = 7 and later adding int64 discount = 7 makes old records decode ambiguously: a reader expects one wire type and finds another, or reads garbage. Reserve 7 instead.

Misunderstanding	Consequence	Do instead
“The registry accepted it, so it is safe”	Old consumers break on a BACKWARD change	Follow the mode’s deploy order
“BACKWARD checks all versions”	v3 cannot read v1 records still retained	Use _TRANSITIVE
“Compatible schema means compatible meaning”	Units or semantics change silently	Add a new field with a new name

Self-check

1. A v2 schema passes BACKWARD, and checkout deploys before analytics. Which records can analytics still read, and from which offset does it fail?
2. Why does adding an UNKNOWN enum default in v2 not help a v1 reader?
3. Why is renaming a Protobuf field safe while reusing its number is not?

Summary

Key takeaways

• Compatibility is about a writer–reader pair: backward = new reader, old data; forward = old reader, new data.
• Schema Registry defaults to BACKWARD; deploy consumers first, and use _TRANSITIVE when old data stays in the topic.
• Avro matches fields by name and needs defaults; Protobuf matches by number, which must never be reused.
• Renames, narrowing and meaning changes go through expand–contract, verified by contract tests.

References

• Confluent — Schema Evolution and Compatibility: modes, default, upgrade order.
• Confluent — Formats, Serializers, and Deserializers: wire format, subject naming.
• Avro 1.12.0 Specification: Schema Resolution, promotions, aliases.
• Protobuf proto3 guide: updating messages, reserved numbers, unknown fields.
• Fowler — Tolerant Reader: reading only what you need.
• Pact — How Pact works: message contracts.
• Related: Expand–contract migration, contract tests, Kafka as a distributed log, retention.