Kafka message keys: per-order ordering exists only within one partition Intermediate

The problem

The order system publishes created, paid, shipped to one multi-partition topic without a key. Events of the same order land in different partitions, and Kafka promises no ordering across partitions. Result: shipped can be processed before paid.

The right question is “ordering of which object?”. For orders, orderId is the message key: the producer hashes the key to choose a partition, so every event of that order lives in one log and is read by offset.

Rules to remember

What you want	How	Limit
Ordering of one entity’s events	Key = entity id (orderId)	Only holds while the partition count and partitioner do not change
Even load distribution	Keys with many evenly distributed values	A very hot key creates a hot partition
No ordering needed, need throughput	No key (sticky partitioner)	No per-entity ordering
Ordering of side effect completion	Sequential processing within the partition	The key does not guarantee this

A key is not a unique ID: thousands of records can share the key order-42. It also does not prove the payload is valid or that a record is processed exactly once.

How a key reaches a partition

For keyed records, the Java producer’s default partitioner hashes the key and takes the remainder by the partition count:

• Kafka docs: “If no partition is specified but a key is present, choose a partition based on a hash of the key”.
• The KafkaJS docs state the algorithm is murmur2 and that KafkaJS’s DefaultPartitioner is compatible with the Java client.

Records without a key are assigned a “sticky” partition, which changes after every batch.size bytes.

The same key goes to one partition, the consumer reads offsets in order

Enlarge diagram

Producer → Partitioncreated

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The producer assigns key order-42 to the created event; the partitioner picks a partition.

Formula: partition = hash(key) mod N. With producers in other languages, check that they use the same hash algorithm; otherwise the same key can still map to different partitions.

Adding partitions breaks the mapping

Loading diagram…

• Old records of order-42 stay in partition 1; new records go to partition 2.
• Two different consumers handle the two partitions, so shipped can run before paid is applied.
• The safe approach: create a new topic with the desired partition count, drain the old topic to a clear cutoff, then switch producers; or over-provision partitions from the start.

Three layers of ordering

Layer	Guarantees	Does not guarantee
Producer	Sends in intended order	Multiple concurrent producers have no shared order
Partition	Append and read by offset	No ordering across partitions
Consumer effect	Applies events to the DB	No completion order if processing is async or retried

On the producer side: with idempotence on (the default in 4.x) and max.in.flight.requests.per.connection<=5, retries do not reorder batches within a partition (idempotent producer).

On the consumer side: if eachMessage only calls void doWork() and returns immediately, the offset advances before the work finishes, so paid can complete before created. await the durable work, or use a state machine that rejects invalid transitions and routes them to a DLQ instead of silently skipping.

Choosing a key

Key	Ordering	Risk
orderId	Per order: usually enough	Rarely hot
customerId	Per customer	Large customers pile too many orders into one partition
eventId or a fresh UUID each time	None	Events of the same order no longer share a partition
No key	None	Most even distribution

Recommendation: by default choose the id of the smallest entity that needs ordering (order, shipment), not the tenant id.

When to use / not use

• Use a key when the consumer applies events against entity state (state machine, projection).
• No key needed when events are independent (logs, metrics, unrelated jobs).
• Do not rely on keys for dedupe: that is the consumer’s idempotency job.

Real-world example

Fulfillment adds partitions

A fulfillment platform keys by orderId on a 12-partition topic (illustrative assumption). When it needs 24, the team does not increase in place: they create a new 24-partition topic, stop producers, wait for consumers to drain the old topic, then switch producers to the new topic.

Consumers store eventId for dedupe and commit offsets only after the effect is durable. For flows that only need per-shipment ordering, they change the key to shipmentId to reduce hot partitions.

Demo (two partitions, three records with the same key, replication factor 1). The demo uses the Kafka 3.6 broker in the repo compose; key-to-partition behavior is the same in 4.x.

docker compose -f examples/docker-compose.yml up -d zookeeper kafka
npx --prefix examples ts-node examples/src/kafka-message-key/key-order-demo.ts
docker compose -f examples/docker-compose.yml stop kafka zookeeper

Expected: three lines with the same partition, increasing offsets, values in order created, paid, shipped. KafkaJS JSON logs, the partitioner warning and TimeoutNegativeWarning are client noise.

Trade-offs

Keying by entity

• Events of the same entity live in one partition and are read by offset.
• Different keys still spread load across consumers.
• The ordering scope can be chosen per business need.

The limits that come with it

• A very hot key turns one partition into a bottleneck.
• Adding partitions or changing the partitioner changes the mapping.
• Offset order does not create idempotency or side effect completion order.

Production checklist

1. Choose the key as the id of the entity that needs ordering; document the decision.
2. Provision partitions for growth, because adding them later changes the mapping.
3. All producers (including other languages) use the same partitioning algorithm.
4. Consumers await side effects before committing; a state machine rejects invalid transitions.
5. Measure lag and throughput per partition to detect hot keys.

Common mistakes

Common pitfall

Assuming Kafka has topic-wide ordering. Ordering exists only within a partition; offsets cannot be compared across partitions.

Common pitfall

Adding partitions without planning for keys. New records of the same key can move to another partition and run ahead of the old backlog.

Common pitfall

Starting async side effects without waiting. Kafka delivers the right offsets, but effects complete out of order.

Summary

Key takeaways

• A key is routing information, not a unique ID.
• Ordering lives within a partition: partition = hash(key) mod N; changing N changes the mapping.
• The consumer must still control concurrency, retries and idempotency.
• Default: key by the smallest entity that needs ordering.

Self-check: (1) Why is offset 5 of partition 0 not “earlier” than offset 6 of partition 1? (2) What does going from 3 to 4 partitions do to new records of order-42? (3) Why does an eachMessage that returns right after void sendEmail() break effect ordering?

References

• Producer Configs — Apache Kafka 4.2
• Design — Apache Kafka 4.2
• Producing Messages — KafkaJS
• Consuming Messages — KafkaJS