Kafka offset commit: before or after the side effect, and auto.offset.reset Intermediate

The problem

A consumer reads the event at offset 7, updates the DB, then tells Kafka it is done. The process can die between any two operations, and the Kafka offset and the DB are two systems with no shared transaction. So there are only two options: accept loss or accept repetition.

Default recommendation: write the side effect first, commit the offset after (at-least-once), and make the side effect idempotent. Losing business data usually costs more than reprocessing.

Two orders, two failure modes

Crash right after the first operation

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

Both consumers receive the record at offset 7 (the group's committed offset is 7).

Order	Crash right after the first step	On restart	What you need
Write DB, then commit 8	DB written, group still at 7	Record 7 is redelivered	Idempotent write
Commit 8, then write DB	Group at 8, DB not written	Record 7 is skipped	Unfixable without a manual replay

The committed value is always the offset to read next: after processing 7, commit 8.

Consumer config cheat sheet (Java client 4.2)

Config	Default	Recommendation
enable.auto.commit	true	false when side effects must not be lost; commit yourself after processing
auto.commit.interval.ms	5000	Only relevant when auto-commit is on
auto.offset.reset	latest	earliest for consumers that need all data; none when you prefer a loud error over a guess
max.poll.records	500	Lower it if each record takes long to process
max.poll.interval.ms	300000 (5 minutes)	Must exceed the time to process one batch (rebalance)
isolation.level	read_uncommitted	read_committed when reading topics with transactions (transactions)
offsets.retention.minutes (broker)	10080 (7 days)	Monitor it if the group can sit idle for long

Auto-commit in the Java client periodically commits, per auto.commit.interval.ms, the offsets that poll() has returned. The KafkaConsumer Javadoc is explicit: auto-commit only gives at-least-once if you finish processing all records from each poll() before the next call or before closing the consumer.

• Synchronous processing inside the loop: at-least-once.
• Handing records to another thread/queue and going back to poll: offsets can be committed before the work is done (work lost on crash).

KafkaJS has autoCommit, resolveOffset, commitOffsetsIfNecessary with their own behavior; test the crash window with the exact version.

auto.offset.reset: the real cause of incidents

This config only applies when the group has no valid offset: a new group, an offset expired after offsets.retention.minutes (empty group, no consumers left, more than 7 days), or an offset pointing at data already removed by retention.

Value	Consequence when the offset is lost
latest (default)	Skips everything old, silently losing unread data
earliest	Rereads from the start of the log, replaying everything, side effects rerun in bulk
none	Throws an error, forcing an operator to decide

For consumers with side effects, set earliest with idempotency, or none for sensitive data. Do not leave the default latest on a pipeline that must not lose data.

Commit strategies

Approach	When	Notes
Sync in the loop (commitSync after each batch)	Default for sequential processing	Simple, safe, slightly slower
commitAsync	High throughput	Cannot be retried safely because it may overwrite a newer offset; commit sync when closing the consumer
Commit in onPartitionsRevoked	Always worth having with manual commit	Saves progress before losing the partition
Offset in the same DB transaction	Want exactly-once with the DB	Manage positions yourself, seek on partition assignment
Kafka transaction	The output is also Kafka	Transactions article

With batches: the offset is contiguous progress, not a bitmap. If 7 and 9 are done but 8 failed, you cannot commit 10. Either process sequentially per partition, or track the highest contiguously completed offset.

The four-step flow

Record at offset 7: the database commit and the offset commit are two different milestones

Enlarge diagram

Kafka → Consumeroffset 7

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The consumer reads the record at offset 7; the group still stores 7 as the next position to read.

1. The broker delivers offset 7. Crash now: the group rereads 7 (normal redelivery).
2. The handler writes the DB in one transaction, including inserting event_id into a processed table with a unique constraint (idempotent consumer). SELECT then UPDATE is not enough when two processes run concurrently.
3. The DB commits. Crash now still redelivers 7; the handler sees the event_id already exists and skips it.
4. Commit offset 8. A commit failure does not mean the DB rolled back; accept the redelivery.

When to use / not use

• At-least-once + idempotent write: the default for billing, ledgers, projections.
• Commit first (at-most-once): only for data where loss is acceptable, like metrics or raw logs.
• Kafka transaction: when the output is a Kafka topic; it cannot cover PostgreSQL.

Real-world example

A deploy repeats an event, the ledger does not change

Billing reads OrderPaid, writes the ledger to PostgreSQL, then commits the offset; the ledger has a unique key on event_id.

During a deploy, the old consumer commits the DB but is stopped before committing the offset. The new consumer gets the partition and rereads the event; the insert hits a unique conflict, no money is added again, and the group moves on.

The redelivery metric rises slightly and the balance does not change. Switching to “commit before the DB” to get rid of duplicates would lose an entry on the next crash.

An in-memory model of the failure window (two variables, no Kafka or PostgreSQL):

npx --prefix examples ts-node examples/src/kafka-offset-commit/commit-window-demo.ts

Output: db-first gives dbApplied=true, committedNextOffset=7 (redelivery); offset-first gives dbApplied=false, committedNextOffset=8 (record 7 lost). Verify for real with an integration test that has a crash hook between the DB commit and the offset commit.

Trade-offs

Committing after the side effect

• Never skips a record while its side effect is not yet durable.
• Redelivery enables recovery after a crash.
• A unique event ID keeps the business write single even when delivery repeats.

The cost

• Handlers must be idempotent because both replays and rebalances bring records back.
• Manual commits require managing contiguous offsets and the partition lifecycle.
• There is no shared transaction for Kafka offsets and a DB outside Kafka.

Production checklist

1. Disable enable.auto.commit for consumers with important side effects, or make sure processing is synchronous inside the poll loop.
2. Set auto.offset.reset deliberately (earliest or none); do not rely on the latest default.
3. Idempotent side effects keyed on event_id (unique constraint).
4. Commit in onPartitionsRevoked and commit sync on shutdown.
5. Alert when a group has been inactive close to offsets.retention.minutes (7 days).
6. Test a crash between the DB commit and the offset commit.

Common mistakes

Common pitfall

Committing 7 after processing record 7. Commit 8; otherwise record 7 is read again after a restart.

Common pitfall

A callback that only enqueues a job and returns. The client resolves/commits the offset before the side effect. Await the right durability boundary.

Common pitfall

Leaving auto.offset.reset=latest on a pipeline that must not lose data. A group that loses its offset jumps to the end of the log and skips data without any error.

Summary

Key takeaways

• A commit records the next read position: process 7, commit 8.
• Commit before the DB and you lose; commit after the DB and you repeat; choose repeat plus idempotency.
• Java auto-commit commits offsets poll() has returned; safe when each batch finishes before the next poll, dangerous when work moves to another thread.
• auto.offset.reset decides between loss and replay when the group has no valid offset.

Self-check: (1) The DB committed but the offset is still 7; what does a restart do? (2) Why is committing 10 wrong when 8 has not been processed? (3) A group sits idle for 8 days and restarts with auto.offset.reset=latest; what happens?

References

• Message Delivery Semantics — Apache Kafka 4.2
• Consumer Configs — Apache Kafka 4.2
• Consuming messages — KafkaJS