Kafka replication: how ISR, acks and min.insync.replicas protect data Intermediate

The problem

The producer gets “success” for OrderPaid; a few seconds later the broker holding the leader dies and the event is gone. “We have three brokers, so the data is tripled” is not enough.

You need to know the RF, which replicas are keeping up, which acks the producer uses, and whether the cluster accepts writes when only one replica is in sync.

Durability is the combined result of producer config, topic/broker config and cluster health.

Recommended recipe

For important topics:

Setting	Value	Kafka default
Replication factor (RF)	3	default.replication.factor=1
min.insync.replicas	2	1
acks (producer)	all	all in the 4.x producer
unclean.leader.election.enable	false	false
enable.idempotence	true	true in 4.x (article)

The broker defaults (RF=1, min ISR=1) are not durable: an auto-created topic on a default cluster has only one copy. Disable topic auto-creation in production and create topics with an explicit RF=3.

Concepts

• RF: the number of replicas assigned to each partition; one leader, the rest are followers fetching from the leader. RF is a target, not a promise that all three have the latest record.
• ISR: the set of replicas keeping up with the leader. A follower that does not catch up within replica.lag.time.max.ms (default 30000 ms) is removed from the ISR, and rejoins once it catches up.
• acks=all: the leader acks when the entire current ISR has the record, provided the ISR size meets min.insync.replicas. It does not wait for replicas that left the ISR.
• min.insync.replicas: the minimum ISR size to accept an acks=all write; below it the broker returns NOT_ENOUGH_REPLICAS.
• acks=1: only the leader records it; if the leader dies before followers fetch, data reported as successful is lost. acks=0: the producer does not know whether the broker received it.

The ISR shrinking step by step

RF=3, min ISR=2: the ISR goes from 3 to 1

Enlarge diagram

Producer → LeaderOrderPaid

◀ Prev▶ PlayNext ▶↺ RestartStep 1/3

RF=3, min.insync.replicas=2, ISR = 3 replicas. The leader waits until the whole ISR has the record before acking: the write succeeds.

RF	ISR	min ISR	acks=all
3	3	2	Accepted
3	2	2	Accepted (one replica outside the ISR)
3	1	2	Rejected: durability chosen over write availability
3	1	1	Accepted, but only one copy: high risk

With min ISR=1, “all” no longer means three copies. minISR only constrains acks=all writes; producers using acks=1 are not protected, so audit every client’s configuration.

Leader failure and unclean election

When the leader dies, the controller picks a new leader from the ISR, so no committed record is lost. If the ISR is empty:

Configuration	Behavior	Trade-off
unclean.leader.election.enable=false (default)	The partition stays offline until an ISR replica comes back	Lose availability, keep data
true	A replica outside the ISR becomes leader	Acked records may be lost

For money-related data, keep false.

Note for Kafka 4.1+: Eligible Leader Replicas (ELR, KIP-966) is enabled by default on new clusters.

• When the ISR is empty, the controller can pick a leader from the ELR (replicas that left the ISR but are guaranteed to have data up to the high watermark) before considering the partition offline.
• The docs state that the semantics of min.insync.replicas change when ELR is enabled.

Check your cluster’s ELR state before reasoning with the table above.

A record becomes visible to regular consumers only once it is replicated to the whole ISR (committed, per the high watermark). A producer timeout does not mean the record was not written: the ack can be lost on the way back, so use an idempotent producer and an event ID.

When to trade off

Goal	Configuration
Must not lose (billing, ledger)	RF=3, minISR=2, acks=all, unclean=false
Tolerate small loss for faster writes (metrics, logs)	RF=3, acks=1 or minISR=1
2-broker cluster	minISR=2 with RF=2 blocks writes when one broker is under maintenance; consider 3 brokers

The cost of acks=all: waiting for followers, so ack latency depends on the slowest replica in the ISR. Measure it on your cluster before committing to an SLA; this article gives no latency numbers because they depend on hardware and network.

Real-world example

Broker maintenance without losing writes

The OrderPaid flow feeds billing; the topic has RF=3, minISR=2, and the producer uses idempotent acks=all. One broker goes into maintenance: the ISR drops to 2, and writes keep flowing. If a second broker also falls behind, writes start failing.

The API never returns “paid and event published” when the event is not durably written; it uses an outbox. The team monitors under-replicated partitions, ISR size and per-partition produce error rate, not just the number of live brokers.

A model of the configuration conditions (it does not simulate fetches, elections or timeouts; the repo compose has only one broker):

npx --prefix examples ts-node examples/src/kafka-replication/ack-policy-demo.ts

Output for RF=3, minISR=2: eligible at ISR=3, eligible at ISR=2, reject at ISR=1 with acks=-1; the acks=1, ISR=1 line is eligible. A real test needs at least three brokers and stopping them one by one.

Trade-offs

More data safety

• RF lets a partition survive broker failures.
• acks=all plus minISR blocks acks with too few in-sync copies.
• The ISR is a more specific health signal than the broker count.

Trade-offs

• A high minISR rejects writes early when the cluster degrades.
• More disk, network and ack latency.
• The producer still has to handle timeouts, retries and duplicates.

Production checklist

1. RF=3, min.insync.replicas=2 for every important topic; disable topic auto-creation.
2. Producer acks=all with idempotence on; audit every client so no acks=1 slips in.
3. unclean.leader.election.enable=false.
4. Spread replicas across different racks/AZs.
5. Alerts: under-replicated partitions, ISR size, per-partition produce error rate.
6. Be ready to handle NOT_ENOUGH_REPLICAS in the producer: retry with backoff and a deadline, report upstream when the write cannot be made.

Common mistakes

Common pitfall

Reading acks=all as “all replicas per RF”. It considers the current ISR; set minISR to get a floor.

Common pitfall

minISR=2 but the producer uses acks=1. The threshold does not protect that write. Audit the real producer config.

Common pitfall

Using the default RF=1. Losing a broker loses the partition. Create production topics with an explicit RF=3.

Summary

Key takeaways

• The ISR is the set of replicas keeping up (replica.lag.time.max.ms window, default 30 seconds).
• acks=all waits for the ISR; minISR sets a floor: RF=3 and minISR=2 accept writes with two left and reject with one.
• The broker defaults (RF=1, minISR=1) are not durable; production needs explicit configuration.
• Keep unclean.leader.election.enable=false when the data matters.

Self-check: (1) With RF=3, ISR=1, minISR=1, is an acks=all write rejected? (2) Why does minISR=2 not save an acks=1 producer? (3) Does a produce timeout prove the record was not written?

References

• Design: Replication — Apache Kafka 4.2
• Topic Configurations — Apache Kafka 4.2
• Broker Configurations — Apache Kafka 4.2