Kafka Share Groups (Kafka Queues): many workers processing one partition Intermediate

The problem

Topic image-jobs has one partition. You grow the group from 1 to 8 consumers but throughput does not change: in a regular consumer group, a partition is assigned to only one consumer, and the other seven sit idle. Adding partitions changes key distribution and ordering, and cannot be undone.

Share Groups (also called Kafka Queues, KIP-932) let many consumers take different records from the same partition. Each record is held by a time-limited lock and then acked individually. Producers still write to a regular Kafka topic.

Status by version

Version	Status	Source
4.0	Early access (test clusters only, cannot upgrade to 4.1)	4.0 release announcement
4.1	Preview, enabled with kafka-features.sh (share.version=1)	Upgrade notes
4.2	Production-ready	Upgrade notes and release announcement

Operational notes for 4.2: clusters with fewer than 3 brokers must set share.coordinator.state.topic.replication.factor and share.coordinator.state.topic.min.isr to 1 before use; the docs state that 4.2.1 fixes a serious deadlock in the share group path, so use the latest patch.

This repo uses the cp-kafka:7.6.0 broker (Kafka 3.6, ZooKeeper) and KafkaJS 2.2 for older demos, so it cannot run Share Groups; KafkaJS has no Share Consumer API. The example below is an in-memory model.

Comparison

	Consumer group	Share group	RabbitMQ quorum queue
Unit of work distribution	Partition	Record	Message
Useful workers	At most the partition count	Not limited by partitions	Many consumers on one queue
Processing order	By offset within a partition	Not guaranteed	Not guaranteed with multiple consumers
Ack	Contiguous offset commit	Per record (ACCEPT/RELEASE/REJECT/RENEW)	Per message (ack/nack)
Redelivery limit	None	group.share.delivery.count.limit, default 5	delivery-limit, default 20 since 4.0
Lock expiry	None	share.record.lock.duration.ms, default 30 seconds	Per consumer ack timeout
Replay	Seek to offset	Not a goal	No (regular queues)
Multiple groups reading the same data	Yes	Yes (topic retention still applies)	Needs fan-out through an exchange

Config cheat sheet (Kafka 4.2)

Config	Default	Notes
group.share.record.lock.duration.ms (broker) / share.record.lock.duration.ms (group)	30000	Minimum 1000 in the group config
group.share.delivery.count.limit (broker)	5	Beyond this many deliveries a record is not delivered again
group.share.partition.max.record.locks (broker)	2000	Cap on locked records per partition
share.auto.offset.reset (group)	latest	A new share partition starts at the end of the log, not the beginning
share.isolation.level (group)	read_uncommitted	read_committed to skip uncommitted transactions
share.session.timeout.ms	45000	Detects dead clients
share.heartbeat.interval.ms	5000

Watch out for share.auto.offset.reset=latest: records produced before the first share group subscribes will not be delivered; to process an existing backlog, set earliest.

Mechanism

Two workers take different records from the same partition

Enlarge diagram

Partition → Worker Aoffset 7

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

A acquires offset 7. The record is temporarily locked for A and has not been successfully acked yet.

Kafka picks available records and acquires them for a consumer with a time-limited lock. While the lock is valid, the record is not delivered to another consumer; the next record of the same partition can be taken by a different worker.

With explicit acknowledgement (Java client):

Outcome	AcknowledgeType	Meaning
Success	ACCEPT	Marks it as processed
Transient error	RELEASE	Releases it for redelivery
Cannot be processed	REJECT	Not redelivered within the group
Long-running work	RENEW	Extends the lock

acknowledge() only records the intent on the client; acks are sent with the next poll or with commitSync()/commitAsync().

The default implicit mode treats records from the previous poll as successful when you poll again: convenient, but dangerous if the batch has a failing record. Use explicit mode for jobs that need error classification.

Three situations to think through first

1. Out-of-order completion. A takes offset 7, B takes offset 8 and finishes first. If 8 depends on 7, the result is wrong. Share groups promise no completion order.
2. The lock expires while the worker is still running. A takes longer than 30 seconds, record 7 is delivered to B, and A still calls the external API: a duplicate side effect. A broker lock does not replace idempotency; use RENEW for long work.
3. Permanent errors. REJECT or running out of deliveries only stops redelivery within the share group; it does not create a dead-letter topic. You must design the error store, alerts and replay yourself.

Share group or RabbitMQ quorum queue

Situation	Choose
Already on Kafka, independent jobs, need more workers than partitions, the same data is also read by analytics	Share group (Kafka 4.2+)
Need pattern routing, priorities, rich TTL/DLX	RabbitMQ (queue types)
Per-key ordering matters	Consumer group, not a share group
Not yet running Kafka 4.2	Keep the consumer group with enough partitions, or use RabbitMQ

When to use / not use

• Use: independent jobs (thumbnails, sending email, calling APIs), “queue on Kafka” workloads.
• Do not use: per-entity state machines, ledgers that need ordering, anywhere offset order is an invariant.
• Do not use if you cannot yet upgrade the cluster to 4.2 and clients to the Share Consumer API.

Real-world example

A thumbnail service

A thumbnail service receives independent image jobs from a topic with few partitions. A share group lets many workers take jobs from the same partition without changing the partition count.

Each job has a stable jobId; the DB stores status with a unique jobId, and object storage writes use idempotent keys.

Processing outcome	Worker sends
Result durably stored	ACCEPT
Transient provider error	RELEASE
Corrupt file (already written to the error table)	REJECT

The team measures attempts, lock expiries and jobs with duplicated side effects.

An in-memory model (not output from real Kafka): A acquires 7, B acquires 8 and accepts, the lock on 7 expires and B receives 7 on the second delivery; the last line is completion order: 8, 7.

The lock in the demo is 30 ms to run fast; the real default is 30 seconds.

npx --prefix examples ts-node examples/src/kafka-share-groups/share-group-model.ts

To verify for real: a Kafka 4.2+ cluster and the Java KafkaShareConsumer, a one-partition topic, two consumers with the same group.id and share.acknowledgement.mode=explicit; stop one worker after acquire and before ack to observe redelivery.

Trade-offs

When share groups fit

• Many workers on one partition, not limited by the partition count.
• Record-level ack and retry, per-job error classification.
• Still a Kafka topic: other groups read independently.

Trade-offs

• No completion order guarantee.
• Expired locks, uncommitted acks and retries still cause duplicate processing.
• Needs 4.2+ brokers and clients; extra operational work for delivery attempts, errors, retention.

Production checklist

1. Use only for independent jobs; confirm there is no ordering invariant.
2. Every job has a stable jobId and writes results idempotently.
3. Use explicit acknowledgement; ACCEPT only after the result is durable.
4. Set the lock duration above p99 processing time, use RENEW for long work.
5. Design your own error store for REJECT and jobs out of deliveries (default 5).
6. Set share.auto.offset.reset deliberately (earliest if you need to process a backlog).
7. Track record age against the topic retention.

Common mistakes

Common pitfall

Switching a consumer group to a share group just for throughput. OrderPaid can complete before OrderCreated. Keep the consumer group if ordering is an invariant.

Common pitfall

Treating ACCEPT as an exactly-once side effect. The ack may not be committed yet, and the lock may expire while the old handler is still running. Business writes must be idempotent.

Common pitfall

Assuming “Kafka Queues” come with a DLQ. REJECT is only a share group state; design error storage and replay yourself.

Summary

Key takeaways

• Share Groups: early access in 4.0, preview in 4.1, production-ready in 4.2.
• Work is distributed per record with a 30-second lock, a redelivery limit of 5, and at most 2000 locked records per partition (defaults).
• They trade ordering for throughput; locks and retries do not replace idempotency.
• The repo uses Kafka 3.6, so it cannot run Share Groups.

Self-check: (1) One partition and eight workers: how many can a consumer group use, and a share group? (2) Can offset 8 complete before 7? (3) Why can worker A, still running after its lock expired, cause a duplicate side effect?

References

• The Share Consumer — Apache Kafka 4.2
• Group Configs — Apache Kafka 4.2
• Upgrade notes — Apache Kafka 4.2
• KafkaShareConsumer API — Apache Kafka 4.2
• Apache Kafka 4.2 release announcement
• Quorum Queues — RabbitMQ