Saga pattern: undoing a tour booking that spans four services Intermediate

The problem

A customer books a tour package saga-123. Four services each own a database: Seat holds seat 12A, Hotel holds room 204, Payment charges the card, Ticket issues the e-ticket.

Seat and hotel succeed and commit. Payment is declined. Seat 12A and room 204 are now held for a customer who will never pay, and other customers see them as taken.

Inside one database this is a ROLLBACK. Across four databases there is no shared transaction to roll back.

How does the system undo committed work in other services, in what order, and what do other users see in the meantime?

Which approach? (summary first)

Situation	Choose	Condition
All steps write to one database	A local transaction, not a saga	Constraints and locks do the work
2–3 services, linear flow, one team	Choreography saga (events)	Each service has an idempotent consumer and publishes via outbox
4+ steps, branches, timeouts, several teams	Orchestration saga	The orchestrator persists its state and is itself restartable
A step cannot be undone (email sent, money moved out)	Put it after the pivot	Make it retriable until it succeeds
The flow lasts hours or days, many timers	Durable execution engine (Temporal, Step Functions)	Activities are idempotent; workflow code follows the engine’s rules

Core concepts

A saga is a sequence of local transactions T1 … Tn. Each Ti commits in one service and has a compensation Ci. The 1987 paper by Garcia-Molina and Salem guarantees that either T1 … Tn runs, or T1 … Tj, Cj … C1: compensations in reverse order.

Why not a database transaction or 2PC?

• A database transaction ends at the database boundary; Seat and Hotel are different databases.
• Two-phase commit (2PC) makes participants prepare and hold locks until the coordinator decides. Many stores and brokers in a database-per-service design do not support it, and a coordinator failure leaves participants blocked.
• Holding locks in four databases while a payment gateway answers is the “long lived transaction” problem the saga paper set out to avoid.

Three kinds of step

Kind	Meaning	In the tour saga
Compensable	Has a semantic undo	T1 ReserveSeat → C1 ReleaseSeat; T2 ReserveHotelRoom → C2 CancelHotelRoom
Pivot	Point of no return: if it succeeds, the saga must finish	T3 ChargePayment
Retriable	After the pivot; cannot fail for business reasons, only retried	T4 IssueTicket

Placing the steps is a design decision:

• Holds are cheap to release, so they go first.
• The charge goes last among the risky steps, because a refund is a visible, slow undo.
• Ticket issuing goes after the pivot, because “no ticket for a paid seat” has no acceptable compensation.

Compensation is semantic undo, not rollback

ROLLBACK erases a change as if it never happened. A compensation is a new business transaction with the opposite effect. Other transactions may have run in between, so it cannot restore an old snapshot.

• ReleaseSeat flips only this saga’s hold; it does not write back an old seat count.
• A refund still shows a charge and a refund on the statement, can take days, and fees may not come back. Refunded is not never charged.

Choreography or orchestration

Criterion	Choreography	Orchestration
Who decides the next step	Each service reacts to events	The orchestrator sends commands
Where the flow lives	Spread across consumers	One state machine
Adding a step	Touch several services	Touch the orchestrator
Seeing “where is saga-123?”	Correlate events across services	Read one row
Risk	Cyclic event dependencies, hard integration tests	The orchestrator is a critical component to keep available

The rest of this article uses orchestration, because the tour saga has four steps, a pivot and timeouts.

How it works

The orchestrator runs T1 and T2, gets PaymentDeclined at T3, then compensates in reverse order.

Payment declined: state goes to COMPENSATING first, then hotel, then seat, then COMPENSATED

Enlarge diagram

Orchestrator → SeatReserveSeat

◀ Prev▶ PlayNext ▶↺ RestartStep 1/7

T1: the orchestrator stores saga-123 as RUNNING and sends ReserveSeat. Seat 12A is held.

After step	Seat 12A	Room 204	Saga state	Can be resumed after a crash?
T2	held	held	RUNNING, step charge_payment	Yes, resend the charge (same key)
PaymentDeclined persisted	held	held	COMPENSATING, step cancel_hotel	Yes, resend CancelHotelRoom
C2 confirmed	held	available	COMPENSATING, step release_seat	Yes, resend ReleaseSeat
C1 confirmed	available	available	COMPENSATED	Terminal, nothing to resume

Why reverse order: after C2 the system is in a state that also existed during the forward run (seat held, no hotel). Undoing the seat first would create “a hotel room for a passenger without a seat”, a state no validation was written for.

Saga state is a persisted state machine

The orchestrator can crash between any two steps, so the database row is the saga, not process memory.

Loading diagram…

Each transition and the next command are written in one local transaction: the state row plus a row in the transactional outbox. Otherwise the state can say COMPENSATING while CancelHotelRoom was never sent, a dual write.

BEGIN;
UPDATE saga_instance
   SET state = 'COMPENSATING', step = 'cancel_hotel', version = version + 1, updated_at = now()
 WHERE saga_id = 'saga-123' AND state = 'RUNNING' AND version = 3;   -- 0 rows: someone else moved it
INSERT INTO saga_outbox (saga_id, type, payload)
VALUES ('saga-123', 'CancelHotelRoom', '{"idempotencyKey": "saga-123:hotel:cancel"}');
COMMIT;

The version check stops two orchestrator instances from both driving the same saga.

Operational details

• Idempotent steps and compensations. The relay and the broker deliver at least once, so every command carries a stable key (saga-123:hotel:cancel) and participants deduplicate it, see idempotent consumer and idempotency.
• Compensating something that never happened. A timed-out ReserveHotelRoom may or may not have committed. CancelHotelRoom must succeed on “no such hold”, and must leave a tombstone so a late ReserveHotelRoom does nothing.
• Compensations must eventually succeed. Retry with backoff and the same key, never “skip and move on”. Exhausted retries go to NEEDS_ATTENTION and a human queue.
• Timeouts per step. A timeout means “unknown”, not “failed”: query by key or retry the same command before compensating.
• Stuck sagas. Sweep non-terminal sagas with an old updated_at and alert. Example threshold (illustrative): 15 minutes for this flow.
• Durable execution. Engines such as Temporal persist workflow progress and retry activities, but activities may still run more than once and should be idempotent.

Isolation: others see the saga mid-flight

A saga has no “I” from ACID. Each Ti commits and becomes visible before the saga finishes.

Saga B decides on room 204 while saga A is unfinished, then A compensates

◀ Prev▶ PlayNext ▶↺ RestartStep 1/5

Saga A commits its hotel step locally: room 204 = BOOKED. No lock spans the whole saga.

1. Saga A commits room 204 = BOOKED; nothing locks it for the rest of the saga.
2. Saga B reads BOOKED and treats the room as sold: a dirty read at business level.
3. A’s payment is declined, and A compensates.
4. Room 204 is free, but B’s decision is already made.

Countermeasures (Azure, microservices.io):

Countermeasure	In the tour saga
Semantic lock	Hold status PENDING(saga-123) with an expiry, not BOOKED; search shows “on hold, check again in 10 min”
Reorder steps (pessimistic view)	Risky, reversible steps first; irreversible updates only in retriable steps after the pivot
Reread value	Before T4, reread the hold and verify it still belongs to saga-123

Real-world example

Tour holds that outlived declined payments

An illustrative tour platform marks holds BOOKED. During a promotion, declines rise and compensations lag: rooms look sold out for minutes while their sagas are already failing.

Fix: holds become PENDING with expires_at = now() + 15 min, search counts them separately, and the hotel service expires them itself. Compensation is the fast path; expiry is the safety net.

Demo on PostgreSQL 17 (state transition, stale duplicate, tombstone, stuck-saga sweep):

docker compose -f examples/docker-compose.yml exec -T postgres psql -U hub -d hub < examples/src/saga-pattern/demo.sql

Result: the transition returns UPDATE 1, the stale duplicate UPDATE 0, the redelivered cancel and the late reservation both INSERT 0 0, and the sweep finds saga-123 in COMPENSATING.

Trade-offs

What a saga gives

• Each service keeps its own database and short local transactions; no cross-service locks.
• Partial failure ends in a defined business state (COMPENSATED), not a half-booked tour.
• Orchestration gives one place to see and resume every saga.

What it costs

• Every compensable step needs a designed, tested compensation.
• No isolation: intermediate states are visible and need semantic locks or reordering.
• More moving parts: outbox, idempotency keys, timeouts, a sweep, a manual queue.

🎮 Try it

• Saga compensation: put the recovery steps after a declined payment in the only correct order.

Production checklist

1. Label each step compensable, pivot or retriable; move irreversible actions after the pivot.
2. Persist saga state with a version column; commit each transition with its outbox command.
3. Give every command and compensation a stable key; treat “not found” on cancel as success.
4. Set a timeout per step; on timeout, query or retry with the same key before compensating.
5. Use PENDING + expiry for anything others can read mid-saga.
6. Alert on old sagas in COMPENSATING and on any NEEDS_ATTENTION.
7. Test a failure at every step, including a crash between sending a command and its reply.

Common mistakes

The most common one treats compensation as a database rollback.

Restoring an old value instead of applying the opposite change

✕ Common

-- C1: write back the count read before T1
UPDATE flight SET seats_left = 37 WHERE id = 'FL-123';

✓ Prefer

-- C1: undo only this saga's hold, idempotently
UPDATE seat_hold SET status = 'RELEASED'
 WHERE saga_id = 'saga-123' AND status = 'HELD';

Why

Other bookings changed seats_left after T1. Writing back 37 erases them; flipping this hold only undoes what this saga did.

A second trap sits in the orchestrator’s own code.

Marking the saga COMPENSATED before compensations are confirmed

If the orchestrator writes COMPENSATED right after sending ReleaseSeat, a lost message leaves seat 12A held forever, and the stuck-saga sweep no longer looks at this saga. Move to the terminal state only after every compensation reply arrives.

Misunderstanding	Consequence	Do instead
“A refund undoes the charge”	Charge and refund both visible	Charge as the pivot, after cheap holds
“Timeout = failed”	A late commit is never compensated	Query or retry by key; tombstone on cancel
“Process memory is the state”	A crash loses the saga	State row + outbox in one transaction

Self-check

1. Room 204’s ReserveHotelRoom timed out and the saga compensates. What must CancelHotelRoom do if the reservation arrives after the cancel?
2. Why is ChargePayment the pivot rather than ReserveSeat?
3. Saga B reads room 204 as BOOKED while saga A is still running. Which countermeasure changes what B sees, and what does B see instead?

Summary

Key takeaways

• A saga is local transactions plus compensations run in reverse order.
• Compensation is a new business transaction; a refund is not “never charged”.
• Order steps compensable → pivot → retriable, cheap holds first.
• Persist state with the outbox, keep every step idempotent, sweep for stuck sagas.
• No isolation: use semantic locks, reordering and rereads.

References

• Garcia-Molina & Salem — Sagas (1987): original definition, semantic compensation, the reverse-order guarantee.
• Microservices.io — Pattern: Saga: choreography vs orchestration, atomic publishing.
• Azure — Saga design pattern: pivot and retryable steps, anomalies, countermeasures.
• AWS — Saga orchestration: why 2PC does not fit database-per-service; Step Functions example.
• Temporal — Activity Definition: activities may run more than once.