Design: move checkout from synchronous to asynchronous Intermediate

Checkout does everything in one request: p99 is 3.5 s, and when the email provider goes down nobody can buy anything. Drag components into four tiers to move side effects out of the request without losing events or sending duplicates.

• Goal: meet every requirement within the complexity budget.
• Also try: find one design using RabbitMQ and one using Kafka + CDC. Then try “publish after COMMIT” instead of an outbox and see how the number of lost events changes.

Design

Split a synchronous checkout into an async flow that never loses events

POST /checkout currently does everything sequentially in one request: create the order, charge the PSP, send the email, add loyalty points, push analytics. The result: p99 ~3.5 s, an email provider outage takes checkout down, and a deploy cutting in leaves orders with no email or points.

Payment must stay synchronous: the customer needs to know whether they paid. Everything else can run later. The Checkout API and PostgreSQL are already in place; you add components around them.

Assumptions for the game: DB ~40 ms, PSP p99 ~700 ms with a timeout (~2.2 s without), email ~900 ms, loyalty ~120 ms, analytics ~300 ms. Lost/duplicate counts are per 10k orders over a day with 2 deploys and a 30-minute email provider outage. Illustrative numbers, not benchmarks.

Requirements

• p99 of POST /checkout ≤ 1,000 ms; payment stays synchronous, with a timeout and an idempotency key
• Email provider down: 0 failed checkouts
• 0 lost and 0 duplicate side effects / 10k orders
• Confirmation email arrives within ≤ 60 s under normal conditions
• Complexity ≤ 14 units

Press ＋ Add on each layer to choose a component. On a computer you can drag a component onto a layer. Metrics update right away.

🛡️ PSP timeout + circuit breaker⚙️ Publish to broker after COMMIT🗄️ Idempotency key table🗄️ Outbox table🗄️ Inbox dedupe (event_id)⚙️ Send email inside the transaction🧩 2PC / XA DB + broker📨 Kafka📨 RabbitMQ📨 Debezium CDC (Kafka Connect)📨 Retry with backoff + DLQ🔧 Outbox relay (polling)🔧 Email worker🔧 Loyalty worker🔧 Analytics worker🔧 5-minute cron scanning orders

Tap a component to see what it does.

Users👤 Client

Checkout API⚙️ POST /checkout🔒＋ Add

PostgreSQL🗄️ orders, payments🔒＋ Add

MessagingBroker, CDC, retry/DLQ＋ Add

Background workersProcesses running outside the request＋ Add

Checkout p993,560 ms

Failed checkouts during email outage100 %

Lost side effects / 10k orders3

Duplicate side effects / 10k orders5

Email delay1 s

Complexity0 units

• The PSP call has no timeout: every time the PSP is slow, the request hangs with it, p99 climbs to ~2,200 ms and the worker pool runs dry.
• No idempotency key: the mobile app retries on a flaky network and creates duplicate orders and charges (~5/10k orders).
• confirmation email still runs synchronously in the request: +900 ms p99, and when the email provider goes down, checkout goes down with it; if the process dies midway, this work is lost.
• loyalty points still runs synchronously in the request: +120 ms p99; if the process dies midway, this work is lost.
• analytics push still runs synchronously in the request: +300 ms p99; if the process dies midway, this work is lost.

↺ Start over

Why an outbox prevents loss but not duplicates, and how relay polling differs from CDC: Transactional outbox.