Scenario: a retry storm takes down the whole chain Intermediate

A slow service rarely takes down a whole system on its own. What does is retries at every tier plus long timeouts: load multiplies exactly when the dependency is weakest, and the pools drain from the bottom up.

• Four steps: read the dashboard, stop the bleeding, fix the policy, verify.
• Watch the request/s ratio between hops, not just each service in isolation.
• Only your first choice at each step counts toward the score.

Scenario

Retry storm: pricing slows to 3 seconds and the whole checkout chain collapses

You are the backend on-call. The call chain: gateway → order → inventory → pricing. The metrics below change with each decision.

Load on pricing / load on gateway7 ×

5xx at the gateway38 %

Checkout p99 at the gateway28,000 ms

order thread / connection pool100 %

Step 1/4 · Dashboards red from the outside in

At 20:05, pricing deployed a new version that dropped an index; pricing p99 went from 200 ms to 3 s. At 20:12, the gateway reports 38% 5xx errors. All three upstream services share the same downstream call config: 3 attempts, no backoff, 10 s timeout.

You open the dashboards. What is your first conclusion?

order pool

200/200 busy

inventory pool

200/200 busy

pricing CPU

98%

user traffic

unchanged

# incoming request/s, 20:12
gateway     2,000   (normally 2,000)
order       5,400
inventory   9,800
pricing    14,000   (capacity when slow ~1,200)

# trace of one failed request
gateway → order ×3 → inventory ×3/each → pricing ×3/each  = 19 pricing spans

What do you do?

Aorder-service is broken: pool at 100%, not accepting new requests. Restart all order podsBAn attack or a traffic spike: turn on a strict rate limit at the gatewayCRetry storm: request/s grows at each hop even though outside traffic is flat. Slow pricing is the spark; retries at 3 layers are what fan it into a collapse of the whole chainDPricing p99 of 3 s is the cause. Page the pricing team and wait for their rollback

The amplification table by number of tiers and the jitter formulas are in Retry, backoff and jitter.