Ordering: the incident response sequence Basic
Estimated reading time: 1 min read
During an incident, everyone knows they must roll back, report and write a postmortem. What goes wrong is the order: investigating before mitigating, restarting before saving evidence, or announcing “fixed” before measuring the SLI.
Order the 8 steps below. Press Check to see which steps are out of place and what breaks when they are.
Incident response order: from the pager firing to action items
20:07 UTC, the checkout SLO burn-rate alert fires. 5xx errors are rising, p99 is up to 4 seconds; 40 minutes ago api v3.14 was deployed (no migration).
Put the 8 tasks below in the order an on-call engineer should do them.
Use ▲▼ to move each step, or tap a step’s ⠿ and then tap its new position. On a computer you can drag and drop.
- Ack the alert, open an incident doc, declare a SEV and assign an IC
record UTC times; with enough people, split out the Ops and Comms roles - Send the first status update: impact, what is being done, time of the next update
- Mitigate with the cheapest reversible action: roll back v3.14 → v3.13
- Blameless postmortem, each action item with an owner, a due date and a ticket
- Preserve evidence in under one minute
snapshot pg_stat_activity + pg_blocking_pids, diagnostic report, running version - Confirm recovery with the SLI, watch for another 15 minutes, then resolve
- Assess impact and ask "what just changed?"
which SLI, what % of requests, since when; deploys, config, flags, traffic - Find the root cause from the saved evidence, hypothesis by hypothesis
The full lifecycle and the exceptions to “mitigate first”: Incident response.