Incident response: declare early, mitigate first, root cause later Intermediate

This page answers: when the alert fires, what to do first, which metrics to look at, and which way of reducing damage to choose.

The lifecycle of an incident

Loading diagram…

Three points in the diagram that are often done wrong:

• Triage needs predefined criteria. The SRE book treats an event as an incident if any of three holds: a second team is needed to help; the issue is visible to users; it is still unresolved after an hour of focused analysis.
• Mitigate and investigate form a loop. If the SLI has not recovered after a rollback, the “it was the deploy” hypothesis is wrong; go back to investigating.
• Resolve is not the end. The postmortem and action items are what make the system better.

First response: symptom → what to look at

Use two frameworks so nothing is missed:

• Services → RED (Tom Wilkie): Rate, Errors, Duration.
• Resources → USE (Brendan Gregg): CPU, memory, connection pool, disk — “for every resource, check utilization, saturation, and errors”.

Symptom	Look at first	Common quick mitigation
5xx or latency rising right after a deploy	The deploy diff, when errors started relative to the rollout	Rollback
Latency up on every endpoint, high CPU	Event loop delay p99/max, then a CPU profile	Rollback; turn off a heavy feature with a flag
Latency up, CPU low	Waiting on I/O: pool.waitingCount, DB/downstream trace spans, pg_stat_activity	Reduce load, turn off the feature that calls the slow downstream
“timeout acquiring connection” from the pool	pg_stat_activity: many idle in transaction, old xact_start	pg_terminate_backend the stuck sessions (see note below), then fix the code holding transactions and set idle_in_transaction_session_timeout
Write requests hang	pg_blocking_pids + pg_locks	Cancel the blocking transaction, if you are sure it is safe
One endpoint slow, the rest normal	Slow query log / pg_stat_statements, then EXPLAIN (ANALYZE, BUFFERS)	Turn off that feature, add an index after review
Memory grows steadily, pod gets OOMKilled	Heap used over time, compare two heap snapshots	Scheduled restarts to buy time, limit payload size
Errors on only one pod / one zone	Metric breakdown by instance	Drain that instance or zone
Consumer lag rising	Lag per partition	Scale consumers (within the partition count)

A note on idle in transaction sessions: pg_terminate_backend kills the whole session and rolls back the transaction. pg_cancel_backend only cancels the running statement, so it cannot clear an idle session.

Going from an alert down to a specific request:

• Drill-down alert → exemplar → trace → log.
• Correlation ID to connect one request’s logs across services.
• Severity tied to SLOs and error budgets, not gut feeling.

Mitigate first, root cause later

The SRE book is blunt: the first instinct in a major outage is to rush to find the root cause — “ignore that instinct”. The top priority is to stop the bleeding: make the system work as well as it can under the circumstances.

Mitigation	Use when	Cost
Roll back the deploy	Errors started at the same time as the rollout	Lose the new feature; DB migrations must be backward compatible (expand/contract)
Kill switch (feature flag)	The error is confined to one feature	The flag must already exist; does not apply to errors in shared code
Scale out	Real saturation (CPU, workers) and downstreams can take more	May push load onto the DB and widen the incident
Shed load / rate limit	Overloaded, need to keep the important parts alive	Some users deliberately get 429/503
Drain an instance / zone	The error is localized	The rest has to carry extra load

Why mitigation comes first:

• Users do not care about the root cause. They only see how long the errors last.
• A rollback is a cheap experiment. If the SLI recovers, you have just narrowed the cause down to that deploy.
• Investigating under pressure creates new mistakes. Only once the system is stable can you dig deep calmly.

Before rolling back or restarting, preserve evidence if it takes under a minute: save a diagnostic report, snapshot pg_stat_activity, record the running version. A restart wipes the state you need for the postmortem.

When to apply / exceptions

Apply in full	Can be shortened
User-visible incidents, data loss, anything needing a rollback or manual intervention	Errors only in staging that affect nobody
Monitoring missed it and users reported it first	Small incidents with an existing postmortem for the same cause: update the old postmortem and push its action items
Near-misses that almost became a major incident	Teams of 2-3 people: one person is both IC and Ops, but there is still an incident doc and a postmortem

The exception to “mitigate first”: when mitigation can lose data — for example restoring a backup over newer data, or failing over while the replica is lagging. Then stop, assess, and let the IC decide.

On-call checklist

1. Alert received → acknowledge within the committed time, open the incident doc, record times in UTC.
2. Check against the declare criteria; when in doubt, declare and name an IC explicitly.
3. Ask “what just changed?”: deploys, config, flags, traffic, dependencies.
4. Preserve evidence in under a minute (diagnostic report, pg_stat_activity, version), then mitigate.
5. Mitigate with the cheapest reversible option: rollback, kill switch, drain, shed load.
6. Debug by hypothesis, one change at a time, record negative results in the incident doc too.
7. Update stakeholders regularly, always stating the time of the next update.
8. Resolve when the SLI is back to normal; Planning lists every temporary change to revert; open the postmortem.

Next: Hypothesis-driven debugging.