Scenario: probes trigger cascading restarts across the whole Pod fleet Intermediate

A reporting query slows the database down. Because the liveness /health endpoint calls the database with timeoutSeconds: 1, kubelet restarts all 8 Pods. The new Pods pile load onto the database as they start, and the whole fleet ends up in CrashLoopBackOff.

You go through four decisions: stop the bleeding, root cause, probe redesign and action items.

• If you pick wrong, read the explanation and Choose again.
• Only your first choice at each step counts toward the score. Hint: for each probe, ask “would a restart fix this failure?”.

Scenario

The database slows down, and every Pod goes into CrashLoopBackOff

You are on call for the order API team (NestJS, 8 Pods, each Pod with a pool of 20 connections to one shared PostgreSQL primary). Every choice shows its consequences right away on the metrics below.

Pods ready / 80

5xx error requests94 %

Container restarts in the last 15 minutes41

Database CPU98 %

Step 1/4 · 14:09, no Pod is receiving traffic

At 14:02 a reporting query doing a full scan on the primary slowed the database down: ordinary queries went from 5 ms to 1–2 seconds. Two minutes later, all 8 Pods were restarted. A new Pod has to load config and warm its cache from the database, which now takes 70 seconds instead of 25, and then it gets killed again.

Now 6/8 Pods are in CrashLoopBackOff, the Service has no endpoints left, and almost every request returns 503.

Pods

0/8 ready, 6 CrashLoopBackOff

reporting query

running for 7 minutes, pid 48213

Pod event loop delay

normal, Pod CPU 20%

livenessProbe:  { httpGet: { path: /health, port: 3000 }, periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3 }
readinessProbe: { httpGet: { path: /health, port: 3000 }, periodSeconds: 10, timeoutSeconds: 1 }
# /health = TypeOrmHealthIndicator.pingCheck('database')

Events:
  Warning  Unhealthy  Liveness probe failed: Get "http://10.0.3.17:3000/health":
           context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    Container api failed liveness probe, will be restarted

What do you do?

AScale from 8 to 20 Pods for more capacitykubectl scale deployment/order-api --replicas=20BRestart PostgreSQL for a clean slateCPatch liveness to tcpSocket (no database call), then cancel the reporting querykubectl patch deployment order-api ... livenessProbe: { tcpSocket: { port: 3000 } }; SELECT pg_cancel_backend(48213);DTemporarily remove the livenessProbe from the Deployment entirely

When you finish, read Kubernetes probes for a table that picks probes by failure type and a minimal manifest.