SIGTERM during deploys: giving in-flight requests time to finish Intermediate

The problem

A deploy replaces an old API Pod with a new one. The old Pod is handling a create-order request: it holds a database connection and is waiting on the payment gateway. If the process is killed immediately:

• The client sees the connection drop, even though the payment succeeded downstream.
• A client retry without an idempotency key → risk of double payment.

Conversely, if the old Pod keeps accepting requests and waits indefinitely, the rollout drags on and the Pod is force-killed anyway.

Graceful shutdown means: stop accepting new work → let in-flight work finish within a deadline → close resources → exit.

It does not guarantee every request completes (node death, crashes, deadline exceeded), so business-level idempotency is still needed. But for normal deploys and scale-downs, it sharply reduces the number of requests cut off midway.

Core concepts

When a Pod is deleted, two things happen in parallel, not synchronized:

Flow	What happens	Notes
Traffic	The Pod’s endpoint in the EndpointSlice turns terminating, ready = false → kube-proxy/ingress stop sending new traffic	Takes a few seconds to propagate; clients may still hold old connections
Process	Run preStop (if any) → send SIGTERM → when the grace period ends, SIGKILL	Default terminationGracePeriodSeconds = 30

The most important consequence: SIGTERM arrives before traffic has fully stopped. If the app closes its listener as soon as it receives SIGTERM, requests still being routed to the Pod get connection refused.

A Pod moves from serving to draining before it is stopped

Enlarge diagram

Kubernetes · App

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The Pod starts terminating; the grace period begins and the endpoint is marked not ready.

Time budget: everything comes out of one grace period

preStop is not added on top; it eats into the same grace period as the drain. Example with a 30-second budget:

0s          5s                           25s        30s
├───────────┼────────────────────────────┼──────────┤
│  preStop  │   SIGTERM → drain requests │ cleanup  │ SIGKILL
│  sleep 5s │   server.close() waits     │ close    │
│           │                            │ pool     │
└───────────┴────────────────────────────┴──────────┘
 old traffic   stop accepting new, wait   flush logs,
 still arrives for in-flight to hit 0     close DB

Rule: terminationGracePeriodSeconds ≥ preStop + max drain time + cleanup margin. Measure the real drain time (p99 of the longest request); don’t pick an arbitrary number.

Configuring it correctly

1. Kubernetes: wait for traffic changes to propagate before sending SIGTERM

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["sleep", "5"]   # wait for the endpoint to be removed from the load balancer

An exec sleep hook works when the image has a sleep binary.

Kubernetes also has a native sleep action (preStop: { sleep: { seconds: 5 } }, where the kubelet waits itself, no binary needed in the image): alpha in v1.29, beta and on by default since v1.30, stable since v1.34 (feature gate PodLifecycleSleepAction); check the version of the cluster you use.

2. Container: make sure Node receives SIGTERM

# Right: exec form, node is PID 1 and receives SIGTERM directly
CMD ["node", "dist/main.js"]

# Wrong: shell form or npm start, the signal may never reach node
# CMD npm start

This is the most common cause of “graceful shutdown doesn’t run”: a wrapper process (sh -c, npm) does not forward SIGTERM, so the app never learns it is about to stop and gets SIGKILL after 30 seconds.

If you need an init process to reap zombies and forward signals, use tini or docker run --init.

3. Node application: drain with a hard stop

process.once('SIGTERM', () => {
  ready = false;                                    // readiness returns 503
  const hardStop = setTimeout(() => process.exit(1), 20_000);
  hardStop.unref();                                 // hard stop, does not keep the process alive

  server.close(async () => {                        // wait for in-flight requests
    await dbPool.end();
    process.exitCode = 0;
  });
});

preStop already handles waiting for propagation, so the listener can be closed right away here. Without preStop, wait a few seconds before server.close().

The code is still a sketch: it lacks closing consumers/workers, error handling in the callback, and WebSocket handling (upgraded sockets must be tracked and closed by you). With NestJS, use enableShutdownHooks() and lifecycle hooks, then verify the shutdown order.

Draining in Node: two calls to tell apart

Call	What it does	When to use it
server.close()	Stops accepting new connections and waits for running requests to finish. Since Node 19 it also closes idle (keep-alive) connections	The default on SIGTERM
server.closeAllConnections()	Forcibly closes connections, may cut running requests	Only at the final deadline, after close()

server.close() only closes the HTTP server. Database pools, broker consumers, workers, crons and long-lived sockets must be stopped separately, in order: stop taking new jobs → finish or release jobs in hand → flush logs/telemetry → close connections.

With queues, acking before a durable write can lose jobs and acking after can cause redelivery, so handlers must still be idempotent.

Real-world example

A rollout throws 502s despite a SIGTERM handler

An ordering API catches SIGTERM and calls server.close() by the book, yet every deploy still produces a few 502s. Cause: the Pod receives SIGTERM and closes its listener before the ingress has removed the endpoint, so new requests get connection refused.

The fix has three steps:

1. Add preStop: sleep 5 to wait for traffic changes to propagate.
2. Raise terminationGracePeriodSeconds from 30 to 45, because preStop eats into the budget.
3. Use timeouts on outbound calls and an idempotency key for payment operations, so cut requests can still be retried safely.

They verified it with a real rollout on staging under load: send deliberately slow requests, do a rolling restart, and check client-side status and the old Pod’s logs. A health check returning 200 does not prove the shutdown flow is correct.

Run the state simulation (no real Pods or signals):

npx --prefix examples ts-node examples/src/graceful-shutdown/shutdown-demo.ts

Expected output: old request started → shutdown reports it stopped accepting → new request rejected → old request completed → drained. Lower the deadline to 5 ms to see deadline reached before the old request finishes. In the demo, the deadline only reports state and does not kill the task; real Kubernetes sends SIGKILL.

Trade-offs

Benefits

• Fewer requests cut off midway during normal rollouts or scale-downs.
• A clear moment to stop accepting work, drain and close resources.
• Drain time can be measured, so the grace period is chosen from data.

Limits

• A long grace period slows rollouts and holds resources longer; too short and work gets cut.
• No protection against crashes, node loss or forced kills.
• HTTP, queues, WebSockets and databases each have their own lifecycle; one SIGTERM handler doesn’t cover them all.

🎮 Try it

• Graceful shutdown: put in order what happens when a Pod is terminated.
• Probe restart cascade: a misconfigured probe causes cascading restarts.

Production checklist

1. The Dockerfile uses exec form (CMD ["node", ...]), not npm start or shell form.
2. preStop waits a few seconds for the endpoint to be removed; terminationGracePeriodSeconds exceeds preStop + drain + cleanup combined.
3. The SIGTERM handler runs only once (process.once), has a hard time limit and does not call process.exit() immediately.
4. Readiness switches to failing when shutdown begins (health probes).
5. Close in order: HTTP → consumers/workers → pools → flush logs/telemetry.
6. Outbound calls have timeouts; important writes have idempotency keys (webhooks & idempotency).
7. Test rollouts with slow requests and keep-alive connections on staging, and track drain time and requests cut as metrics.

Common mistakes

Common pitfall

Calling process.exit() right inside the SIGTERM handler. Node does not wait for pending requests and callbacks, which can lose logs and side effects. Stop accepting new work, drain with a deadline, close resources, then let the process exit.

Common pitfall

Setting preStop: sleep 30 with a 30-second grace period. preStop uses the same time budget; the app has almost no time left to handle SIGTERM. Add up preStop, drain and cleanup when choosing the number.

Common pitfall

Treating graceful shutdown as a substitute for idempotency. Sudden node death or an exceeded deadline still produces ambiguous outcomes. Important side effects need their own retry, dedupe or reconciliation.

Summary

Key takeaways

• SIGTERM arrives before traffic fully stops; use a short preStop sleep to avoid connection refused.
• preStop, drain and cleanup share one grace period.
• The Dockerfile must let Node receive SIGTERM directly (exec form, or tini).
• server.close() only closes HTTP; pools, consumers, workers and WebSockets must be stopped separately.
• Test shutdown under load, and still design for idempotency when draining fails.

Self-check: (1) Does preStop add time beyond the grace period? (2) Has server.close() closed the database pool? (3) Why can a request still arrive after the Pod has started terminating?

References

• Termination of Pods — Kubernetes
• Liveness, Readiness, and Startup Probes — Kubernetes
• HTTP server.close — Node.js