Zero-Downtime Deploys for Long-Running Services — Connection Draining, Readiness Gating, and the Graceful-Shutdown Contract

A deploy done right is one the user never feels happen.

§ IFrame

A deploy replaces a running process with a newer one. The naive way kills the old process and starts the new. For a few seconds, requests that were already inside the old process die with it, and requests arriving during the gap hit nothing. The user sees a 502. The trader sees a frozen book. Nobody intended an outage, yet the deploy caused one, and it caused one every single time the team shipped.

The Sunday arc has worked the release surface from the top down. The 05-31 lesson built atomic rollouts, canary slicing, and the rollback surface — the orchestration that decides which instances move to the new version and when. That lesson assumed something it never named: that an instance, once told to step down, steps down cleanly. This lesson names it. The orchestration is the general moving regiments; the graceful-shutdown contract is what a single soldier does when ordered to fall back, which is to finish the round he is firing before he turns.

Zero-downtime deploy is not one technique. It is a contract between three parties: the load balancer that routes traffic, the orchestrator that sequences the rollout, and the process that must drain before it dies. Break any one term of the contract and the deploy drops requests. Hold all three and the user never knows you shipped.

§ IIFoundations

Three foundations carry the discipline.

The old instance must outlive its own shutdown signal. When an orchestrator decides to replace an instance, it sends a termination signal — SIGTERM in the POSIX world, the same thing a Kubernetes pod receives before its grace period. The wrong response is to exit immediately. The right response is to stop accepting new work, finish the work already in hand, and then exit. The window between the signal and the exit is the drain window, and the whole art of zero-downtime is using that window correctly. An instance that exits on the first signal is an instance that hangs up on every caller mid-sentence.

Readiness and liveness are different questions, and the deploy lives in the gap between them. A liveness check asks is this process alive enough to keep running, or should it be killed and restarted. A readiness check asks should this process receive traffic right now. The two are not the same, and conflating them breaks deploys. Kubernetes: Up and Running separates the health surfaces precisely so the platform can quarantine a container from traffic without killing it (Ch. on health checks, p. 130): a draining instance is alive but not ready, and the readiness signal is how it tells the load balancer to stop sending new requests while it finishes the old ones. A new instance is the mirror case — alive the instant it starts, but not ready until its caches are warm and its dependencies answer.

Traffic-shifting is asynchronous, so the drain must wait for the routers to notice. When an instance flips its readiness to false, the load balancer does not stop routing to it instantly. There is a propagation delay — a health-check interval, a DNS time-to-live, a service-mesh reconcile loop — during which traffic still arrives at an instance that has already said it is going away. The instance must keep serving through that delay. Flip readiness to false, then keep accepting connections for a few more seconds anyway, because the routers are still catching up. Only after the propagation delay has passed is it safe to stop the listener.

§ IIIMechanism — The Drain Sequence

Name the steps and the contract becomes executable. Call this the five-beat drain: the ordered sequence every long-running instance runs between its termination signal and its exit. The order is the whole point; a swap of any two beats reintroduces dropped requests.

The mirror sequence governs the new instance. It starts, reports alive, but holds readiness false until its startup work is genuinely done — connection pools opened, caches primed, a dependency ping answered. Only then does it flip readiness true and begin receiving traffic. A rolling update is these two sequences interleaved across the fleet: one instance running the five-beat drain while its replacement runs the warm-up, the orchestrator never removing an old instance until a new one is ready, so capacity never dips below the line.

§ IVWorked Example — Rolling a Market-Data Gateway

Trace one deploy of a service that holds long-lived connections: a gateway that fans streaming market data out to browser clients over WebSockets, eight instances behind a load balancer, deploying a new build.

The orchestrator picks instance one and sends SIGTERM. Beat one: the handler fires, no exit. Beat two: instance one fails readiness; the load balancer's next health poll, two seconds later, marks it unhealthy and stops routing new WebSocket upgrades to it. Beat three: instance one sleeps fifteen seconds — comfortably past the load balancer's de-registration window — and during those fifteen seconds it still accepts the handful of upgrades from routers that had not yet re-polled, and it keeps streaming to the hundreds of clients already connected.

Beat four: instance one stops accepting new connections and begins draining. Streaming clients are long-lived, so naive draining would wait forever. The gateway handles this by sending each connected client a "reconnect" control frame, which tells the browser to open a fresh connection — which the load balancer routes to a healthy instance, because instance one is already unhealthy. Within the drain timeout, clients migrate themselves off instance one. Beat five: drained, instance one exits clean. The orchestrator brings up the new build, waits for it to warm its upstream feed and flip readiness true, then moves to instance two. Eight instances, eight drains, zero dropped streams. The user saw, at most, one reconnect — the same thing a brief network blip would cause, which their client already handles.

The naive deploy of the same gateway kills instance one outright. Every client streaming through it drops at once, all reconnect in the same instant, and the thundering herd of simultaneous reconnections hammers the remaining seven instances while the dead one's replacement is still warming. The naive version turns a deploy into a self-inflicted traffic spike. Same eight instances, same new build. The difference is the drain.

§ VConnection to Prior Lessons

The 05-31 release-engineering lesson built the rollout orchestration — atomic moves, canary slices, the rollback surface. This lesson supplies the term that orchestration depends on: an instance that drains cleanly when the rollout tells it to step down. Canary slicing is meaningless if removing the canary drops its in-flight requests; the drain sequence is what makes "remove the canary" a safe operation rather than a small outage.

The 06-07 observability lesson built the degradation ladder and the readiness to see production failing in real time. The drain window is exactly where deploy-induced failures hide, because they are brief and self-healing and easy to dismiss as noise. The instrument that catches a bad drain is a request-success-rate panel sliced by deploy event: if success rate dips at every deploy, the drain contract is broken even though the service "works." The 06-14 performance-budget lesson taught the regression gate that refuses a merge that overspends a budget. The same gate guards the drain: a deploy that drops requests is a deploy that overspent the availability budget, and the pipeline should be able to fail it. The DevOps Handbook frames this whole posture as low-risk releases — the deploy made routine and safe enough to do constantly (p. 479) — and a release cannot be low-risk if every instance of it drops live traffic.

§ VIConnection to Today's Dev Lesson

The Ops contract is language-agnostic; today's Dev lesson makes it concrete in one runtime. The Node and Express lesson implements the five-beat drain in JavaScript: catching SIGTERM in the process, flipping a readiness flag that the health endpoint reads, sleeping past the propagation delay, calling server.close() to stop the listener while letting in-flight requests finish, and draining keep-alive sockets that would otherwise hold the close open. Where this lesson names the beats, the Dev lesson writes them — and shows the runtime-specific traps, like Express keep-alive connections that defeat a naive server.close() unless you handle them. Building Microservices sets the frame both lessons share: continuous delivery means the deploy is so routine it becomes invisible (Ch. on build pipelines and continuous delivery, p. 256), and invisible requires that no user ever feels a deploy happen.

§ VIIClosing

Zero-downtime is a contract with three signatories and five beats. The load balancer agrees to stop routing when an instance says it is not ready. The orchestrator agrees never to remove an old instance until a new one is warm. The process agrees to outlive its own termination signal long enough to drain. Catch the signal, flip readiness, wait out the routers, drain the in-flight work under a timeout, then exit. Reverse the warm-up for the new instance and interleave the two across the fleet. The user never sees the seam, which is the entire definition of a deploy done right.

Find the longest-lived connection your service holds — a WebSocket, a streaming response, a slow query. Trace what happens to it when an instance receives SIGTERM today. If the answer is "it dies," you have an outage on every deploy that you are paying for in reconnect storms and 502s. Write beat four first.

Paired lessons → Dev Polyglot-Dev/Web/2026-06-21-graceful-shutdown-... · Cert Cert-Prep/Anthropic/2026-06-21-claude-code-as-the-operators-harness-...