Inference Autoscaling and Capacity Governance for Multi-Agent Cognition

Queue-depth scaling, the cold-start tax, and the cost-per-token budget.

§ IFrame

A multi-agent fleet does not fail the way a web service fails. A web service under load returns a slow page. A cognition fleet under load returns a slow page that another agent is already waiting on, which makes a third agent wait, until the planner at the top of the chain times out and retries the whole tree. The queue you cannot see is the one that takes you down.

Yesterday's lesson built the un-promotion path: the way to pull a regressed model back to a known-good version without a rebuild. That lesson assumed the model was running on some number of replicas and said nothing about how that number is chosen. Today names the choosing. How many copies of each model the fleet runs, when it adds more, when it sheds work it cannot serve, and what every served token costs. This is capacity governance, and for an agent fleet it is a safety surface, not a billing footnote.

Examine the fleet as a set of queues. Every model endpoint has a line of requests in front of it. The length of that line, measured in requests waiting and in seconds-to-drain, is the truest signal of whether the fleet has enough compute. CPU utilization lies on a GPU-bound inference box; the GPU can sit at sixty percent while the request line grows without bound, because the bottleneck is memory bandwidth and batch scheduling, not arithmetic. Watch the line.

§ IIFoundations: Three Meters and a Floor

A cognition endpoint is governed by three meters and one floor. Name them; the mechanism follows.

The floor is the minimum warm count: the number of replicas held loaded even when no traffic arrives, paid for in idle compute, kept against the next burst.

Google's site-reliability practice names the discipline above these meters: a server must protect itself from overload before it protects its latency. Preventing Server Overload and Load Shedding and Graceful Degradation (Site Reliability Engineering, Part III, pp. 291–293) make the case plainly. A server that accepts every request and serves all of them slowly is worse than a server that accepts what it can serve and refuses the rest cleanly, because the first fails every caller at once while the second fails only the callers it turns away and keeps its promise to the rest. The refusal is the feature.

AI Engineering (Ch. 9, pp. 439–440) gives the meters their units. Inference performance is read in latency percentiles, not averages: the p99 time-to-first-token and p99 time-per-output-token, because the planner waiting on the tail is the planner that retries. Throughput is tokens per second across the batch. Batching more requests raises throughput and lowers cost-per-token, but it raises tail latency, because a request that joins a large batch waits for the whole batch to schedule (pp. 464–465). The throughput-latency trade is the central tension of every serving decision.

§ IIIMechanism: Scale on the Line, Not on the Box

Here is the rule worth keeping: scale on the line, not on the box. Drive the autoscaler from queue depth and queue age, not from CPU or GPU utilization. The mechanism has four parts, and each answers a question the others cannot.

Part one — the scaling signal

Publish one derived metric per endpoint: requests-in-queue divided by served-requests-per-second, which gives seconds-to-drain. When seconds-to-drain crosses the threshold you tolerate for that endpoint, add replicas. A planner endpoint with a chain waiting behind it tolerates perhaps two seconds of drain; a background summarization endpoint tolerates thirty. The threshold is per-endpoint because the cost of waiting is per-endpoint.

Part two — the warm floor against the cold-start tax

A model copy that is not resident pays the cold-start tax: the tens of seconds to pull weights, load them to GPU, and compile the first forward pass. If the autoscaler only adds replicas when the line is already long, every burst eats the tax at exactly the moment the fleet can least afford it. Keep a warm floor sized to the burst you expect, and pre-warm ahead of known load — a retraining sweep, a market-open spike, a batch of evaluations. Pay the tax early, on your schedule, not late, on the burst's.

Part three — load shedding when floor and ceiling both fail

Autoscaling has a ceiling: a budget cap, a quota, a hardware limit. When the line grows past what new replicas can drain in time, the fleet must shed. Shedding means refusing the lowest-priority work cleanly and immediately, with a typed signal the caller understands, so the caller backs off rather than retries into the same wall. A cognition fleet sheds by class: background enrichment before interactive planning, speculative branches before committed ones. The fleet that cannot say no serves everyone badly.

Part four — the cost-per-token budget as a governor

Every replica added has an hourly price; every token served has a marginal cost. Attach a budget to each endpoint and let it bound the autoscaler's ceiling. When an endpoint's projected spend for the day crosses its budget, the autoscaler stops adding capacity and the fleet sheds instead.

§ IVWorked Example: A Market-Open Burst Across a Three-Tier Fleet

Walk one burst through the mechanism. The fleet has three model tiers. A small fast model fields routing and classification. A mid model handles per-asset analysis. A large model runs the planner that composes the day's directional thesis. Each tier is a separate endpoint with its own queue, warm floor, and budget.

It is 08:25, five minutes before the cash-session open the fleet watches. Overnight the small model ran a warm floor of two, the mid of two, the large of one. No burst was in the line, so no autoscaling had fired since the prior evening.

At 08:25 a scheduled pre-warm fires, because market-open is known load and the fleet does not meet known load cold. The pre-warm raises the small floor to eight, the mid to six, the large to three, and pays the cold-start tax now, in the quiet, while the lines are empty. By 08:29 the new replicas are resident and the first forward pass on each is compiled.

At 08:30 the open hits. Routing requests flood the small model; its seconds-to-drain climbs to four against a two-second threshold. The autoscaler reads the line, not the box, and adds four more small replicas; they join in single-digit seconds because the floor was raised. The mid tier fills next as routed work fans out to per-asset analysis, scaling from six to ten.

The large planner endpoint is where governance earns its keep. Each planner call is expensive in tokens, and the budget for the large tier is set to a dollar ceiling for the open window. At 08:34 the planner line grows because every asset desk wants a fresh thesis at once. The autoscaler would add large replicas, but projected spend has reached the endpoint's budget. The valve closes. The fleet sheds: speculative thesis branches are refused with a typed at-capacity signal while the committed primary-thesis calls are served. The planners that matter get answered. The speculative ones get a clean no and retry after the open settles.

By 09:10 the open has digested. Drain times fall below threshold across all tiers. The autoscaler scales in, but holds the raised floor until the post-open echo passes, then steps down in stages so a second smaller spike does not re-pay the cold-start tax. By 10:00 the fleet is back to its overnight floors, the day's large-tier spend is logged against budget, and not one planner chain timed out. The shed speculative work cost nothing and blocked nothing.

The lesson of the burst is the rule stated three ways. Scale on the line. Pay the cold-start tax early. Let the budget shed what it cannot afford.

§ VConnection to Prior Lessons

This lesson is the capacity layer above an arc that built the rest of the lifecycle. The model-serving topology lesson (05-25) laid out how requests reach models across tiers; that topology is the set of queues this lesson now meters and scales. The continuous-training lesson (06-08) introduced the scheduled retraining sweep; a sweep is exactly the kind of known load the warm floor pre-warms against, so the two interlock — training schedules its compute, serving pre-warms ahead of it.

Yesterday's rollback lesson (06-15) is the closest neighbor. Rollback pulls a bad model back to a known-good version without a rebuild; today's autoscaling decides how many copies of whatever version is live the fleet runs. The two share a registry: the un-promotion path and the autoscaler both read the same record of what is deployed where. Roll back the version, and the autoscaler keeps scaling the line, which does not care which version drains it.

§ VIConnection to Today's Dev Lesson

Autoscaling adds and removes whole replicas; it is the horizontal control, and it is coarse. It cannot help a single replica that has accepted more concurrent requests than its GPU memory can batch. That finer control lives inside the replica, and today's Python lesson builds it: asyncio.Semaphore to bound how many requests one replica admits at once, a bounded asyncio.Queue to give backpressure when the bound is reached, and a token-bucket limiter to cap the cost-per-token budget at the process edge.

The autoscaler decides how many lines exist; the in-process primitives decide how long each line is allowed to grow before the replica itself says no. Read them together: horizontal scaling and per-replica admission control are the same load-shedding principle at two scales.

§ VIIClosing

The fleet's compute bill and the fleet's safety are the same decision read twice. Scale on the line, not on the box, because the box hides the queue that takes you down. Keep a warm floor and pre-warm against known load, because the cold-start tax is cheapest when you pay it in the quiet. Attach a budget to every endpoint and let it shed, because a fleet that cannot say no serves everyone badly when it matters most.

Examine your own fleet for the queue you are not watching. The endpoint with no drain metric is the endpoint that will surprise you. Name its line, meter it, and give it a floor and a budget before the next open finds the gap.