Concurrent Fleet Automation in Python — the Budget, the Deadline, the Ledger

Friday taught what one step owes when it fails. Today the script has two hundred steps, and they all want to run at once.

A survey where the first refusal ends the interview is worthless. Collect every verdict; grade at the end.

§ IFrame

Friday's lesson taught what one step owes when it fails: retry it, degrade around it, or fail loud. Today the script has two hundred steps, and they all want to run at once.

The task is ordinary fleet work. Probe every service endpoint the fleet exposes, collect a health verdict from each, and produce one report. Run serially at ten seconds per probe, two hundred hosts cost thirty-three minutes, and the report describes a fleet that has already changed by the time the last probe lands. Run concurrently, the same sweep finishes inside a minute. Concurrency is not an optimization here. It is what makes the reading simultaneous enough to be one reading.

The naive version is one line: gather every probe and await the lot. That line hides three failures. Unbounded, the script opens two hundred connections in one instant and becomes a stampede that knocks over the very services it came to check. Undeadlined, one hung socket holds the whole report hostage. And ungraded, the first exception tears down the sweep with a traceback about one host and silence about the other hundred ninety-nine. The craft of concurrent automation is three disciplines applied together, and this lesson names them: the fan-out budget, the per-task deadline, and the partial-failure ledger.

§ IIFoundations — Three Disciplines for a Fan-Out

The fan-out budget. Concurrency is a quantity, and it must be chosen, never defaulted. A bound of ten in-flight probes protects both directions at once: the targets, which receive a survey instead of a stampede, and the prober, which holds ten sockets and ten response bodies instead of two hundred. Ramalho makes the point with his throttled downloader in the chapter that grounds this lesson: the semaphore is what separates a polite client from an accidental denial-of-service tool. Pick the number deliberately. A sweep of your own internal services can afford fifty; a sweep that touches a rate-limited third-party API may deserve three.

The per-task deadline. The deadline belongs to the task, never to the batch. Give the whole sweep sixty seconds and a single hung socket spends the budget for everyone; the tasks behind it die innocent. Give each probe its own five-second deadline and a hung socket costs exactly one row in the report, marked timeout, while the other probes proceed untouched. A deadline that travels with the task turns the worst network behavior, the silent hang, into an ordinary recorded outcome.

The partial-failure ledger. A sweep is a survey, and a survey where the first refusal ends the interview is worthless. Every probe must end in exactly one recorded outcome: ok, degraded, timeout, unreachable. No outcome may abort its siblings, and no outcome may vanish. The report is the full ledger, graded at the end. This is Friday's taxonomy applied at batch scale. Each row routes its own failure to retry, degrade, or fail loud; the batch decides its exit code only after every row is in.

Hold the inheritance from Friday on one more point: fan out only what is safe to run again. Two hundred health checks can be re-swept without harm. Two hundred payment postings cannot. The fan-out budget bounds how hard you hit the world, but idempotency decides whether you may hit it at all.

§ IIIMechanism — The Semaphore, the Timeout, and the Grade

The fan-out budget is a counter with a queue behind it. asyncio.Semaphore(10) holds ten permits; each probe acquires one on entry and releases it on exit, and the eleventh probe simply waits. The semaphore is an async context manager, so the acquire-release pair is a with-block the probe cannot exit past. That is the same shape as the spend governor from two weeks ago: a bound expressed as a scope, enforced on the way out no matter how the block ends.

The per-task deadline is asyncio.timeout(5.0), a context manager that cancels whatever is running inside it when the clock expires and surfaces the cancellation as TimeoutError at the block's edge. Because it wraps only the awaited request, the deadline covers the network and nothing else. Put the timeout inside the semaphore block, not outside it. A probe that waits forty seconds for a permit has not timed out; it is queued, which is the budget working as designed. Wrap the timeout around the semaphore instead and a busy sweep marks healthy hosts as dead purely because they stood in line.

Grading requires that exceptions become data. The probe catches its two expected failures narrowly, the deadline expiry and the transport error, and converts each into a ledger row. Anything else, a bug in the probe itself, flies loud and kills the sweep, exactly as Friday's discipline demands: a typo is not a fleet condition, and no report should be published over one. Ramalho's downloader chapter walks this same progression, sequential script to throttled concurrent client with per-item error accounting, and lands on the identical rule: handle the failures you named, let the rest propagate.

§ IVWorked Example — The Two-Hundred-Host Sweep

The whole discipline fits in two functions. First, the probe: one host in, one ledger row out, permits and deadlines applied in the right order, expected failures converted, unexpected ones propagated.

import asyncio
import httpx

SEM = asyncio.Semaphore(10)

async def probe(client, host):
    async with SEM:
        try:
            async with asyncio.timeout(5.0):
                resp = await client.get(f"https://{host}/healthz")
        except TimeoutError:
            return {"host": host, "outcome": "timeout", "detail": "5.0s deadline"}
        except httpx.TransportError as err:
            return {"host": host, "outcome": "unreachable", "detail": str(err)}
    ok = resp.status_code == 200
    return {"host": host, "outcome": "ok" if ok else "degraded",
            "detail": str(resp.status_code)}

Read the ordering. The semaphore is acquired first, so at most ten probes hold sockets. The timeout wraps only the request, so queue time is never billed as network time. TimeoutError and httpx.TransportError are the two failures a healthy fleet still produces daily, and each becomes a row. A KeyError from a typo in this function is not caught anywhere, and that is correct.

Second, the sweep: fan out, collect every row, grade the whole.

async def sweep(hosts, essential):
    async with httpx.AsyncClient() as client:
        rows = await asyncio.gather(*(probe(client, h) for h in hosts))

    tally = {}
    for row in rows:
        tally.setdefault(row["outcome"], []).append(row["host"])

    bad_essential = [r["host"] for r in rows
                     if r["outcome"] != "ok" and r["host"] in essential]
    if bad_essential:
        raise SystemExit(f"essential hosts failing: {bad_essential}")
    return tally

gather is safe here precisely because probe never raises an expected failure; every awaited coroutine returns a row, so the sweep always receives two hundred rows. The tally is the quorum-reducer shape from the consensus lesson three weeks ago, a fold over outcomes. Then the grade: failures on essential hosts fail loud with a non-zero exit so the scheduler pages someone, while failures on non-essential hosts stay in the ledger as recorded degradation. One sweep, three responses, every host accounted for.

The number to watch in production is the timeout column. A steady two or three timeouts per sweep is network weather. Forty timeouts on a sweep that ran clean yesterday is a finding, and the ledger is what makes the difference visible, because a sweep that died at the first hung socket would have reported nothing at all.

§ VConnection to Prior Lessons

Friday's lesson decided what one failed step deserves; this lesson runs two hundred of those decisions concurrently and adds the batch-level judgment neither step could make alone. The spend governor from the fourteenth built a with-block whose exit logic cannot be bypassed; the semaphore and the timeout are both that same protocol, which is why the probe reads as three nested scopes rather than a page of bookkeeping. And the Counter-based quorum reducer from the seventh is the tally in miniature: many independent verdicts folded into one decision. The shapes repeat because the problem repeats. Bound a resource, scope a risk, fold the outcomes.

§ VIConnection to Today's Dev and Cert Lessons

The Dev lesson goes beneath this one, into the machinery the sweep deliberately avoided: asyncio.TaskGroup, the scope that cancels every sibling when one task fails and reports the failures as an ExceptionGroup. The sweep wanted the opposite posture, every task runs to its own verdict, which is why it converted failures to rows before they could trigger anyone's cancellation. Knowing both postures, and choosing, is the actual skill. The Cert lesson lifts the same shape into the managed tier: a Vertex AI Pipeline is a fan-out you describe as a DAG and hand to Google to run, with the per-step retry policy and caching standing in for the budget and the ledger. Today's trio is one idea at three altitudes: many steps, bounded, deadlined, graded.

§ VIIClosing

A concurrent sweep owes three things it cannot borrow from its steps: a fan-out budget chosen on purpose, a deadline that travels with each task, and a ledger in which every task ends exactly once. The failure mode is never that a host was down. It is a sweep that stampeded its own fleet, or hung on one socket, or reported one traceback instead of two hundred verdicts.

Take an automation you run against more than ten targets. Find its concurrency bound, its per-task deadline, and what happens to task one hundred ninety-nine when task three raises. If any of the three answers is "there isn't one," you have found the next outage's shape before it arrives.

Paired lessons → Dev Polyglot-Dev/Python/2026-07-27-pythons-structured-concurrency-... · Cert Cert-Prep/Google/2026-07-27-pmle-vertex-ai-pipelines-...