Inference Caching and the Cost-Reduction Surface for Multi-Agent Cognition — Prompt Caching, Semantic Caching, and the Staleness-versus-Savings Calculus

Day / pairMon — DevOps + Cognition (anchor Mon; W2/C2; Cognition AI+ML pair)
Lesson classOps — DevOps + Cognition (AI/ML)
Shipped2026-06-22T05:30Z — MD + HTML in-cycle

Inference Caching and the Cost-Reduction Surface for Multi-Agent Cognition — Prompt Caching, Semantic Caching, and the Staleness-versus-Savings Calculus

§I — Frame

Last Tuesday's lesson scaled the supply side. When the queue grew, the autoscaler added replicas; when scaling could not catch up, the shed policy dropped the lowest-value work. The whole apparatus answered one question: how does the fleet meet demand without melting down or overspending?

It never asked the prior question. How much of that demand should have reached a model at all?

A multi-agent fleet repeats itself constantly. The same system prompt rides in front of every call an agent makes. The same retrieval context gets re-sent on every turn of a conversation. Two users ask the same question an hour apart and the fleet pays full price twice. The orchestrator fans a task to four sub-agents that share a thousand-token preamble, and the preamble is processed four times. Every one of those repetitions is a token bill the fleet chose to pay for an answer it already had.

The named claim: caching is the lever that cuts inference demand before it reaches a model, and it pays in the same currency autoscaling spends. Autoscaling buys capacity to serve the calls. Caching removes the calls. The first is supply; the second is demand. A fleet that scales without caching is buying horsepower to haul a load it could have left at home.

Three primitives carry the lesson. The prompt cache reuses computation across calls that share a prefix. The semantic cache reuses whole answers across calls that mean the same thing. The staleness guard is the discipline that decides when a cached answer has gone wrong enough to throw away. Read them as one surface, because the third one governs the first two.

§II — The three primitives

The prompt cache reuses the model's internal work on a shared prefix. A transformer processing a 1,000-token system prompt builds a key-value representation of those tokens before it generates a single output token. If the next call carries the same 1,000-token prefix, that work is identical, and a prompt cache stores it so the second call skips straight to the new tokens. Huyen's inference-optimization chapter gives the arithmetic plainly: a 1,000-token system prompt across one million daily calls is roughly one billion repetitive input tokens processed every day, and a prompt cache spares almost all of it. The cache key is the prefix itself. The hit condition is an exact prefix match.

The semantic cache reuses the answer, not the computation. Where the prompt cache asks "have I processed these exact tokens," the semantic cache asks "have I already answered a question that means this." It embeds the incoming query, searches a vector index of prior query embeddings, and if the nearest neighbor sits above a similarity threshold, it returns the stored answer without calling the model at all. The prompt cache saves the input-processing cost. The semantic cache saves the entire call. It is also the riskier of the two, because "means the same thing" is a judgment, and a wrong judgment serves a real user a wrong answer.

The staleness guard is the discipline that bounds the risk. Newman's chapter on scaling states the rule every caching system eventually relearns: the hard part is not storing the value, it is knowing when the stored value has stopped being true. A cached answer is a snapshot of a world that keeps moving. The guard decides the snapshot's shelf life: how long an entry lives, what events expire it early, and how a wrong hit gets caught and evicted. Without the guard, a cache is a machine for serving yesterday's truth at today's prices.

The three compose into one rule. Cache the computation freely, cache the answer carefully, and never cache anything you cannot expire.

§III — Prompt caching: the cheap, safe win

Prompt caching is the win to take first because it carries almost no correctness risk. A prompt cache returns the same tokens the model would have produced; it only skips the redundant arithmetic of re-encoding a prefix the model has already seen. The output is bit-identical to the uncached path. The only cost is memory, because the stored key-value representation is large, and the only discipline is structuring prompts so the stable part comes first.

That ordering discipline is where fleets win or lose the prompt cache. The cache matches on prefix, so everything you want reused has to sit ahead of everything that varies. The system prompt goes first. The tool definitions go second. The retrieved context goes third if it is stable across the turn. The user's new message goes last. A fleet that splices the timestamp or a request ID into the top of the prompt has destroyed its own cache on every call, because the prefix never repeats. The cheapest optimization in the stack is free only to the fleet that orders its prompts to earn it.

Huyen's Chapter 10 places caching as a distinct architecture step, sitting between the orchestration layer and the model, precisely because it changes the cost profile of every component above it. Vendor APIs now expose this directly. Google's Gemini offers cached input tokens at a reduced rate, which means the fleet that structures prompts for cache reuse pays a published discount rather than building the cache itself. Tuesday's cost-per-token budget is the metric this primitive moves: the same calls, fewer billed input tokens, a lower median spend per request with no change to what the user receives.

For a multi-agent fleet the multiplier is the fan-out. When the orchestrator dispatches one task to four sub-agents that share a preamble, prompt caching turns four full encodings into one encoding and three cache hits. The cost reduction scales with the width of the fan-out, which is exactly the dimension a multi-agent system grows along as it matures.

§IV — Semantic caching: the larger win at the higher risk

Semantic caching is where the savings get large and the discipline gets serious. A semantic cache that hits returns an answer the model never recomputed, which means it skips not just the input encoding but the entire generation. On a fleet where many users ask overlapping questions — a support agent, a documentation agent, an internal knowledge agent, the hit rate can carry a real fraction of traffic, and each hit is a call the fleet did not make.

The mechanism borrows the retrieval machinery the fleet already runs for RAG. Embed the incoming query with the same embedding model used to index prior queries. Search the vector index for the nearest stored query. Read the cosine similarity of the top match. Above the threshold, serve the stored answer; below it, call the model and store the new query-answer pair for next time. Huyen's observability note is worth carrying here: real semantic-retrieval systems pair the cache with a reranker and with sampling, because the nearest neighbor by embedding distance is not always the right answer by meaning.

The threshold is the whole game. Set it loose and the cache hits often and serves wrong answers, because two questions that embed near each other can want genuinely different replies. "How do I cancel my subscription" and "how do I pause my subscription" sit close in embedding space and deserve different answers. Set the threshold tight and the cache rarely hits, and the savings evaporate. The threshold is not a constant to be guessed once. It is a number to be set against the eval pipeline from the 06-01 lesson: sample the cache's hits, score them against what the live model would have answered, and move the threshold until the wrong-hit rate sits under the contract the fleet will accept. The semantic cache without an eval behind it is a guess that compounds.

§V — The staleness-versus-savings calculus

Every cache trades freshness for cost. Newman names the tension directly: the more aggressively you cache, the more you save, and the staler your answers become. The calculus is the deliberate placement of the fleet on that curve, per class of work, rather than a single global setting applied to everything.

Three eviction disciplines do the work. Time-to-live is the floor: every entry carries an expiry, and a fleet should be able to state the maximum age of any answer it serves. A pricing question might tolerate an hour; a "what is our current incident status" question tolerates seconds, or should not be cached at all. Event-driven invalidation is the precision instrument: when the underlying truth changes, a document is updated, a model is rolled back, a policy is rewritten, the affected entries expire immediately rather than waiting out their clock. The 06-15 rollback lesson connects here directly: when the model version changes, every semantic-cache answer the prior version produced is suspect, and the rollback should flush the cache as part of the un-promotion path. Sampled validation is the safety net: a small fraction of cache hits are shadowed against a live model call, scored, and used to catch a threshold that has drifted wrong before users feel it.

The calculus also names what must never be cached. Anything personalized to a single user, unless the cache key includes the user identity. Anything that carries authority rather than information, a cached "yes, you may proceed" is a stale grant, and stale grants are how seams fail. Anything whose wrong answer costs more than the call it saved. The cache is a cost optimization, and a cost optimization that produces a wrong action has spent more than it saved.

§VI — Worked example

The documentation agent serves 200,000 calls a day. Its system prompt and tool definitions run 1,400 tokens and never change between deploys. Each call retrieves roughly 2,000 tokens of context and adds a user question averaging 80 tokens.

Prompt caching goes in first. The fleet reorders the prompt so the 1,400-token stable prefix leads, and routes through Gemini's cached-input pricing. The 1,400 tokens that were billed at full rate on all 200,000 calls now bill at the cached rate, cutting roughly 280 million daily input tokens to a fraction of their prior cost. The user sees an identical answer. The cost-per-token budget from Tuesday's lesson drops without a single replica added.

Semantic caching goes in second, behind an eval gate. The fleet finds that 31% of incoming questions are near-duplicates of prior questions, the same dozen onboarding questions, asked endlessly. It stands up a semantic cache with the threshold initially set conservatively at 0.95 cosine similarity, then runs the 06-01 eval pipeline against a week of shadowed hits. The eval shows the wrong-hit rate at 0.4% at threshold 0.92 and 2.1% at 0.88. The fleet sets 0.92, accepting a 0.4% wrong-hit rate against a measured 27% hit rate. Twenty-seven percent of calls now return without touching the model.

The staleness guard is wired before the cache serves a single user. Every entry carries a 6-hour TTL. The documentation-update webhook flushes the affected entries on every publish. The rollback runbook from 06-15 gains one line: flush the semantic cache on model pin, because the pinned version did not produce the cached answers. A 1% sample of hits shadows a live call nightly and re-scores the threshold.

The fleet now serves the same traffic. It bills cached input on the stable prefix, skips the model entirely on 27% of calls, and can state in writing the maximum age of any answer it serves. Autoscaling still meets the demand that remains. There is simply less of it.

§VII — Closing discipline

Autoscaling answers how to serve the calls. Caching answers whether to make them. The prompt cache is the cheap, safe win: identical output, paid in memory and prompt ordering. The semantic cache is the larger win at the higher risk: whole calls skipped, governed by a threshold the eval pipeline must set and keep honest. The staleness guard is the discipline that makes both safe: nothing is cached that cannot be expired, and nothing carrying authority is cached at all.

Scale the demand you cannot avoid. Cache the demand you can. Examine the difference well, per class of work, before the invoice does it for you.

Related

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura