Context Window Budget Management for Multi-Agent Cognition Systems — Token Accounting, the Summarization Gate, and the Priority-Truncation Discipline
§I — Frame
The inference caching lesson cut the calls the fleet did not need to make. This lesson cuts the cost of the calls it must make.
A context window is a fixed rectangle. Every token inside it costs money at write-time and latency at read-time. In a multi-turn conversation the rectangle fills from the top: system prompt, tool definitions, prior turns, retrieved documents, then the new message. In a multi-agent system, several agents fill their own rectangles simultaneously, each accumulating history at its own rate. The rectangle does not grow with demand. When it fills, the model errors or the calling code panics.
Three disciplines keep the context rectangle from filling against the operator's will. Token accounting knows, at every call site, how many tokens are already committed and how many remain. The summarization gate compresses old turns into a compact summary before the window fills, trading fidelity for headroom. Priority-truncation is a ranked eviction order that decides which content leaves first when compression alone is not enough. Together they define the context budget.
Huyen's Chapter 9 frames inference optimization as two separate surfaces: input and output. Input optimization — the context budget — is addressed before output optimization, because input is paid before the model generates a single token (AI Engineering, Ch. 9, pp. 434–435). The accounting comes first.
§II — Foundations
A transformer model processes the entire context window at once before generating the first output token. The cost is roughly proportional to the square of the input length for the attention mechanism, though serving implementations vendors ship are more linear at scale than the raw math suggests. What holds without qualification: every token in the input is paid for, every time, on every call. A 4,000-token context costs more than a 1,000-token context. The margin is not small.
Multi-agent systems make this worse by default. A conversation agent accumulates turns: user message, assistant reply, user message, assistant reply, across a session that may run minutes or hours. If the code naively appends every turn, the context grows linearly with session length. On the tenth turn, the system is paying for nine prior turns on top of the system prompt and retrieved documents. Many of those nine turns are no longer relevant to the current question. The cost is paid again anyway.
The retrieval step from a RAG system adds another layer. Each call retrieves N chunks from the vector store and injects them into the context. If retrieval is not pinned to the current turn, old retrieved chunks accumulate alongside conversation turns. The context grows from two directions at once.
The budget disciplines exist because "just throw more context in" is not a cost model. A 200,000-token context window does not cost the same as a 4,000-token one, and a fleet at scale pays the difference on every call, at the volume of its entire user base, every day.
§III — The Three Disciplines
Before assembling the context for a call, count what is already committed. System prompt: counted at startup. Tool definitions: counted on load. Conversation history: counted incrementally. Retrieved documents: counted at retrieval time. The sum is the committed spend; the remainder is the budget for the new message and any additional retrieval. The discipline is about placing the counter correctly, not about the arithmetic.
When the committed token count crosses a threshold — typically 70 to 80 percent of the window limit — the gate fires. It takes the oldest N conversation turns and sends them to the model for compression. The model returns a compact summary; the raw turns are replaced with it. Two decisions determine whether the gate works: the threshold (set at 70 percent to leave room for the summary itself) and the summarizer prompt (must preserve concrete facts, decisions made, and open questions). Huyen identifies keeping input context small as the first lever in inference cost management, before quantization or hardware.
When the gate cannot recover enough headroom, truncation picks what leaves. A practical ranking from lowest to highest priority: oldest conversation turns; retrieved documents not cited in recent turns; retrieved documents with expired recency scores; completed tool-call pairs. System prompt and pinned content rank highest and never truncate. The eviction order should be tuned against the eval pipeline — not set by convention.
§IV — Worked Example
A customer support agent handles sessions averaging 12 conversation turns. System prompt: 800 tokens. Each retrieval pull: 5 documents at 400 tokens each. Context window: 32,000 tokens. At turn 12, the context holds roughly 7,600 tokens. Well within range. So where does the overflow come from in production?
The agent also receives escalation sessions — conversations handed off mid-session from another agent. The transferred history arrives pre-filled: 40 turns averaging 300 tokens each. That is 12,000 tokens before the support agent adds anything. Add system prompt, tool definitions, and documents: the window sits at 16,000 tokens on the first call. By turn 8 of the transferred session, it reaches 18,400 tokens. By turn 20, overflow.
The fix applies all three disciplines in sequence. The accounting layer detects the risk at handoff time: the transferred history is counted before the session begins. If the count exceeds 70 percent of the window, the summarization gate fires against the transferred history before the first support-agent turn. The compressed summary replaces the 40-turn transcript. If the summary itself pushes past threshold, priority-truncation removes the oldest transferred turns. The agent begins with a window that has headroom.
The session then proceeds under normal accounting. The gate fires at 70 percent; older turns compress again when needed. The agent never errors on a full window because the budget disciplines fire before the window fills.
§V — Connection to Prior Lessons
The inference caching lesson (06-22) cut calls the fleet had already answered. This lesson cuts the cost of calls the fleet must make fresh. Together they define the two levers on the input side of the inference cost surface: remove the call entirely, or reduce what the call costs when it must proceed.
The autoscaling lesson (06-16) showed that latency under load depends on queue depth and replica count. A smaller context per call means faster processing time per call, which means the same replica count clears the queue faster. Budget management is a latency optimization that shows up on the cost line.
The rollback lesson (06-15) connects here in one specific way: when the model version changes, summarization behavior may change. A summary written by model version N may not read as intended by version N+1. The rollback runbook should include a check: does the summarization prompt produce equivalent-quality output on the new model before the promotion gate opens?
§VI — Connection to Today's Dev Lesson
The Go lesson implements the token accounting layer as an atomic counter and the gate check as a middleware function wrapping the LLM call site. The mechanism: sync/atomic.Int64 for the per-session counter, incremented at each turn, checked against the budget threshold before context is assembled. The middleware pattern means the gate fires without changing logic at any individual call site. Go's context.Context carries the per-request budget through the call chain, so sub-agents in a fan-out receive their share of the remaining budget rather than each claiming the full window independently.
§VII — Closing
The context window is the invoice that arrives before the call. The operator who reads it before sending pays once. The operator who reads it only when the error arrives pays twice.
Count what goes in. Set the gate before the rectangle fills. Rank the evictions before the need arises to make them.
Examine well.
Related
- Prior arc: Inference Caching and the Cost-Reduction Surface (2026-06-22) — caching cut repeated calls; this lesson cuts context per remaining call
- Pair hub: α-Cognition (AI + ML synthesis pair)
- Grounding tome: AI Engineering — Chip Huyen (Ch. 9 Inference Optimization, pp. 434–460)