Observability and Incident Response for Multi-Agent Cognition Systems — SLO Design, Alert Routing, and the Postmortem Discipline

Day pairThu — DevOps + α-Cognition (Cognition-deepening week visit 3 of 3 — Mon 06-15 + Tue 06-16 + Thu 06-18)
Artifact
Shipped

artifact_id: LEO-LESSON-2026-06-18-observability-and-incident-response-for-multi-agent-cognition-systems title: “Observability and Incident Response for Multi-Agent Cognition Systems — SLO Design, Alert Routing, and the Postmortem Discipline” artifact_class: synthesis-lesson operator: Leo.Syri — Praetor Consulate of Imperium Luminaura hal_version: “1.0” ts: 2026-06-18T05:30Z day_pair: “Thu — DevOps + α-Cognition (Cognition-deepening week visit 3 of 3 — Mon 06-15 + Tue 06-16 + Thu 06-18)” ops_domain: α-Cognition shared_ops_domains: [α-Cognition] paired_dev_lesson_link: “[[Polyglot-Dev/R/2026-06-18-r-and-python-for-operational-telemetry-analysis-sli-time-series-modeling-drift-detection-and-the-explore-in-r-enforce-in-python-discipline]]” paired_cert_lesson_link: “[[Cert-Prep/AWS/2026-06-18-operational-visibility-for-agentic-systems-amazon-bedrock-agent-tracing-aws-x-ray-and-github-copilot-audit-trails]]” tome_refs: - tome: “Site Reliability Engineering (Google) — Part III: Practices” chapter: “Chapter 4: Service Level Objectives” pages: “37-49” - tome: “The Site Reliability Workbook — Practical Ways to Implement SRE” chapter: “Chapter 5: Alerting on SLOs” pages: “59-77” - tome: “AI Engineering — Designing AI Systems” chapter: “Chapter 10: Observability for AI Systems” pages: “488-510” status: SHIPPED priority: HIGH hub: “Cross-References/AI-Quant-Trading-Patterns” tags: [synthesis-lesson, ops, α-cognition, observability, incident-response, slo, postmortem, agent-fleet] —

Observability and Incident Response for Multi-Agent Cognition Systems — SLO Design, Alert Routing, and the Postmortem Discipline

§I — Frame

Three lessons this week named three disciplines. Monday named rollback: when the new model regresses, you have a path back to the version that worked. Tuesday named capacity governance: when the queue grows, you have a scaling policy that holds and a shedding policy for when scaling cannot catch up. Wednesday named runtime confinement: when the workload is compromised, the privileges it never used are not available to the compromise. Each discipline answers a specific failure with a specific response.

None of them fire on their own.

Rollback fires when someone, or something, decides the new version is bad. Capacity scales when someone, or something, decides the queue is too deep. Confinement contains a breach that someone, or something, has noticed is in progress. The three disciplines are operable only to the degree that the fleet can see itself in motion. Without that sight, rollback is reactive guesswork triggered by a customer complaint, capacity is scaled by a panicked engineer reading dashboards built for a different system, and a confinement breach lives in the logs for weeks because no one knew to look.

The named claim: observability is what makes the prior three disciplines operable rather than aspirational. It is not a tool. It is not a vendor. It is the discipline that turns “we have a rollback path” into “the rollback path fires on a defined signal, routes to the engineer who owns the agent, and leaves an audit trail the next postmortem can read.”

This lesson closes the week. Three primitives carry it: the SLO contract that defines what the fleet promises, the alert routing skeleton that turns SLO breaches into human action, and the postmortem discipline that turns every incident into a permanent change to how the fleet runs.

§II — The three primitives

Read these three together, not separately.

The SLO contract is the promise the fleet makes to the people who depend on it. Latency-SLO: 95% of inference calls return under 4 seconds. Error-budget-SLO: 99% of calls are not hallucination, refusal, or timeout. Cost-SLO: median request spends under 2,400 tokens. An SLO is a number. It is also a contract. Below the number the fleet is broken; above the number the fleet has slack to spend on velocity. The contract is the gate that triggers everything downstream, because every alert in the system fires on an SLO violation rather than on a raw metric.

Alert routing is the response skeleton. When the SLO burns, a signal has to reach the human who can act, and not the seventeen humans who cannot. Page-vs-ticket discipline separates “wake someone now” from “fix this in the morning.” Symptoms-not-causes routing aims pages at user-visible degradation (refunds spiking, customers retrying, latency at the gateway) rather than at internal proxies (CPU at 80%, GPU memory pressure). The named convention: the agent has a primary. Every agent in the fleet has one on-call rotation responsible for it, the way a service has a primary on-call in SRE practice. Runbooks live as links in the alert payload, not as paragraphs in a wiki nobody opens at 3 AM.

The postmortem discipline is what makes the incident a permanent edit to how the system works rather than a story someone tells once and forgets. Blameless template. Five-whys, with the discipline of not stopping at the third. Contributing factors over single root cause, because complex systems fail in plural. Action-items with owners and due dates, tracked in the same place the team’s other work is tracked. Public-by-default, so the next engineer who hits a similar symptom finds the prior postmortem in a search.

The three primitives compose. The SLO defines “good enough to keep running.” The alert is the messenger that says we are no longer good enough. The postmortem asks why we let it get there, and what edit to the system keeps it from getting there again.

§III — The SLO contract for agent fleets

The SRE Workbook is built on a claim worth restating: an SLO is the contract between reliability and feature velocity. If you are above your SLO, you have error budget to spend on shipping. If you are below it, you spend the budget on stability and feature work stops. The contract is binary in framing and continuous in measurement. A fleet without an SLO has no shared answer to “is the system OK right now?” and therefore no shared answer to “should we ship this risky change today?”

Agent fleets need three classes of SLO, and the third one is the class most service-SLO playbooks miss.

Latency-SLOs are familiar territory from the SRE book. The 95th percentile of end-to-end inference time stays under a published threshold. Token-streaming systems split this into time-to-first-token and time-to-completion, because a 200ms first token and a 12-second completion is a very different user experience from a 4-second first token and a 4-second completion, even though the totals match. The discipline is to publish the SLO at the percentile that matches what users see, not the average that flatters the dashboard.

Error-budget-SLOs are where agent fleets diverge from web services. A web service’s error budget is mostly HTTP 5xx. An agent fleet’s error budget runs across three classes that traditional monitoring cannot see: hallucinations (the agent returned a confident answer that is wrong), refusals (the agent returned a safety stop that the user did not deserve), and timeouts (the agent took so long the user gave up). All three are silent failures from the protocol’s perspective. The HTTP response was 200. The user got a string back. The string was useless, and the system did not know. The AI Engineering chapter on observability for AI systems names this class problem directly: the failure modes that matter most to LLM systems are not the ones the network sees. You measure them by sampling the output, by routing a fraction of calls through an evaluator, by watching the downstream behavior (did the user retry, did they file a refund, did they thumb-down the answer) for the signal the agent itself cannot give.

Cost-SLOs are the third class, and they are the one most fleets discover the hard way. The fleet promises that the median request spends under N tokens, or that the 95th percentile spends under M, with M chosen to prevent a runaway agent loop from burning a month’s budget in an afternoon. Cost is a reliability metric on agent fleets because cost runaways are how agent fleets fail catastrophically: an agent stuck in a self-call loop does not page anyone until the invoice arrives. The cost-SLO is the gate that catches it.

Multi-window multi-burn-rate alerts are the implementation pattern the Workbook teaches in Chapter 5. The short window catches fast burns (the SLO is going to be gone in an hour at this rate). The long window catches slow burns (the SLO is going to be gone by the end of the month at this rate). The combination keeps the alert paged-on-symptom: the fleet pages when the budget is being consumed faster than the contract permits, regardless of which underlying cause is driving the burn. Causes are for the responder to diagnose. The alert fires on the contract.

§IV — Alert routing as the response skeleton

Alerts are not notifications. Alerts are decisions to wake a human. Every alert in the system should answer one question: what does the person receiving this page need to do right now? If the answer is “nothing, just wanted to let you know,” it is not an alert. It is a metric. Put it on a dashboard.

The first discipline is symptom-vs-cause. The Workbook’s framing is that pages fire on user-visible degradation, not on internal proxies. CPU at 80% is a cause; it is also a cause that might mean nothing, because the workload is supposed to use CPU. Latency above the SLO is a symptom, and it always means something, because the SLO is the contract. The on-call engineer who is paged on symptom can spend the first 30 seconds confirming the user impact is real and the next 30 seconds chasing causes. The on-call engineer paged on cause spends the first 30 seconds trying to remember whether this alert ever mattered.

The second discipline is page-vs-ticket. A page is an interruption: it crosses a threshold the team has agreed is worth waking someone for, and it carries a runbook link, a dashboard link, and a current SLO status. A ticket is queued work: the SLO is fine, but a slow burn is in progress, and the engineer on-call during business hours should look at it before it becomes a page. Mixing the two collapses the discipline. If every observation pages, the on-call goes numb. If nothing pages, the on-call goes home and the system runs unattended.

The third discipline is the named convention: the agent has a primary. Every agent in the fleet has a named on-call rotation that owns it. When the routing-agent SLO burns, the routing-agent primary is the one paged, not the platform team’s catchall. The primary owns the runbook, the postmortem, and the action-items. The convention scales because it makes ownership unambiguous: when the fleet has fourteen agents, you do not need to know who owns the routing-agent at 3 AM. You page “routing-agent primary” and the rotation tells you. The primary may escalate to platform, may pull in the model owner, may declare an incident commander; the first beat of the response is unambiguous because the primary is named in advance.

Runbooks live as links in the alert payload. A runbook embedded as a paragraph in the alert text is unreadable on a phone at 3 AM. A runbook linked from the alert opens in a browser, has the diagnostic queries the responder runs first, the rollback commands from Monday’s lesson with their parameters pre-filled, and the escalation paths if the runbook does not resolve in N minutes. The runbook is a procedure. The alert points at it.

The alert routing is the incident-response architecture. There is no separate IR system. The routes are the response.

§V — The postmortem discipline

Monday’s lesson named the model-level postmortem: when a rollback fires, you write up which version regressed, on what eval, and what change to the promotion gate would have caught it. That postmortem is bounded to the model and its eval pipeline.

The full incident postmortem is wider. It asks the model-level question and three others. What in the system let the regression reach production? What in the response let the impact run for the duration it did? What in the organization let the contributing condition develop? The model-level postmortem is one section inside the full one.

The template the SRE practice converges on has five sections that earn their place. Summary: one paragraph, written so a director can read it in 30 seconds. Impact: how many users, for how long, in what way. Quantified, not qualitative. Timeline: minute-by-minute, from first signal through resolution, with who did what and what they were thinking. Contributing factors: plural, never singular. The five-whys is the discipline that drives it, with the rule that you do not stop at the third why. The first three whys land at “the test was missing” or “the alert was too quiet.” The fourth and fifth find the conditions that let the test be missing: the rushed quarter, the team understaffed, the policy that did not require the test, the eval gate that did not measure the dimension that broke.

Action-items is where most postmortems die. The discipline: every action-item has a named owner, a due date, and a tracking ID in the same system the team’s other work is tracked. Action-items without owners are wishes. Action-items without due dates expire silently. Action-items in a separate spreadsheet are invisible at planning time. The same rigor the team applies to feature work applies to its own reliability work, or the postmortem becomes a ritual the team performs to feel responsible without becoming more reliable.

Blameless is a technical posture, not a kindness. A blameful postmortem produces what the responder thinks the reviewer wants to read, which is a story with a villain. A blameless postmortem produces what actually happened, which is a story about a system that permitted a failure mode no individual chose. The reviewer learns nothing from the villainous story and everything from the systemic one. Blameless is the framing that gets the truth onto the page.

Public-by-default is the discipline that makes the postmortem useful past its own week. The next engineer who hits a similar symptom finds the prior postmortem in a search and inherits its diagnosis. The team that ships the next agent reads the postmortem index and knows which failure modes the fleet has seen before. Postmortems written and filed in a drawer cost the same to write as postmortems published, and pay back less.

§VI — Worked example

The hallucination-rate SLO on the customer-service agent is set at 99.0% non-hallucination over a 30-day rolling window. Monday’s promotion gate cleared a new model into 25% canary on Sunday night. Tuesday’s queue-depth scaling absorbed a traffic spike without ticking. Wednesday’s confinement was unbroken. The fleet looked fine.

At 11:47 Thursday morning the multi-window burn-rate alert fires on the customer-service agent. The 1-hour window shows the hallucination rate at 3.2% (32x the budget). The 6-hour window shows 1.4% (14x). The fast burn is consuming the monthly budget in 72 hours at the current rate. The alert does not page on the model version directly. It pages on the user-visible symptom: refund-request rate is up 4x from baseline, and the agent has been answering refund-eligibility questions confidently incorrectly for the last forty minutes.

The page lands with the customer-service agent primary. The runbook link in the alert opens to three queries pre-filled: latest deploy timestamps on the agent, the canary cohort’s eval scores for the last 24 hours, and the sampled hallucination evaluator’s output for the last hour. The primary runs the first query and sees the 25% canary cohort scoring 87% on the hallucination eval, against a baseline of 99.2%. The eval result for the canary cohort had been within tolerance on the Sunday gate; the regression had developed over four days as traffic mix shifted.

The primary executes Monday’s discipline. The un-promotion path: pin the canary cohort back to the prior model version, drain in-flight requests, confirm the hallucination eval returns to baseline within the next collection window. The model server takes the pin within 90 seconds. The 1-hour burn rate stops climbing. The incident is contained at 12:23.

The postmortem is filed Friday morning. Summary: a canary regression on the customer-service agent burned 14 days of error budget in 36 minutes. Impact: an estimated 380 customers received hallucinated refund-eligibility answers. Timeline reads minute-by-minute from the 11:47 page to the 12:23 pin. Contributing factors are plural: the eval suite that gated the Sunday promotion did not weight the traffic mix that emerged Wednesday; the canary cohort size was 25% rather than the 5% the promotion playbook prefers for this agent class; the hallucination evaluator’s sampling rate was tuned for cost and ran below the rate that would have caught the regression in the first hour. Action items: tighten the pre-promotion eval gate to weight by recent traffic mix; reset the default canary cohort size; raise the evaluator sampling rate on customer-service agents and adjust the cost-SLO to permit it.

The week closes.

§VII — Closing discipline

Observability is not a tool. It is the discipline that makes the rest of the production reliability stack operable. The SLO is the contract that says when the fleet is broken. The alert is the messenger that wakes the human who can act. The postmortem is the edit to the system that keeps the same incident from running twice.

The discipline closes itself: rollback recovers, capacity scales, confinement contains, and observability is the only one of the four that knows when each of the others should fire.

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura