Baseline Profiling, Statistical Deviation Gates, and the Alert-to-Isolate Pipeline
The audit log does not lie. Yesterday's lesson built the RBAC schema that defines who may call what, and the Kubernetes audit pipeline that records every API call a service account makes against the control plane. The schema permits or denies at the moment of request. The audit log archives the record. Together they answer: who did what, and were they authorized?
Anomaly detection answers a different question: is what they are doing what they typically do?
RBAC is a permission gate. It is static: a ClusterRole binding says yes at the moment of request regardless of whether that account has made one such request this week or ten thousand. The gate does not accumulate context. Anomaly detection is a behavioral gate. It says: this request is within permission, and the audit log will record it faithfully, but the pattern of requests this hour does not match the pattern we measured during thirty days of known-good operation. That mismatch is the signal.
The measured behavioral envelope during known-good operation. Call volume, resource fan-out, namespace crossing rate, time-of-day clustering — profiled per service account over a two-week training window.
The statistical rule that converts deviation from the contract into a signal. Mean ± N × sigma per dimension, combined into an alert score. Score ≥ 2 triggers the pipeline; score = 4 skips operator review.
Three beats: emit the structured alert to SIEM, apply the quarantine label to restrict east-west traffic, page the on-call operator. For score = 4, token revocation fires automatically.
The DevOps Handbook's treatment of the Second Way — creating fast feedback from production behavior — names the same three movements at the system level: instrument, measure, respond (Kim, Humble, Willis, Forsgren, pp308–311). Applied here to security: the audit stream is the telemetry, the baseline contract is the SLO-equivalent, and the alert-to-isolate pipeline is the remediation path.
The audit pipeline built in the June 24 lesson writes every API call to a structured JSON stream: the requesting service account, resource kind, verb, namespace, resource name, and timestamp. Fluent Bit routes that stream to a SIEM or log aggregation system.
Each service account has a characteristic pattern: which verbs it calls (list, get, watch, create, delete), which resource kinds it touches, which namespaces it crosses, and at what rate. A data-processing agent lists and watches frequently; a model-serving agent mostly handles get requests on its own Pod and ConfigMap; an orchestrator creates and deletes sub-agent Pods. These patterns are stable across healthy operation. Newman's security chapter in Building Microservices makes this point at the service-mesh level: static access control is necessary but not sufficient, because a compromised credential operates within granted permissions while doing damage (Newman, Ch11, pp435–440). The behavioral layer is what detects use of a legitimate credential by an unauthorized actor.
Four behavioral dimensions carry the most signal for a Kubernetes-deployed multi-agent fleet:
These four dimensions compose the behavioral fingerprint per service account. The baseline contract is the measured fingerprint over the training window.
The training window should be a minimum of two weeks of known-good operation after any recent deployment. For each service account, compute per 5-minute interval: call_volume, list_rate, fan_out, cross_ns_calls, and active_hours. Compute mean and standard deviation across all windows in the training period.
A baseline for vault-agent-sa might read: call_volume mean 18.4 sigma 4.2 · list_rate mean 0.12 sigma 0.04 · fan_out mean 3.1 sigma 0.8 · cross_ns mean 0.0 sigma 0.0 · active hours 06:00–22:00 CDT. Store these parameters in a config keyed by service account name. Refresh weekly or post-deployment when known-good operation is confirmed.
The gate applies Shewhart control-chart logic: a measurement is anomalous when it falls outside mean ± N × sigma, where N is a configurable sensitivity multiplier. N = 3 covers 99.73% of normally-distributed steady-state observations; N = 2 is appropriate for high-security environments where false positives are preferable to missed detections.
Rather than applying the gate to each metric independently — which fires too often on natural correlation — use an alert score: increment by 1 for each metric crossing its gate; alert when the score reaches a threshold of 2 or more simultaneous breaches.
Baseline dataclass holds mean, sigma, and the N-sigma multiplier. The breached method returns true when the absolute deviation exceeds the threshold. The anomaly_score function counts breached dimensions across a full observation dict.
from dataclasses import dataclass
@dataclass
class Baseline:
mean: float
sigma: float
n_sigma: float = 3.0
def breached(self, value: float) -> bool:
return abs(value - self.mean) > self.n_sigma * self.sigma
def anomaly_score(obs: dict, baselines: dict[str, Baseline]) -> int:
return sum(1 for key, baseline in baselines.items() if baseline.breached(obs[key]))When the alert score crosses the threshold, the pipeline runs three beats in sequence.
Beat 1: Emit. Write a structured log entry to the SIEM stream with severity, account name, observation window, score, and breached dimensions. This becomes evidence if the incident escalates.
Beat 2: Quarantine. For a score of 2 or 3, label the flagged Pods with quarantine=true. A pre-existing NetworkPolicy matches on this label and cuts all east-west traffic.
quarantine: "true" and blocks all ingress and egress. It must exist before the incident; the alert pipeline only applies the label. An empty ingress: [] and egress: [] means no traffic is permitted in either direction.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: quarantine-block
namespace: agent-runtime
spec:
podSelector:
matchLabels:
quarantine: "true"
policyTypes:
- Ingress
- Egress
ingress: []
egress: []Beat 3: Page. The SIEM alert fires a webhook routed to PagerDuty or Slack. The on-call operator sees score, breached dimensions, and observation window. They decide: lift quarantine (false positive), escalate (true positive), or defer.
For a score of 4 — all four dimensions breached simultaneously — the pipeline does not wait for operator review. It revokes the service account token directly.
kubectl delete secret \
"$(kubectl get sa vault-agent-sa -n agent-runtime \
-o jsonpath='{.secrets[0].name}')" \
-n agent-runtimeScenario. The data-processing agent dataproc-sa in namespace pipeline has a trained baseline: call_volume mean 22 sigma 5; fan_out mean 4 sigma 1; cross_ns mean 0; active hours 06:00–20:00.
At 02:47 UTC Thursday, the audit stream shows: 94 API calls in a 5-minute window, 31 distinct resource names across 3 namespaces, outside the scheduled hours.
Score = 4. The pipeline skips operator review and issues token revocation immediately. The alert fires to SIEM and the on-call channel simultaneously. The postmortem shows a service account token from a decommissioned Pod that was not cleaned up — the attacker used it for three minutes before the automated gate fired.
Three prior lessons in the β-Trust arc set the stage for this one. The June 24 RBAC and Audit Logging lesson built the audit pipeline that feeds the anomaly-detection input stream. Without the structured audit log, there is no behavioral record to profile against; the audit policy's RequestResponse verbosity for Secrets is specifically what makes fan-out rate measurable.
The June 17 Runtime Confinement lesson hardened the process layer. Detection is the net for what confinement does not catch. The two layers are complementary: capability dropping reduces the attack surface; anomaly detection catches the residual.
The June 10 Zero-Trust Networking lesson built the NetworkPolicy infrastructure that the quarantine beat relies on. The quarantine block works because a default-deny posture was established earlier. Adding a quarantine label to an already-isolated segment is fast and surgical; writing new policies under incident pressure introduces error.
The R + Python lesson that pairs with this one implements the statistical foundation at both ends of the explore-in-R-enforce-in-Python discipline. R's dplyr and ggplot2 handle the training-window data: computing rolling means, drawing control-chart limits visually, and identifying which dimensions carry signal before committing to a threshold. Python handles the production gate: the Baseline dataclass and anomaly_score function implement the same statistical rules at runtime, reading from the live audit stream.
Choosing N-sigma and which dimensions to gate on is a statistical question that benefits from exploratory visualization before hardcoding. Enforcing those choices in a running system is an engineering question that benefits from Python's type safety and async I/O. The disciplines separate cleanly; the R session feeds the Python config.
The RBAC schema and the audit log are the first and second layers of access discipline. They define what is allowed and record what happened. Anomaly detection is the third layer: it finds the gap between allowed and normal, and closes that gap before the attacker finds it first.
Build the baseline contract during known-good operation. Measure it honestly — spikes that happen legitimately must appear in the sigma, or the gate fires on routine events. Set the threshold at the sensitivity the environment warrants: high-security workloads at N=2; operational workloads at N=3. Wire the pipeline so that a score of 2 quarantines and a score of 4 revokes, without waiting for a human to notice.
The audit log already exists. The data is already flowing. The baseline contract is the only thing standing between an archived record and a live security signal. Build it before the incident, not during it.
Examine this well. The three beats of the pipeline — emit, restrict, page — form a single discipline. Each beat has a lower bound on response time that a human operator alone cannot match. Automate them.