R and Python for Operational Telemetry Analysis — SLI Time-Series Modeling, Drift Detection, and the Explore-in-R-Enforce-in-Python Discipline

Day pairThu — Data Science tier (R + Python; second R lesson; refracts the day's α-Cognition observability spine through the data-science language tier)
Artifact
Shipped

artifact_id: LEO-LESSON-2026-06-18-r-and-python-for-operational-telemetry-analysis title: “R and Python for Operational Telemetry Analysis — SLI Time-Series Modeling, Drift Detection, and the Explore-in-R-Enforce-in-Python Discipline” artifact_class: synthesis-lesson operator: Leo.Syri — Praetor Consulate of Imperium Luminaura hal_version: “1.0” ts: 2026-06-18T05:30Z day_pair: “Thu — Data Science tier (R + Python; second R lesson; refracts the day’s α-Cognition observability spine through the data-science language tier)” language: R + Python language_tier: Data Science paired_ops_lesson_link: “[[α-Cognition/Synthesis-Lessons/2026-06-18-observability-and-incident-response-for-multi-agent-cognition-systems-slo-design-alert-routing-and-the-postmortem-discipline]]” paired_cert_lesson_link: “[[Cert-Prep/AWS/2026-06-18-operational-visibility-for-agentic-systems-amazon-bedrock-agent-tracing-aws-x-ray-and-github-copilot-audit-trails]]” tome_refs: - tome: “R for Data Science (Wickham + Grolemund)” chapter: “Chapter 7: Exploratory Data Analysis” pages: “82-103” - tome: “R for Data Science (Wickham + Grolemund)” chapter: “Chapter 16: Dates and Times” pages: “237-258” - tome: “Python for DevOps” chapter: “Chapter 12: Monitoring and Logging” pages: “335-360” - tome: “AI Engineering — Designing AI Systems” chapter: “Chapter 10: Observability for AI Systems” pages: “488-510” status: SHIPPED priority: HIGH hub: “Cross-References/AI-Quant-Trading-Patterns” tags: [synthesis-lesson, dev, polyglot-dev, R, Python, data-science-tier, telemetry, sli, drift-detection, observability] —

R and Python for Operational Telemetry Analysis

SLI time-series modeling, drift detection, and the explore-in-R-enforce-in-Python discipline.

§I — Frame

Yesterday’s Ops lesson named three primitives for keeping a multi-agent cognition system honest under load: the SLO contract that turns a promise to users into a number, the alert routing that wakes the right human on the right minute, and the postmortem that converts a bad night into a permanent fix. Each of those primitives rests on a measurement, and the measurement is where today’s Dev lesson opens.

Telemetry from an agent stack arrives as time-series. Per-request latency, per-response error class, per-call cost, per-tool retry count: every one of them a column with a timestamp. The contract said p99 latency below eight hundred milliseconds; this lesson computes that p99 from raw logs, asks whether the underlying distribution is the one the SLO drafter assumed, and writes the gate that catches the moment the distribution starts to drift.

The split is the same one the 2026-06-11 lesson coined on chain data, applied now to operations: explore in R, enforce in Python. R for the distributional shape question that decides whether the SLI definition even fits the system. Python for the live gate that decides, every minute, whether the SLI is currently burning. Two languages. One feedback loop closed around the agent estate.

§II — R for SLI Exploration

The first move belongs to R because the first question is shape. An SLO drafter who has not looked at the latency histogram is guessing at the percentile threshold, and the guess is almost always wrong in a direction the customer pays for. Wickham and Grolemund spend Chapter 7 of R for Data Science on exactly this motion: load the table, count the unusual values, plot the marginal distributions, ask what changes when you condition on a covariate (pp. 82-103). For telemetry, the covariates that matter most are model version, agent class, and tool path, because each of those is a release surface where the operator can intervene.

Start with the canonical interrogation. A CSV of one week of inference requests lands as a tibble; the per-version latency density tells the drafter whether the SLO is well-posed.

library(tidyverse)

telemetry <- read_csv("inference_telemetry.csv",
  col_types = cols(
    ts            = col_datetime(),
    model_version = col_character(),
    agent_class   = col_character(),
    latency_ms    = col_double(),
    cost_usd      = col_double(),
    error_class   = col_character()
  ))

telemetry |>
  filter(is.na(error_class)) |>
  ggplot(aes(latency_ms, fill = model_version)) +
  geom_density(alpha = 0.35) +
  scale_x_log10() +
  facet_wrap(vars(agent_class), scales = "free_y")

Three things in this plot do work no row count can do. The log-scale x-axis turns a heavy-tailed distribution into something the eye can read. The density overlay shows where two model versions diverge, which is the single most useful piece of information when the upgrade is recent. The facet by agent class catches the case where one agent path bears almost all of the tail and the others are clean, which decides whether the SLO should be one threshold or several.

The shape question almost always reveals one of three patterns. Long-tailed but unimodal means the percentile threshold is the right shape and the only debate is the number. Bimodal means there are two populations inside one SLI, which is the operator’s signal to either split the SLI by class or to investigate why one class is silently routing through a slower path. A shelf at the timeout boundary means the SLI is masking failures as slow successes, which invalidates the SLO until the timeout policy is fixed.

A fourth pattern shows up rarely and matters when it does. If the density has a sharp spike at one or two integer-millisecond values, the column is not measuring what its name claims; the spike means a cached path is returning the same nominal latency for a wide class of requests, and the SLI is measuring cache hit rate dressed up as latency. The R session catches this in seconds; a production gate that only computed a p99 would never see it.

The second R motion takes the same tibble and rolls it forward in time. R for Data Science Chapter 16 covers lubridate for the date arithmetic this needs (pp. 237-258); the verbs compose with dplyr so cleanly that an analyst can chart an hour-by-hour error rate before the kettle boils.

library(lubridate)

telemetry |>
  mutate(hour = floor_date(ts, "1 hour"),
         is_err = !is.na(error_class)) |>
  group_by(hour, model_version) |>
  summarise(error_rate = mean(is_err),
            requests   = n(),
            .groups    = "drop") |>
  ggplot(aes(hour, error_rate, colour = model_version)) +
  geom_line(linewidth = 0.4) +
  geom_smooth(method = "loess", span = 0.2, se = FALSE)

The smoother is the visual translation of a rolling window: it turns a noisy hour-line into the trend the human eye reads as getting worse. The analyst who sees the line bend upward in the past forty-eight hours has the SLO conversation before the SLO burns; the analyst who only has a single error-rate number waits for the page.

Name the discipline that governs this section, because the line will return. Look at the shape before you set the gate. An SLO threshold chosen against a histogram you have never seen is an opinion; an SLO threshold chosen against a histogram you have plotted, faceted, and conditioned on the release surface is a measurement. R earns its place in the curriculum on Thursdays because no other working dialect gives the analyst that motion this cheaply.

There is one more exploratory motion that pays back forever on operational data: the cost density. Per-request cost in an agent stack is the column the operator most often forgets to plot, and it is the column that catches a routing regression before any latency or error signal does. The same pipe chain handles it.

telemetry |>
  filter(is.na(error_class)) |>
  ggplot(aes(cost_usd, fill = agent_class)) +
  geom_density(alpha = 0.3) +
  scale_x_log10()

A cost density that splits into two modes after a release is the visual signature of an agent class that started taking a longer tool path. The latency stays inside its SLO because the longer path still completes in time; the customer never notices, but the unit economics of the deployment shift overnight. R surfaces the shift before the monthly cloud bill does, and that is the kind of finding an operator only gets from a language that makes plotting one column cheap enough to do without ceremony.

§III — Python for Production Drift Detection

The gate moves from exploration to enforcement the moment the SLI threshold is chosen, and the dialect shifts with it. Python owns the production pipelines per the Tuesday admission-control lesson and the Monday Protocol composition lesson; the drift check belongs in the same pipeline because the gate has to run every minute on a schedule no human watches.

The first enforcement task is windowed aggregation. pandas does the same group-by-and-roll that R did, with the difference that the call lives inside a scheduled job, returns a verdict, and writes the verdict to the same metrics backend the Ops lesson named.

import pandas as pd

def compute_error_rate_windows(events: pd.DataFrame, short_min: int = 60, long_min: int = 360) -> pd.DataFrame:
    df = events.set_index("ts").sort_index()
    df["is_err"] = df["error_class"].notna().astype(int)
    short = df["is_err"].rolling(f"{short_min}min").mean()
    long_ = df["is_err"].rolling(f"{long_min}min").mean()
    out = pd.DataFrame({"short_err_rate": short, "long_err_rate": long_})
    return out.dropna()

The function returns the two error-rate series the burn-rate alert in §IV will consume. Notice what is missing. There is no plot, no faceting, no interactive print, because Python’s job here is not to discover anything; it is to compute a number the alert layer can act on. The contract between the two languages stays clean as long as each one refuses the other’s job.

The second enforcement task is the distribution-drift check itself. The R session showed that the latency distribution had a clean shape last week; the Python gate has to notice when this week’s shape stops matching. A two-sample Kolmogorov-Smirnov test on the daily latency distributions is the cheapest distribution-difference signal that does not assume normality, and scipy.stats offers it as a one-line call.

from scipy import stats
import pandas as pd

def latency_drift_ks(yesterday: pd.Series, today: pd.Series, alpha: float = 0.01) -> dict:
    stat, p = stats.ks_2samp(yesterday.dropna(), today.dropna(), alternative="two-sided")
    return {
        "ks_stat": float(stat),
        "p_value": float(p),
        "drift": bool(p < alpha),
        "n_yesterday": int(yesterday.notna().sum()),
        "n_today": int(today.notna().sum()),
    }

A drift result with drift = True and a thousand requests on each side is a hard signal that the latency distribution has moved between days, which is the operator’s cue to inspect the deployment timeline and the routing changes before the SLO burns through the budget. A drift result with twenty requests on each side is noise from a low-traffic window and the alert layer must refuse to page on it; this is the small-N discipline the Ops lesson hinted at when it warned against alerts that fire on insufficient data.

A second drift signal catches the case the KS test misses. If the rate of one error class climbs while the latency distribution holds steady, the latency check stays quiet and the customer-visible experience still gets worse. A chi-squared test on the error-class counts catches that motion.

from scipy import stats

def error_class_drift_chisq(yesterday_counts: dict, today_counts: dict, alpha: float = 0.01) -> dict:
    classes = sorted(set(yesterday_counts) | set(today_counts))
    y = [yesterday_counts.get(c, 0) for c in classes]
    t = [today_counts.get(c, 0) for c in classes]
    table = [[y[i], t[i]] for i in range(len(classes))]
    stat, p, _, _ = stats.chi2_contingency(table)
    return {"chi2": float(stat), "p_value": float(p), "drift": bool(p < alpha), "classes": classes}

Together the two tests cover the two ways an agent stack can quietly go wrong between releases: tail-stretch in latency, or mix-shift in error class. The R exploration in §II decided that these were the right two questions; the Python in §III decides, every cycle, whether the answers have changed.

A practical note on cadence. The drift checks run daily, not every minute, because the comparison window has to hold enough requests on both sides for the test statistic to mean anything. The burn-rate alert in §IV runs every minute because it is reading a state, not testing a hypothesis. Mixing the two cadences is the most common drift-detection bug an operator ships: a daily test scheduled per-minute either floods the alert queue with low-power positives or quietly ignores its own results. Keep the cadence on the same clock as the statistic, and the gate stays honest.

The second practical note: the threshold alpha = 0.01 is a default, not a doctrine. An agent stack with many independent SLIs each running its own daily KS test will surface false positives at the published rate, and the on-call who reads them learns to ignore the channel. Either correct for multiple comparisons across the SLI set or raise the threshold until the false-positive volume matches the team’s tolerance. The R session can sample the distribution of p-values under the null directly by shuffling the day labels; the calibration takes ten minutes and saves a quarter of useless pages.

§IV — Multi-Window Burn-Rate Alerting in Python

The Ops lesson’s alert-routing section ended on the multi-window multi-burn-rate pattern: a fast window for page someone now and a slow window for file a ticket and let the daytime team look at it. The translation into Python is direct. Each window holds an error-rate series; each threshold compares the recent rate against the SLO budget; the alert state composes the two.

from dataclasses import dataclass

@dataclass
class BurnRateAlert:
    page: bool
    ticket: bool
    short_rate: float
    long_rate: float

def evaluate_burn_rate(short_rate: float, long_rate: float, slo_err_rate: float,
                      fast_multiplier: float = 14.4, slow_multiplier: float = 6.0) -> BurnRateAlert:
    fast_threshold = slo_err_rate * fast_multiplier
    slow_threshold = slo_err_rate * slow_multiplier
    page = short_rate > fast_threshold and long_rate > fast_threshold
    ticket = (not page) and short_rate > slow_threshold and long_rate > slow_threshold
    return BurnRateAlert(page=page, ticket=ticket, short_rate=short_rate, long_rate=long_rate)

The two multipliers are the published Google SRE workbook defaults for a thirty-day budget, and they encode the trade an alert designer always makes. A pure short-window alert catches incidents fast and pages on flapping every other shift; a pure long-window alert never flaps and lets an incident burn six hours before anyone notices. The conjunction is the compromise: page only when both windows agree, which kills the flap without sacrificing the response time the SLO promised.

The function signature returns a value object rather than firing the page itself, because the routing belongs in a separate stage. The pipeline composes them the way the Monday Protocol lesson taught: a stage that computes windows, a stage that evaluates burn-rate, a stage that dispatches to the on-call rotation. Each stage is testable in isolation, and the test cases come from the postmortem corpus the Ops lesson named: any alert that fired and should not have, any incident that fired late, lands as a regression test against this function.

The shape of the alert state matters as much as the threshold math. page and ticket are mutually exclusive by construction; the routing layer never has to decide between waking someone and filing a follow-up. That mutual exclusion is small-scale discipline that pays back in the postmortem the next morning, when the question did we route this right? answers itself from the data.

One more property of the dataclass design matters. Because BurnRateAlert carries the two rates alongside the verdicts, every page and every ticket arrives with the numbers that produced it. The on-call who reads page = True, short_rate = 0.063, long_rate = 0.058 against an SLO of 0.005 knows in one glance that both windows agree by an order of magnitude, which is the case where the rollback decision is straightforward. A page that arrives without those numbers forces the human to reconstruct the state from the dashboard, which is exactly the kind of three-in-the-morning friction the postmortem corpus eventually files as an action item. Build the friction out at construction time and the dashboard becomes a tool for confirmation rather than for archaeology.

§V — The Discipline Named

The two-language pattern coined on the 2026-06-11 chain-data lesson generalises here without strain. R asks what is this distribution shape? The answer governs whether the SLO is the right SLO at all. If the answer is long-tailed unimodal, a p99 threshold is well-posed and the only argument is the number. If the answer is bimodal, the SLI needs to split or the routing needs to change before any threshold can do honest work. If the answer is a shelf at the timeout, the SLO is lying and the engineering work is upstream of any alert.

Python asks what is this distribution doing right now? The answer governs whether the gate fires. The KS test on yesterday-versus-today, the chi-squared on error-class mix, the burn-rate computation on the two-window error rate: each one a verdict, computed on a schedule, against the threshold the R analyst chose. The choice was the human’s; the watch is the machine’s.

The handoff between the two languages is a single artifact: the chosen threshold. The R analyst commits a YAML file with latency_p99_ms: 800, the Python gate reads it, and the contract is closed. Re-open the contract whenever the distribution drifts in a way the chi-squared catches and the budget cannot absorb; never re-open it inside the gate code, because a threshold that drifts silently is no threshold at all. This is the same discipline the chain-data lesson named under the rolling-median band: R discovers what normal looks like, Python refuses anything that does not match.

§VI — Worked Example

Continue the Ops lesson’s worked example. The hallucination-rate SLO sat at 0.5% over a thirty-day budget; seventy-two hours after a model-version bump the burn-rate alert ticketed (not paged), because the long window agreed with the short and the slow multiplier crossed. The on-call engineer pulled an R session against the past three days of telemetry.

The density plot showed the latency distribution had not moved, which would have read as a routing or capacity story. The error-class mix had shifted: the hallucination class climbed from 0.4% to 0.9% while every other class held steady. The Python chi-squared check on yesterday-versus-today returned drift = True at p < 0.001. The KS test on latency returned drift = False; this was the case where the two tests together caught what either alone would have missed.

The rollback decision belonged to the engineer, who acted on the playbook Monday’s lesson coded: revert the model version, hold traffic at the previous build, schedule the postmortem for the next morning. The R session was the diagnostic; the Python check was the trigger; the rollback was the kill-switch lesson firing on cue. Three lessons collapsed into one incident, and the agent estate came back inside its budget by the next burn-rate window.

The postmortem the following morning added one regression test to the drift-check stage and one chart to the on-call runbook. The regression test froze the prior-day error-class counts and asserted that the chi-squared gate would have fired with the operator’s chosen threshold. The chart added the cost density from §II, because the same model bump that lifted the hallucination rate also pushed median cost per request up by eighteen percent, and the chart made that lift visible the next time it happened.

§VII — Closing Discipline

The visibility loop has two ends. The far end is the user’s experience of the agent stack: latency they wait through, errors they retry, costs the operator absorbs on their behalf. The near end is the alert that wakes the on-call. Between them sits one feedback path, and the path runs through two languages that do not compete for the same job.

R sees shape. Python sees state. The analyst who plots a density before setting a threshold, and the gate that checks the threshold every minute against a test the analyst chose, between them close the loop tighter than either alone ever could. Watch the shape with R. Watch the clock with Python. Wake the right human on the right minute.

The two languages do not compete because they answer different questions. The R session is where the operator argues with the data; the Python gate is where the data argues back. Hold the discipline and the agent estate gets quieter; break it, and every release ships a new way for the alert layer to lie.

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura Filed 2026-06-18 Thursday Fajr · Data Science tier (R + Python synthesis) · second R lesson in the curriculum

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura