R and Python for Cross-Sectional Dispersion Analysis

Cohort return panels, the dispersion-and-rank metrics, and the explore-in-R-enforce-in-Python discipline.

A metric no one plotted is a metric no one checked. See the spread before you trust the number.

Today's Ops lesson defines a classifier that reads a cohort's cross-section and labels its regime. It names four measurements — dispersion, structure, breadth, persistence — and treats them as given. This lesson is where they stop being given. The classifier is only as trustworthy as the panel it reads and the metrics computed on that panel, and both are the data scientist's work.

The two-language discipline that has run through every Thursday lesson applies here in its purest form. The cross-section is a distributional object: on each session the cohort's returns form a shape, and the whole question of coherence-versus-rotation-versus-fracture is a question about that shape. R is built to see distributional shape — one geom and the spread is on the screen. Python is built to run the same computation every session close without a human in the loop. The rule is unchanged: explore the shape in R, enforce the metric in Python.

§ ILanguage Idiom

The panel is the object, and R makes it tidy

A cohort return panel is a rectangle: one row per member per session, columns for the return and the ranked exposures. Wickham and Grolemund's transformation grammar (R for Data Science, Part I Explore, pp. 121–123) is exactly the vocabulary this rectangle wants — group_by the session, summarise to collapse each session's cross-section to its statistics, mutate to attach per-member ranks within the session. The tidy form is not cosmetic. It is what lets one line compute a per-session statistic across every session at once, with no loop.

The cross-sectional standard deviation is the dispersion metric, and in tidy R it is a grouped summary:

R · explorepanel_daily <- cohort_panel |>
  group_by(session) |>
  summarise(
    dispersion = sd(ret),
    breadth    = mean(ret > median(ret)),
    members    = n(),
    .groups = "drop"
  )

One grouped summary turns a long panel into a per-session dispersion series. The grouping is the cross-section; the summary is the day's shape reduced to numbers. Nothing here is a market-specific trick — it is the standard split-apply-combine that pp. 121–123 build, pointed at returns instead of the book's diamonds.

Seeing structure before trusting it

Dispersion is one number and it hides the difference between rotation and fracture. Structure is the fraction of the spread that sorts along an axis, and before computing it as a number the analyst should see it. Wickham's treatment of the smooth-versus-noise read (p. 130) is the habit: plot the return against the ranked exposure for a session, and a rotation shows as a visible slope while a fracture shows as a cloud.

R · exploresession_slice <- cohort_panel |> filter(session == target_day)

ggplot(session_slice, aes(x = stack_rank, y = ret)) +
  geom_point(size = 2) +
  geom_smooth(method = "lm", se = FALSE) +
  labs(x = "position in stack (low to high)", y = "session return")

The fitted line is the axis; the tightness of the points around it is the structure. A steep line with points hugging it is high-structure rotation. A flat line with a wide cloud is fracture. The plot answers the classifier's hardest question before a single threshold is set, and it does so in a form a human can veto.

Rank persistence is a rank correlation

Persistence asks whether today's ranking resembles yesterday's, and the honest metric is Spearman's rank correlation — it reads order, not magnitude, which is what rotation is about.

R · explorerank_today <- rank(session_slice$ret)
rank_prev  <- rank(prev_slice$ret)
persistence <- cor(rank_today, rank_prev, method = "spearman")

A persistence near one says the ordering held and the rotation is a regime. A persistence near zero says today's order is unrelated to yesterday's, and the wide spread is noise the classifier must refuse.

§ IICode Worked Example

R showed the shape and set the thresholds. The production job that runs at each session close is Python, and it computes the same three metrics on the same panel with no plots and no analyst. The discipline is that Python does not re-derive the method — it enforces what R already validated.

Python · enforceimport pandas as pd

def session_metrics(panel: pd.DataFrame, axis_col: str) -> pd.DataFrame:
    def per_session(g):
        r = g["ret"]
        disp = r.std(ddof=1)
        breadth = (r > r.median()).mean()
        fit = g[axis_col].corr(r, method="pearson")
        return pd.Series({"dispersion": disp, "breadth": breadth, "structure": abs(fit)})
    return panel.groupby("session", group_keys=True).apply(per_session)

The groupby is R's group_by, the apply is the summary, and the structure metric is the correlation of return against the ranked axis — the numeric form of the slope the R plot drew by eye. The abs on the fit is deliberate: rotation up the stack and rotation down the stack are both structured, and the classifier cares that an axis exists, not its sign.

Persistence needs two adjacent sessions, so it runs across the grouped frame rather than inside one group.

Python · enforcedef rank_persistence(panel: pd.DataFrame) -> pd.Series:
    ranks = (panel.assign(rk=panel.groupby("session")["ret"].rank())
                  .pivot(index="member", columns="session", values="rk"))
    out = {}
    cols = list(ranks.columns)
    for prev, cur in zip(cols, cols[1:]):
        out[cur] = ranks[prev].corr(ranks[cur], method="spearman")
    return pd.Series(out, name="persistence")

The pivot puts members down the rows and sessions across the columns — the panel in the wide shape the Ops lesson's classifier assumed. Each adjacent column pair yields one Spearman correlation, and the series of them is the persistence track the classifier reads to separate a regime from a one-day inversion.

§ IIIConnection to Today's Ops Lesson

The Ops lesson's classifier takes four measurements as inputs and emits a regime label. This lesson produces three of the four — dispersion, structure, persistence — and breadth falls out of the same grouped summary. The Ops rule that structure gates the rotation call is enforced here by the abs(fit) structure metric and the Spearman persistence: rotation requires both to clear their thresholds, and the code returns them side by side so the classifier can apply the gate. The Ops lesson said fracture is the conservative default; this lesson makes that default computable, because a wide dispersion with a low structure metric and a near-zero persistence is exactly the numeric fingerprint of fracture.

§ IVPrior-Lesson Reach

06-25 R and Python for security telemetry. That lesson built baselines and control charts — a per-metric distribution over time, with deviation gates. Today's panel is the same tidy-then-enforce shape pointed at a cross-section instead of a time series: group_by the session rather than the metric, read the spread rather than the deviation from a mean. The exploratory-in-R habit is identical; only the axis of the grouping changed.

06-18 R and Python for operational telemetry. The SLI time-series lesson introduced rolling windows and drift detection. Persistence here is drift's cross-sectional twin — instead of asking whether one series drifted from its own history, it asks whether the ranking drifted from yesterday's ranking. Both are stability questions; one runs down the time axis, the other across the member axis.

§ VClosing

The classifier the Ops lesson described is downstream of a panel and three metrics, and this is where they are built and trusted. R makes the cross-section's shape visible so a human validates that dispersion, structure, and persistence measure what the classifier needs — the slope drawn by eye before it is a threshold. Python computes the same three every session close so the label is reproducible without anyone watching. The panel is the object; the grouped summary is the shape; the rank correlation is the memory. Build it in R until you believe it, then enforce it in Python so it never has to be believed again.

See the spread in R before you trust the number in Python. A metric no one plotted is a metric no one checked.