Validator Liveness Monitoring and the Downtime-Slashing Defense

Missed-block alerting, the jail-and-unjail cycle, and uptime SLOs for sovereign-chain validators.

§ IFrame

A chain punishes a validator two ways, and they are not the same crime. Sign two conflicting blocks at one height and the protocol calls you an attacker: it takes a large fraction of the whole stake and tombstones the key forever. Stop signing for too long and the protocol calls you absent: it takes a small fraction, jails you, and lets you come back. Last Saturday's lesson spent its length on the first crime, because a double-sign is the catastrophic one. Today is the second crime, because the absent one is the crime an operator actually commits.

Almost no validator ever double-signs. A disciplined signer architecture makes that failure rare by construction. Downtime is different. Downtime is the ordinary failure, the one that happens on a Tuesday because a disk filled, a peer set went stale, an upgrade ran long, or a process restarted into a config that pointed at a signer that was not there. The cost of any single missed block is nothing. The cost of missing too many in a window is a jail, and a jail that nobody notices for a day is a stake that earned nothing for that day and shed a slice of itself on the way out.

The whole discipline rests on a measurement. A validator does not know it is failing liveness; the chain knows. The chain keeps a sliding window of recent blocks and counts how many of them carried this validator's signature. Fall below the floor inside that window and the jail fires automatically, with no human in the loop and no warning the validator did not build for itself. So the operator's job is to watch the same window the chain watches, and to react before the chain does. The chain's count is the scoreboard. The operator who is not reading the scoreboard is playing blind.

§ IIFoundations — Three Numbers That Govern a Jail

Three chain parameters decide whether a validator is alive, and every alert in this lesson watches one of them. Name them and the monitoring stops being folklore.

The first is the window. The chain looks back over a fixed number of recent blocks, the signed-blocks window. On a typical Cosmos chain this is ten thousand blocks, which at a few seconds per block is a span of hours, not minutes. The window is forgiving by design: it does not punish a brief stumble, only a sustained absence. An operator who understands the window understands that a thirty-second outage is invisible to slashing and a four-hour outage is fatal.

The second is the floor. Inside the window the chain requires a minimum fraction of blocks signed, the minimum-signed-per-window. Set to five percent, it means the validator may miss most of the window and still survive, which sounds generous until the arithmetic lands: the misses accumulate across the whole window, so a node that goes fully dark burns its entire margin in one continuous outage. The floor is a budget of absence, and a dark node spends it at full rate.

The third is the price. When the floor breaks, the chain applies the downtime slash fraction, a small number, often a hundredth of a percent or a tenth of a percent of the bonded stake, and jails the validator. Jailing means removed from the active set, signing nothing, earning nothing, until the operator sends a transaction that asks to be let back in. The price has two parts: the slice taken at the moment of jail, and the rewards forgone for every block the validator sits jailed afterward. On a long jail the second part dwarfs the first.

Window, floor, price. The window sets how long the chain remembers, the floor sets how much absence it tolerates, the price sets what tolerance exhausted costs. Hold the three and the alerting design writes itself: watch the rate of misses against the window, alarm well above the floor, and treat any jail as an incident whose clock started the instant it fired.

§ IIIMechanism — Three Reads on a Live Validator

A validator's liveness is legible through three operational reads, each a different distance from the jail. One meters the drift, one catches the event, one drives the recovery.

§ IVWorked Example — A Jail That Costs Two Minutes Instead of a Day

Consider a validator on a Cosmos chain with a hundred-thousand-token bonded stake, running the remote-signer-plus-sentry shape from last Saturday's lesson. The chain's window is ten thousand blocks at six seconds each, roughly seventeen hours. The floor is five percent. The downtime slash is one-hundredth of one percent.

At two in the morning the primary sentry loses its peer connections after a network blip, and the validator, which connects only to its sentries, goes quiet. It is not signing. The missed-block meter, polled every minute by the operator's watchdog, begins to climb: one missed block, then ten, then a hundred. None of this is yet a slashing event. The validator has a margin of thousands of misses before the floor breaks, and at six seconds a block that margin is most of a day of darkness. The chain is patient. The operator does not have to be fast, only awake.

The watchdog crosses its own threshold, set far below the chain's floor, at five hundred consecutive misses, well under an hour of outage. It pages the operator. The operator wakes, sees the sentry peer set is empty, restarts the sentry, and watches the validator reconnect and resume signing. The meter stops climbing at six hundred and forty misses and begins to age out of the window. No jail. No slash. The margin was never close to spent, because the alert fired at six percent of the budget rather than at a hundred percent of it. The cost of the incident is one operator's interrupted sleep.

Now run the same outage with no watchdog. The validator goes dark at two, and nobody is reading the scoreboard. The meter climbs through the night. A fully dark node burns its margin in a single span, and on this chain seventeen hours of continuous darkness exhausts the window. The jail fires near dawn. One-hundredth of a percent of a hundred thousand tokens is taken, and the validator is out of the set. The operator discovers it at nine when delegators ask why rewards stopped. The slash was small. The hours of forgone rewards between dawn and nine, on a hundred-thousand-token stake in the active set, were not. The same outage, the same root cause, two outcomes separated only by whether someone was watching the meter.

§ VConnection to Prior Lessons

The Validator Operations lesson (δ-Chain Sat 2026-05-23) named the validator's three concerns as identity, liveness, and safety, and built its length on liveness as the place a new operator loses money first through the slow bleed of downtime jail. It named the bleed; today measures it. The signing-info record, the window, and the jail were the mechanisms that lesson pointed at and deferred. This is the deferred chapter, and it pairs with last Saturday's safety chapter to complete the two-condition model: safety punishes a duplicated signature, liveness punishes an absent one, and a real validator defends both at once.

The RPC and Full-Node Infrastructure lesson (δ-Chain Tue 2026-06-09) gated a public read fleet on caught-up-ness, refusing to serve from a node that had fallen behind the head. The unjail gate in §III is that same caught-up check pointed at a different decision. There, caught-up-ness decided whether a node was fit to answer readers; here, it decides whether a validator is fit to rejoin the active set. The snapshot-restore discipline is also the fast path out of certain jails: a validator jailed because its node fell hopelessly behind recovers fastest by state-syncing from a recent snapshot, then unjailing once caught up.

The Validator Signing-Key Security lesson (δ-Chain Sat 2026-06-13) warned that a restart which drops the signer connection or starts with a local-key fallback can double-sign at the worst moment. That warning has a liveness shadow: a restart that drops the signer connection without falling back to anything signs nothing, and a validator that signs nothing is on the clock toward a downtime jail. The signer architecture that prevents double-sign must also be watched for the silent not-signing it can produce, which is exactly the meter this lesson reads.

§ VIConnection to Today's Dev and Cert Lessons

The paired Dev lesson builds the watchdog from §IV in Bash. The poll loop that reads the signing-info record every minute, the threshold that pages at five hundred misses rather than at the chain's floor, the unjail guard that refuses to fire until the node reports caught-up: each is a shell script, and each is the kind of script where a forgotten set -euo pipefail or a missing lockfile turns a recovery tool into a second outage. The auto-unjail script is especially dangerous, because an unjail that fires against an unhealthy node manufactures the second jail this lesson warned about. The Bash lesson writes the guard that makes the recovery idempotent and gated.

The cert lesson goes underneath both into the protocol. It names where the window and the floor actually live, the signed_blocks_window and min_signed_per_window parameters in the Cosmos SDK x/slashing module, how the chain tracks each validator's missed-block counter in a bit-array, and the precise distinction between the recoverable downtime jail and the permanent double-sign tombstone. The Ops lesson watches the meter; the cert lesson shows exactly how the protocol computes the number the meter is reading.

Paired Dev → Polyglot-Dev/Bash/2026-06-20-bash-for-validator-liveness-watchdogs-polling-the-sign-info-endpoint-missed-block-threshold-alerting-and-the-idempotent-auto-unjail-guard
Paired Cert → Cert-Prep/Interchain/2026-06-20-cometbft-liveness-and-the-x-slashing-downtime-window-signed-blocks-window-min-signed-per-window-and-the-jail-unjail-tombstone-distinction

§ VIIClosing

A validator's second job, after signing in exactly one place, is to keep signing. The first job is defended by architecture; the second is defended by attention. The chain gives generous warning — a window measured in hours, a floor that tolerates most of it — and that generosity is wasted entirely on the operator who reads the scoreboard at nine in the morning. Build the meter that reads it every minute. Page yourself at a fraction of the floor, not at the floor. Gate the recovery on caught-up so that coming back never becomes leaving again.

Set the watchdog's page threshold against your own chain's window and floor before you trust it, and walk the unjail path once on a testnet while nothing is at stake. The operator who first reasons about the jailed-until clock during a real jail has already paid for the lesson in forgone rewards.