Bash for Validator Liveness Watchdogs

Polling the sign-info endpoint, missed-block threshold alerting, and the idempotent auto-unjail guard.

§ IFrame

The paired Ops lesson made one demand of the operator: watch the missed-block meter every minute and page before the chain's jail fires. A human cannot do that. No operator reads a JSON endpoint sixty times an hour through the night, and the value of the watch is precisely in the hours no human is awake. The watch has to be a script, and the script has to be the kind that runs unattended for months and is trusted to wake a person only when waking them is justified.

Bash is the right tool for exactly the reasons it is the dangerous one. The watchdog is a small loop around two external commands, a query against the node and a notification out, with a threshold compare in the middle. That is a shell script's whole natural shape, and reaching for Python here adds a runtime and a dependency set to babysit on the same host whose health is already in question. The cost of Bash is that its defaults are wrong for unattended automation: an unset variable expands to nothing instead of erroring, a failed command in a pipe is invisible, and two copies of the script can run at once and fight. Last Saturday's Bash lesson fixed those defaults for the key-ceremony scripts. This lesson reuses that exact discipline for the watchdog, then adds the one guard a recovery script needs that a ceremony script does not.

The watchdog has three jobs, mapped one-to-one onto the Ops lesson's three reads. It polls the sign-info record and computes the missed-block drift. It compares the drift to a threshold set far below the chain's floor and pages when crossed. And, optionally and carefully, it drives the unjail recovery behind a gate that refuses to fire against an unhealthy node. The third job is where a careless script stops being a safety tool and becomes a second outage.

§ IILanguage Idiom — Strict Mode, jq, and the Confirm-Then-Act Gate

Three shell idioms carry the watchdog, and two of them came from the 06-13 lesson unchanged.

The first is strict mode. Every script in this discipline opens with set -euo pipefail. The -e exits on any unhandled command failure, the -u errors on an unset variable rather than expanding it to the empty string, and the -o pipefail makes a pipeline fail if any stage fails rather than reporting only the last stage's status. For a watchdog this is correctness, not hygiene: a poll that silently produces an empty string because curl failed must not be read as zero missed blocks, because zero missed blocks is the all-clear, and an all-clear manufactured from a failed query is the exact false negative that lets a real outage run to a jail unannounced.

The second is structured parsing with jq. The node returns JSON, and the wrong way to read JSON in shell is grep and cut, which break the moment a field reorders or a whitespace changes. jq parses the document and extracts a field by path, returning a clean value or a clear error. The watchdog reads two fields this way, the missed-block counter and the jailed boolean, and treats a jq failure as a failed poll rather than as data.

The third idiom is new to the recovery path: the confirm-then-act gate. A ceremony script from last week did one irreversible thing once and exited. A recovery script is asked to do something conditional and possibly repeatedly, and the condition has to be re-checked at the moment of action, not at the start of the script, because the node's health can change between the check and the act. The gate is a function that returns success only when the node is verifiably caught up, and the unjail command runs only inside the success branch of that gate, evaluated immediately before the command, never cached from an earlier read.

§ IIICode Worked Example — The Watchdog

The watchdog opens in strict mode and pins its configuration up front, so that an operator reading the script sees every tunable in one place and a typo in a variable name errors at startup rather than at three in the morning.

#!/usr/bin/env bash
set -euo pipefail

VALOPER="cosmosvaloper1examplevaloperaddresshere000000000000"
NODE_RPC="http://127.0.0.1:26657"
NODE_API="http://127.0.0.1:1317"
PAGE_THRESHOLD=500
LOCKFILE="/run/lock/validator-watchdog.lock"
STATE_DIR="/var/lib/validator-watchdog"

The first job is the poll. A single function asks the node for this validator's signing info and returns the missed-block counter as a bare integer. The function routes every failure into one place: a failed curl, a non-JSON body, or a missing field all produce a non-zero exit, and strict mode turns that exit into a script failure rather than a misleading zero. The prose contract is the thing to hold here — this function returns a real missed-block count or it returns nothing at all, and it never returns a fabricated zero.

read_missed_blocks() {
  local body
  body=$(curl -fsS --max-time 5 \
    "${NODE_API}/cosmos/slashing/v1beta1/signing_infos/$(consensus_pubkey)")
  printf '%s' "${body}" | jq -er '.val_signing_info.missed_blocks_counter | tonumber'
}

The second job compares the count to the threshold and pages exactly once per crossing. Paging on every poll while an outage persists trains the operator to ignore the pager, so the script records that it has paged for the current episode and stays quiet until the count recovers below the threshold, at which point it clears the episode and is ready to page again for the next one. The state is a single file whose presence means an unacknowledged page is outstanding.

evaluate() {
  local missed paged_marker="${STATE_DIR}/paged"
  missed=$(read_missed_blocks)
  if [ "${missed}" -ge "${PAGE_THRESHOLD}" ]; then
    if [ ! -f "${paged_marker}" ]; then
      send_page "validator missed ${missed} blocks (threshold ${PAGE_THRESHOLD})"
      touch "${paged_marker}"
    fi
  else
    rm -f "${paged_marker}"
  fi
}

The third job is the recovery, and it carries the guard the other two do not need. The unjail must run under a lock so that two invocations of the watchdog can never submit two unjail transactions at once, and it must re-check caught-up-ness inside the lock, immediately before submitting. The lock is flock on a dedicated descriptor, the same primitive the 06-13 lesson used to keep two key ceremonies from running together. The caught-up check reads the node's sync status and refuses the unjail unless the node reports it is not catching up.

node_is_caught_up() {
  local catching_up
  catching_up=$(curl -fsS --max-time 5 "${NODE_RPC}/status" \
    | jq -er '.result.sync_info.catching_up')
  [ "${catching_up}" = "false" ]
}

attempt_unjail() {
  exec 9>"${LOCKFILE}"
  flock -n 9 || { log "another watchdog holds the lock; skipping unjail"; return 0; }
  if ! node_is_caught_up; then
    log "node still catching up; refusing to unjail"
    return 0
  fi
  log "node caught up; submitting unjail"
  validator-cli tx slashing unjail --from operator --yes
}

The ordering inside attempt_unjail is the whole lesson. The lock is taken first, so only one watchdog acts. The caught-up check runs second, inside the lock, so the health read cannot be stale relative to the action. The unjail runs third, only in the branch where the node verifiably reports itself caught up. Reverse any two of those and the script regains the failure mode it was written to prevent: check health first and unjail later, and a node that was healthy at the check but stalled by the time of the unjail rejoins broken and earns a second jail, which is the precise accident the paired Ops lesson described.

§ IVConnection to Today's Ops Lesson

The Ops lesson named three reads on a live validator, and the script is those three reads made executable. read_missed_blocks is the missed-block meter, polled on the interval a cron or systemd timer sets. evaluate is the page, firing at five hundred misses against a chain floor that tolerates thousands, which is the Ops lesson's discipline of alarming at a fraction of the budget rather than at its exhaustion. attempt_unjail is the unjail recovery gate, and its refusal to act against an uncaught-up node is the shell enforcement of the Ops lesson's rule that recovery is a gate and not a button.

The asymmetry the Ops lesson drew also shapes the script's failure posture. Because no single missed block matters and the cost is latency-to-notice, the watchdog is built to fail loud rather than fail silent: every ambiguous read becomes a script error and a missing heartbeat, and a watchdog that has stopped reporting is itself a page-worthy condition. A liveness monitor whose own death is silent is worse than no monitor, because it converts a known blind spot into a believed-good one.

§ VPrior-Lesson Reach

The 06-13 Bash lesson built the key-ceremony and sentry-firewall scripts on set -euo pipefail, trap-based cleanup, and flock lockfiles, and coined the idempotent guard: a script that can run twice and the second run is a safe no-op. The watchdog inherits all of it. The lock in attempt_unjail is the same flock pattern. The page-marker file in evaluate is the same idempotency idea applied to notification rather than to ceremony: the marker makes a repeated page a no-op until the condition clears, exactly as the ceremony's guard made a repeated key-generation a no-op once the key existed.

The one new element is the confirm-then-act ordering inside the lock, which the ceremony scripts did not need because their action was unconditional. A ceremony either generates the key or finds it already generated; there is no live external condition that can change between check and act. The watchdog's unjail does have such a condition, the node's sync state, and the discipline that condition demands, re-check inside the lock immediately before acting, is the lesson the recovery path adds on top of last week's foundation.

§ VIClosing

A liveness watchdog is three external calls and a compare, and it is trusted with waking a person at night and with deciding whether to put a jailed validator back into the active set. The smallness is the trap: it is exactly the script an operator writes in ten minutes without strict mode, reads zero missed blocks from a failed curl, and never hears from again until the jail. Write it the other way. Make every ambiguous read an error, page once per episode, and gate the unjail behind a lock and a fresh caught-up check in that order.

Run the watchdog against a testnet validator and force the failure paths by hand: kill the node mid-poll and confirm the script errors instead of reading zero, then jail the validator deliberately and watch the unjail refuse until the node is genuinely caught up. The script you trust at three in the morning is the one whose failure modes you have already triggered on purpose.

Paired Ops → δ-Chain/Synthesis-Lessons/2026-06-20-validator-liveness-monitoring-and-the-downtime-slashing-defense
Paired Cert → Cert-Prep/Interchain/2026-06-20-cometbft-liveness-and-the-x-slashing-downtime-window