Short-Lived Credentials, the Zero-Downtime Rotation Discipline, and the Blast-Radius Containment Window
Every earlier Trust lesson treated the secret as furniture. RBAC decided which ServiceAccount could read a secret. Encryption-at-rest kept it unreadable on disk. The admission gate governed how it entered a pod. All of that protects a credential that already exists and never changes. It says nothing about where the credential came from, how long it should live, or what happens the morning you learn it leaked.
That gap is the whole subject here. A long-lived credential is a standing liability: the longer it lives, the more copies exist, the more logs it has passed through, and the wider the window in which a single leak stays useful to an attacker. A multi-agent fleet makes this worse by multiplication. Dozens of agent processes each hold API keys, database passwords, and signing tokens. A static secret shared across all of them is a single key that opens every door and never gets re-cut.
The discipline that replaces it has three moves, and I will name the third now because the lesson leans on it. Issue credentials that expire on their own. Rotate them on a schedule without taking the fleet down. And when one leaks, measure and shrink the blast-radius containment window — the span of time and set of resources a leaked credential can still reach before it is dead. Issue short, rotate often, contain the blast.
Start with what a secret actually is in an operational sense: not the string, but the grant the string represents. A database password is a grant to read and write a schema. A cloud API key is a grant to call a set of endpoints as some identity. The string is only the bearer token for that grant. This reframing matters because it tells you what to shorten. You cannot make the grant safer by hiding the string better; you make it safer by making the grant expire.
Static versus dynamic secrets. A static secret is minted once and reused until someone remembers to change it — which, in practice, is after an incident. A dynamic secret is minted on demand for a specific consumer with a specific lifetime, then revoked when the lifetime ends. A secrets engine like HashiCorp Vault, or a cloud equivalent, generates a fresh database credential each time an agent asks, hands it a fifteen-minute lease, and drops the credential from the database when the lease expires. The agent never holds a password that outlives its own task.
Newman frames the payoff in Building Microservices (pp. 444–445): the security value of a system is judged less by whether it can be breached and more by how quickly it can recover — detect, revoke, and re-key without a human in the critical path. A fleet built on dynamic, leased secrets recovers by doing nothing: the leaked lease was already going to expire. A fleet built on static secrets recovers by a frantic manual rotation across every consumer, which is exactly when mistakes get made.
The rotation problem underneath. Rotation sounds simple — replace the old secret with a new one — until you notice that the fleet is reading the secret at the same moment you are writing it. Rotate naively and half your agents authenticate with the old value and half with the new, and the ones that guess wrong get locked out. The core of this lesson is the pattern that makes rotation invisible to the consumer.
Hold three disciplines apart, because they answer three different questions: how long should a credential live, how do you replace it without downtime, and how far can a leaked one reach.
Mint credentials with a lease measured in minutes, scoped to one consumer and one grant. The credential's own expiry is the primary defense — a leaked fifteen-minute token is worthless in twenty. Prefer a secrets engine that revokes at lease-end over any store that hands out permanent strings.
Never flip a single value in place. Keep two credentials valid at once during a rotation window: publish the new one, let every consumer pick it up, then revoke the old. The overlap is what removes the race. Rotation becomes a state machine, not an edit.
For every credential, know two numbers: its remaining lifetime and the exact set of resources it can reach. The product of those is the blast radius. Shrink both — shorter leases, tighter scopes — so a leak is contained before you even detect it.
The unit of a good secrets system is the lease, not the password. When an agent boots, it authenticates to the secrets engine using an identity it already has — a Kubernetes ServiceAccount token, a cloud instance role — and asks for a database credential. The engine creates a fresh user in the database, grants it exactly the privileges the agent needs, and returns it with a fifteen-minute lease. The agent uses it, and before the lease lapses the engine drops that database user. No credential the agent holds is valid long enough to be worth stealing at rest.
This is also how you break the shared-key problem. Because each agent gets its own leased credential tied to its own identity, a leak is attributable — you know which agent's lease leaked — and revocable in isolation. You revoke one lease, not the one key that every agent shared.
The pattern that makes rotation safe is deliberate overlap. Treat every rotation as three ordered phases, and never collapse them:
ROTATION STATE MACHINE (consumer sees no downtime)
phase 1 PUBLISH_NEW
- mint credential v2 alongside still-valid v1
- write v2 to the secret store; v1 remains accepted
phase 2 DRAIN
- consumers refresh, pick up v2 on their own cycle
- both v1 and v2 authenticate successfully
- wait >= max(consumer refresh interval) + margin
phase 3 REVOKE_OLD
- confirm zero recent auths using v1 (check logs)
- revoke v1 at the source of truth
- only v2 remains valid
invariant: at every instant, at least one valid credential
exists that every consumer either has or can fetch.The invariant at the bottom is the entire discipline. If you can state, at every instant of the rotation, that a valid credential exists that every consumer can reach, then no consumer is ever locked out. Break the overlap — revoke v1 before every consumer has v2 — and you have manufactured an outage out of a routine maintenance task. The DRAIN phase is not optional padding; it is the window in which the race resolves itself.
Two operational details make this real. First, the drain wait must exceed your slowest consumer's refresh interval, so pick a refresh interval short enough that the drain is minutes, not hours. Second, phase 3 is gated on evidence, not a timer alone: before revoking v1, read the authentication logs and confirm nothing has used v1 recently. That check is why the earlier Trust lessons on audit logging were prerequisites — you cannot safely revoke what you cannot observe.
When a credential leaks, two questions decide how bad it is. How long is it still valid, and what can it reach. Call the answer the containment window: the interval during which, and the resource set over which, a leaked credential remains dangerous. Everything in this lesson is aimed at shrinking that window before an incident, so that when one arrives the damage is already bounded.
A static, broadly-scoped admin key has an unbounded window — valid forever, reaches everything. A fifteen-minute lease scoped to one database schema has a window of at most fifteen minutes over one schema. The difference is not a marginal improvement; it is the difference between a fleet-wide incident and a shrug. When Newman's recover framing asks how fast you return to safe, the leased-and-scoped credential answers before you finish reading the alert.
The revoke-before-drain outage. The most common self-inflicted wound: an operator rotates by editing the secret in place and immediately revoking the old value. Consumers that had not yet refreshed authenticate with a dead credential and fall over. The fix is structural, not careful — make rotation a three-phase machine with a mandatory drain, so in-place edits are impossible by construction.
The unbounded lease. A secrets engine configured with a lease of 0 or never reintroduces the static secret wearing dynamic clothing. The credential is minted on demand, which feels modern, but it never expires, so the blast-radius window is back to unbounded. Audit lease durations the way you audit RBAC scopes; a lease longer than the task that needs it is a finding.
The copy that outlives the lease. An agent that reads a leased credential and writes it to a log line, an environment dump, or a cache defeats the lease — the string now lives somewhere the engine cannot revoke. This is where the Dev lesson picks up: the same failure in a browser is a token written to localStorage that survives the session it was scoped to. Hold leased credentials in memory only, use them, and let them fall out of scope with the process.
Rotation without observation. Revoking the old credential on a timer alone, without checking the auth logs, revokes it while a lagging consumer still depends on it. The evidence-gated phase 3 is the guard. A fleet that rotates blind eventually rotates into an outage.
This lesson is the issuance-and-lifecycle floor under the earlier Trust arc. The 06-24 RBAC lesson decided who may read a secret; this one decides how long that secret is worth reading. The 06-25 anomaly-to-isolate pipeline detects a compromised agent; leased credentials make the isolation cheap, because revoking one lease is a single call. The 07-01 admission gate refused unsigned images at the door; the same fail-closed instinct here refuses to run an agent whose credential lease cannot be minted.
Today's Dev lesson takes the identical discipline into the browser. A short-lived access token with a refresh cycle is the client-side lease; the in-memory-only storage rule is the client-side version of "the copy that outlives the lease" failure; and the secure-context boundary is where the browser refuses to hand a credential to code that has not earned it. Today's Cert lesson grounds all of it in Kubernetes objects — Secret resources, encryption-at-rest, external secret stores, and projected ServiceAccount tokens with expiry — which is where a fleet actually implements the lease.
Issue short. Rotate often. Contain the blast. A fleet that holds those three does not defend its secrets by keeping them hidden forever; it defends them by making each one expire before it matters, replacing them without anyone noticing, and knowing to the minute how far a leak could reach.
Build the rotation machine before the leak, not after. The fleet that adds automated rotation in the calm week runs it a thousand times uneventfully and is already safe the day a credential appears in a public log. The fleet that waits does its first rotation under fire, by hand, across every consumer at once, and that is precisely the wrong moment to learn that revoke-before-drain locks everyone out. Measure your blast-radius window today, and spend the quiet time making it smaller.