Terraform as the Operate-and-Recover Provisioning Foundation

Week 5 reach-back synthesis of Vertex rollback, AWS autoscaling, Kubernetes runtime security, and Bedrock agent tracing.

What does an operator declare so that recovery is a command, not a scramble?

The Friday lesson asks the week's four vendors one question through one tool. This week the question was operate-and-recover: run the workload safely, watch it run, and pull it back when it goes wrong. Monday gave the model a safe rollback. Tuesday gave the fleet elastic scale. Wednesday confined the workload to least privilege. Thursday made the agent's work visible. Four faces of running things in production without losing control of them, and Terraform is where all four are declared as code that an apply restores when a 3 a.m. incident has bent them by hand.

The four prior Friday reach-backs each found a single seam. Week 2 found trust, Week 3 found observe-and-enforce, Week 4 found reach. Week 5 finds the operator's recovery posture: every safety this week is a configured resource, and a safety you cannot recreate from configuration is a safety that does not survive the incident that needs it most.

§ IFrame

A production operator lives between two states: the workload running as intended, and the workload running wrong. The week studied four moves that span that gap. A rollback returns a bad model to a known-good version. Autoscaling meets a load surge without a human at the console. Runtime confinement keeps a compromised process from becoming a compromised host. Operational tracing turns an agent's silent work into a record you can read after the fact. Each is a different answer to the same question: when production goes sideways, what does the operator already have in place that makes the recovery a known move rather than an improvisation.

Terraform's claim across all four is narrow and strong. The rollback path, the scaling policy, the security context, and the tracing pipeline are not workload code. They are configuration, and configuration clicked into a console is configuration no one can review, diff, or recreate. The week's lesson, said once for four vendors: declare the recovery posture as code, plan it on a schedule, and read the diff. A console hides drift until an incident reveals it. Terraform shows drift in tomorrow's plan, before the incident.

§ IIWeek in Review

Monday studied Vertex AI endpoint traffic splitting, model version aliasing, and the safe rollback discipline across the Google GenAI Engineer and Pro ML Engineer tracks. The week's central correction landed here: a rollback is a deploy, not an undo. The endpoint holds several model versions and splits traffic between them by a configured weight; a rollback is an edit to that weight, moving traffic from the regressed version back to the last good one, not a rebuild and not a delete. The endpoint, its deployed versions, and the traffic split are configured resources. The rollback the operator depends on is configuration.

Tuesday studied AWS compute elasticity and scaling automation across the Solutions Architect Professional and DevOps Professional tracks. The Auto Scaling group met a load surge by target tracking, holding a metric such as average CPU or request count at a setpoint by adding and removing instances. The cold-start tax, the time a fresh instance needs before it can serve, was answered by warm pools that keep pre-initialized instances ready to join. The scaling group, its target-tracking policy, and its warm pool are configured resources. The elasticity the fleet depends on is configuration.

Wednesday studied Kubernetes runtime security and the SecurityContext across CKA and CKS. The workload was confined to least privilege at the process and kernel boundary: drop every Linux capability and add back only the few the workload truly needs, filter the syscalls it may make with a seccomp profile, constrain it further with AppArmor, and mount its root filesystem read-only so a compromise cannot rewrite the binary. The security context is a Kubernetes resource. The confinement the operator depends on is configuration.

Thursday studied operational visibility for agentic systems across AWS AIP-C01 and GitHub GH-600. Amazon Bedrock agent tracing recorded each step an agent took into a trace; AWS X-Ray stitched those traces into a service map; GitHub Copilot workflow logs did the same for coding agents. The trace configuration, the X-Ray sampling rule, and the log destination are configured resources. The visibility the operator depends on is configuration.

Four lessons, four vendors, one repeated fact: the operator's whole recovery posture is configuration, not workload code. A rollback weight, a scaling policy, a security context, a tracing rule. Each is a setting, and a setting clicked into a console is a setting no one reviewed and no one can recreate after the accident that loosened it. The same setting in Terraform is reviewed in a pull request, diffed against the prior state, and restored by a single apply.

§ IIITerraform as the Connective Tissue

Terraform's model has three moves that turn the week's four postures into one declared, recoverable system. The first is the resource as the unit of declared desired-state. The second is the module as the unit of reusable, versioned topology. The third is the remote state backend with locking and the scheduled plan that reads drift off it. Each is exam material, and each maps onto a discipline the week named.

The resource is the recovery posture as desired-state. Terraform's core loop compares the desired state in configuration against the real state of the world and produces a plan to close the gap. The Vertex traffic split should be 100 percent on the good version; if a console edit pushed it to the regressed one, the plan shows the drift and the apply restores the intended split. The Auto Scaling target should be a chosen setpoint; the seccomp profile should be the named one; the X-Ray sampling rule should be the chosen rate. Each is declared once as desired-state, and the apply is the operator's recovery move made repeatable.

The module is the recovery posture made reusable across many workloads. Brikman makes the module the unit that keeps repetition honest: a module declares a stack once and the variables a stack needs become its input variables, so the same reviewed safety posture instantiates per workload with only its workload-specific inputs changing (Terraform: Up and Running, 3rd ed., Ch 4, pp 205-206). A safe-deploy module that wraps a Vertex endpoint with its rollback-ready traffic split, a confined-workload module that wraps a deployment with its security context, each declared once and instantiated per service, is how a fleet shares one reviewed recovery posture rather than fifty hand-built ones that drift apart.

Remote state with locking, and the scheduled plan that reads it, is the recovery posture watched over time. Brikman's production-grade chapter makes versioned, locked remote state the precondition for any infrastructure more than one engineer touches, so concurrent applies serialize rather than corrupt each other (Ch 8, pp 428-429). On top of that, a Terraform plan run on a schedule is the drift detector for the entire week's posture at once. A non-empty scheduled plan means something changed the live world away from the reviewed configuration: a loosened security context, a deleted warm pool, a traffic split nudged by hand during an incident and never restored.

§ IVWorked Example — One Configuration That Provisions the Week

Consider a single Terraform configuration that stands up the operate-and-recover posture for one production service: a model-backed agent that serves predictions, scales with load, runs confined, and records its work. Each of the week's four lessons provisions one face of its safety.

The configuration opens by declaring its providers and pinning their versions. The google provider is pinned to the project and region that host the Vertex endpoint. The aws provider is pinned to the serving region. The kubernetes provider points at the cluster that runs the agent's sidecars. The Terraform block pins each provider to a version range and configures a remote backend with state locking, because the recovery the configuration promises is only as trustworthy as the state that records what was provisioned.

The Vertex rollback face comes first. A google_vertex_ai_endpoint resource holds the served model, and the deployed-model block declares the traffic split between the current version and the prior good one. This is Monday's lesson as configuration: the split is a declared weight, so a rollback is a one-line edit moving the weight from the regressed version to the good one, reviewed in a pull request and applied in seconds. The endpoint never gets rebuilt.

The AWS autoscaling face comes next. An aws_autoscaling_group declares the serving fleet, an aws_autoscaling_policy of type target-tracking holds a chosen metric at its setpoint, and a warm-pool block keeps pre-initialized instances ready against the cold-start tax. This is Tuesday's lesson as resources: the scaling behavior is declared, so an accidental console change to the setpoint shows as drift and the apply restores the reviewed elasticity.

The Kubernetes confinement face comes third. A kubernetes_deployment for the agent's sidecar declares a security context that drops all capabilities, runs as non-root, mounts a read-only root filesystem, and references a seccomp profile. This is Wednesday's lesson: the least-privilege posture is declared, so a developer who loosens the context during a debugging session leaves a drift line in the next scheduled plan, and the apply re-confines the workload.

The Bedrock tracing face comes last. A trace-and-log configuration declares that the agent's invocations are recorded, an aws_xray_sampling_rule declares the sampling rate, and a log destination declares where the records land. This is Thursday's lesson: the visibility is declared rather than clicked, so the tracing cannot be silently turned off without the change appearing as drift.

The four faces apply in one terraform apply. The operator runs one command and the agent gets a rollback-ready endpoint, an elastic fleet, a confined sidecar, and a recorded trace. More to the point, the operator runs terraform plan the next morning and sees, in one diff, whether any of the four drifted overnight: a traffic split nudged toward a bad version, a warm pool deleted to save cost, a security context loosened in a hurry, a sampling rule zeroed. Each shows as a line in the plan, and the apply that follows restores the reviewed posture. The recovery is the diff, read and applied.

§ VPractice Scenarios

The first scenario tests rollback-as-configuration. A model version regresses in production at noon. The console offers a button; the configuration offers a reviewed edit. The answer is that the traffic-split weight is declared, so the rollback is a one-line change reviewed in a pull request and applied in seconds, leaving an audit trail of who rolled back and when, where the console button leaves no reviewed record.

The second scenario tests the module as shared posture. Twenty services each need the same confined-workload security context, and a new CVE means the context must tighten for all twenty. The answer is a confined-workload module that declares the context once; bumping the module version and correcting the context once rolls the fix to all twenty as reviewed version bumps rather than twenty hand-edits that drift apart.

The third scenario tests state hygiene under an incident. A region degrades, an on-call engineer applies a scaling change, and a teammate applies a conflicting confinement change a minute later. The answer is the remote backend's state lock: the second apply blocks on the lock the first holds, so neither overwrites the other's state, and the recovery posture stays certain at the moment the incident needs it certain.

The fourth scenario tests the scheduled plan as drift detector. A scheduled terraform plan returns a non-empty diff showing a kubernetes_deployment will change its security context back to drop-all-capabilities. The answer is that someone loosened the context by hand and never restored it; the scheduled plan caught the loosened confinement within a day, and treating any non-empty scheduled plan as an alert turns the week's four safeties into monitored configuration rather than hope.

The fifth scenario ties the week to today's Ops lesson. The trading system's regime engine flips to risk-off and the kill-switch pulls exposure; the operator wants the serving infrastructure for the paused strategies scaled to zero until the regime recovers, then restored. The answer is that each strategy's serving fleet is a module instance with a desired-count input; the risk-off posture sets those inputs to zero in a reviewed change, and the regime recovery restores them, so the scale-down and scale-up are both declared moves an apply executes. The regime engine decides; the apply provisions the decision.

§ VIPractice Questions

§ VIIClosing

The week studied four vendors and found one kind of work: recovery, provisioned. Vertex recovers a model by editing a declared traffic split. AWS recovers capacity by a declared scaling policy. Kubernetes recovers least privilege the moment a scheduled plan catches a loosened context. Bedrock recovers the record of what an agent did because the tracing is declared and cannot be silently turned off. Four recovery moves, every one of them configuration, and Terraform is where that configuration lives as reviewed, versioned, drift-checked declaration across every vendor the week touched.

Recovery clicked into a console is recovery no one can prove is still in place. Recovery declared in Terraform is recovery whose absence is a line in tomorrow's plan. Provision the recovery posture as code, plan it on a schedule, and read the diff. The diff is the week's recovery reporting for duty.

Examine well. Reflect on this.