Terraform as the Drift-Detection-and-Response Provisioning Foundation
Week 8 reach-back synthesis of model-cascade serving, automated remediation and drift detection, secrets rotation, and agent-regression baselining.
A workload is easy to deploy. A workload you can trust to keep running is a workload you have to watch, rotate, and remediate — the alarm that catches its drift, the schedule that rotates its secrets, the automation that repairs it, the regression gate that scores it over time. Terraform is where the workload and its whole watcher layer become one apply.
Every Friday this cycle pulls the week's four cert lessons through Terraform's lens and asks one question: how would you deploy what you learned this week as infrastructure, reproducibly, with the safety the individual lessons demanded. Week 8's four lessons share a spine that is easy to miss because it wears four different vendor uniforms. Every one of them is about the loop that watches a running system, notices when it drifts, and acts. Terraform is where that loop stops being an afterthought bolted on after launch and becomes a resource the workload declares a dependency on.
§ IFrame
The week taught detection and response from four angles. Monday built model-cascade serving on Vertex: route a request to a cheap model first, and escalate to the expensive one only when a confidence gate says the cheap answer is not good enough. Tuesday built AWS automated remediation: a CloudWatch alarm or a Config rule detects a drifted resource, and a Systems Manager Automation document acts to bring it back into compliance, the detect-decide-act loop. Wednesday built Kubernetes secrets rotation: a secret is not a static object but a lease that must be reissued, drained, and revoked on a schedule before its blast radius widens. Thursday built agent-regression baselining: run a scheduled Bedrock evaluation, compare it to a frozen baseline, and fire a regression alarm when the score drifts, turning evaluation into a time series.
Read as a list they are four exams for four certs. Read as a system they are one operational posture: nothing that runs is trusted to keep running on its own. Each workload ships with a watcher that reads its live behavior, a policy that decides when the behavior has drifted, and an actor that responds. Name the Terraform discipline that provisions this posture the provision-the-watcher-with-the-workload rule: the alarm, the rotation schedule, the remediation document, and the regression gate are declared in the same module as the workload they watch, so a workload cannot ship into an environment that lacks the machinery to watch it. The watcher is not a later addition. It is a resource the workload depends on, ordered by the graph to exist before the workload it guards.
§ IIWeek in Review
Monday — Vertex AI model selection and the cascade serving discipline
The Google lesson built a serving path that routes by cost and confidence: a cheap model answers first, and an escalation gate promotes the request to an expensive model only when signals like finishReason, a schema-parse failure, or low confidence say the cheap answer is inadequate. The ML-Pro flavor governed endpoint sizing and the cost-per-prediction tradeoff; the GenAI-Engineer flavor tuned the gate signals. The infrastructure underneath is two endpoints, a router, and the gate configuration that decides when to climb the ladder.
Tuesday — AWS automated remediation and drift detection
The AWS lesson built the detect-decide-act loop: AWS Config rules and conformance packs detect a resource that has drifted from its declared state, CloudWatch alarms and EventBridge rules decide when the drift crosses a threshold, and Systems Manager Automation documents act to remediate. The SAP flavor centered Config and conformance packs; the DOP flavor wired CloudWatch and EventBridge to SSM Automation for the closed loop. The infrastructure is a set of detectors, the alarms, the automation documents they trigger, and the IAM authority boundary that governs what the automation may touch.
Wednesday — Kubernetes secrets management and rotation
The CNCF lesson closed the secrets lifecycle: CKA supplied the Secret object, its mounting as a volume versus an environment variable, and the reason a volume mount is what makes rotation zero-downtime; CKS layered encryption-at-rest with an EncryptionConfiguration provider order, external secret stores, and short-lived projected service-account tokens with an audience and an expiry. The through-line was that a secret is a lease with a clock, not a constant. The infrastructure is the encryption provider configuration, the external-store integration, and the rotation schedule that reissues the lease before it becomes a liability.
Thursday — agent-regression baselining and regression detection
The AWS lesson turned evaluation into a time series: a scheduled Bedrock agent evaluation job emits a score to a CloudWatch metric, a regression alarm fires when the score drifts below a frozen baseline, and version attribution ties a regression to the model or knowledge-base change that caused it; the GH-600 flavor read Copilot's survival-rate and task-quality drift against its config history. The coined trap was the silent drift — a system that still runs but answers worse, caught only by baselining the eval and watching the trend. The infrastructure is the scheduled eval job, the metric, the baseline, and the regression alarm.
§ IIIThe Terraform Synthesis
Four watchers, four workloads, one provisioning discipline. The cascade router watches confidence and escalates. The Config rule watches drift and remediates. The rotation schedule watches the clock and reissues. The regression alarm watches the trend and fires. In every case the workload is trivial to declare and the watcher is where the safety lives, and in every case the failure mode is the same: someone ships the workload and defers the watcher to "after launch," and the watcher never arrives. The Terraform discipline exists to make that deferral impossible.
Compose the watcher as a module, not a footnote
Brikman's treatment of composable modules (Terraform: Up and Running, Ch 8, pp. 436–438) is the structural key. A production module is not one flat file; it is a composition of smaller modules, each with a single responsibility, wired together by passing one module's outputs into another's inputs. The watcher layer is exactly such a module. A regression_alarm module takes a metric name and a baseline threshold and produces the alarm and its notification target. A rotation_schedule module takes a secret ARN and an interval and produces the schedule and the Lambda that rotates it. The workload module composes these the way a program composes libraries, and the composition is what guarantees the watcher ships with the workload rather than trailing it.
Wire the watcher to the workload by reference, so the graph orders it
Brikman's dependency-injection point (Ch 9, pp. 505–506) is what makes the composition safe. The watcher module must reference the workload it guards — the regression alarm references the endpoint's metric, the rotation schedule references the secret's ARN, the remediation document references the Config rule's resource type. Those references are dependency edges, and Terraform's graph orders resources by their edges: the workload is created before the alarm that watches it, and torn down after, in reverse. Hard-code the reference as a literal string to speed up the plan and you sever the edge — now a parallel apply can create the alarm before the workload exists, or a destroy can delete the workload while the alarm still points at a metric that no longer emits. The reference is the mechanism that keeps the watcher and the workload lifecycle-coupled.
Tie enforcement to environment with a computed local
Brikman's conditional-expression pattern (Ch 5, p. 233) supplies the last piece: the strictness of a watcher should be derived from the environment, not chosen independently. A regression alarm in prod pages a human; in dev it logs. A remediation document in prod auto-repairs; in dev it only reports. A rotation interval in prod is short; in dev it is relaxed for iteration. Express each as a local computed from the env input, so the watcher tightens exactly where it matters and no operator can set the strictness inconsistently with the environment it runs in.
§ IVWorked Provisioning Sketch
Picture the week as one root module. It composes four workload-plus-watcher pairs. The cascade_serving module declares the two Vertex endpoints and a router_gate submodule whose escalation thresholds come from a local keyed to environment. The remediation module declares the Config rules and, wired by reference to each rule's resource, the SSM Automation documents and CloudWatch alarms that trigger them. The secrets module declares the encryption configuration and, referencing each secret's ARN, a rotation schedule whose interval is a local computed from environment. The agent_eval module declares the scheduled Bedrock evaluation and, referencing its output metric, a regression alarm whose baseline is a variable and whose notification target escalates in prod.
Run terraform plan. The graph shows every watcher downstream of the workload it guards: endpoints before router gate, Config rules before remediation documents, secrets before rotation schedules, eval jobs before regression alarms. Run terraform apply. The workloads come up first, then their watchers, each wired to the live resource it observes. Run terraform destroy. The watchers tear down first, in reverse, so no alarm ever outlives the metric it reads and no rotation schedule ever fires against a secret that is gone. The whole detection-and-response posture of the week is one reproducible apply, and no workload in it can reach an environment that lacks the machinery to watch it.
§ VPractice Scenarios
You have a cascade_serving workload module and a separate regression_alarm module. A teammate wants to put the alarm resources directly inside the serving module "so there's one less module." What does the composable-module discipline say, and what do you lose by inlining?
Keep them separate and compose them. Brikman's composable-module point (Ch 8, pp. 436–438) is that a module should have a single responsibility, and "serve the model" and "watch the model for regression" are two. Inlining couples them so the alarm cannot be reused for a different workload, cannot be tested in isolation, and cannot be versioned independently. Composition — the serving module's metric output passed into the alarm module's input — keeps each module single-purpose while still guaranteeing the alarm ships with the serving stack, because the root module wires them together in one apply.
The rotation_schedule module references kubernetes_secret.db.metadata[0].name. A teammate proposes passing the secret name as a hard-coded string to avoid a cross-module output. What breaks?
Hard-coding severs the dependency edge, so Terraform no longer knows the rotation schedule depends on the secret. A parallel apply could create the schedule before the secret exists, and a terraform destroy could delete the secret while the schedule still fires against it. The reference is what makes the graph create the secret first and the schedule second, and tear them down in reverse (Brikman Ch 9, dependency injection). Passing by reference also lets a test substitute a different secret by changing the referenced resource.
The regression alarm's action is local.on_regression = var.env == "prod" ? aws_sns_topic.pager.arn : aws_sns_topic.log_only.arn. Explain why this is a local and not a variable, and the risk if the default were the pager everywhere.
The action is derived from env, not chosen independently, so it belongs in a local computed from the existing input rather than a second variable an operator could set inconsistently with env (Brikman Ch 5). If the default were the pager everywhere, every dev-environment regression during normal iteration would page a human, training the team to ignore the pager — so a real prod regression gets ignored too. Keying the action to environment puts the human interrupt exactly where a regression matters and nowhere it does not.
The week's four workloads span Google, AWS, and Kubernetes providers. The root module composes a watcher for each. How does Terraform coordinate watchers and workloads across three providers that never call each other?
AnswerTerraform coordinates them through the state and the dependency graph, not through inter-provider calls. Each watcher references the metric, ARN, or resource name of the workload it guards, regardless of provider, and the graph orders resources by those reference edges across provider boundaries. The runtime behavior — Config remediating an AWS resource, a Lambda reissuing a Kubernetes secret — is provider-native application logic; Terraform's job is provisioning-time coordination, guaranteeing every workload and its watcher exist and are wired before anything runs.
The agent_eval module stores a frozen baseline threshold that must survive re-applies and must not be silently reset when the module is updated. Where does the baseline live, and what protects it?
The baseline is an input the operator sets deliberately and Terraform records in state; it survives re-applies because state persists it, and it is not reset by a module update unless the operator changes the variable. If the baseline is a resource that must not be replaced when incidental attributes change, a lifecycle { ignore_changes = [...] } or prevent_destroy guards it. The state is the durable record of the agreed baseline, which is exactly why the backend must be shared-storage-with-locking so two applies cannot corrupt it — the same state discipline the Week 7 reach-back closed on.
§ VIClosing
The week taught four watchers wearing four vendor uniforms, and the Terraform reading is that they are one discipline. A workload nobody watches drifts silently — the model degrades, the resource falls out of compliance, the secret outlives its blast-radius window, the agent answers worse than it did last month. The defense is to provision the watcher with the workload, composed as its own module, wired to the workload by reference so the graph orders it, and keyed to environment so its strictness tightens where it matters. Provision-the-watcher-with-the-workload is the same rule as last week's provision-the-evaluation-before-the-agent, one step later in the lifecycle: last week the evaluation had to exist before the agent shipped; this week the watcher has to keep existing as long as the workload runs.
A workload without its watcher is a workload you are trusting to never drift. Terraform is where you refuse that trust and make the watcher a dependency instead.