Terraform as the Guardrail-Provisioning Foundation

Week 9 reach-back synthesis of Vertex cost attribution, AWS budget enforcement, and Kubernetes workload identity.

A budget threshold in a spreadsheet is a hope. The same threshold in a .tf file, applied alongside the workload it constrains, is a fact about the account. The difference is whether the guardrail exists on the day the workload does, or three weeks later when the invoice arrives.

Every Friday this cycle pulls the week's cert lessons through Terraform's lens and asks one question: how would you deploy what you learned this week as infrastructure, reproducibly, with the safety each lesson demanded.

§ IFrame

Week 9 gives three lessons instead of four, and the missing one is worth naming before the synthesis starts. Thursday's Fajr cycle did not fire. The scheduler runs on a New-York clock against expressions authored in Chicago time, and the drift has now cost two days. There is no AWS AIP-C01 / GH-600 agentic lesson for this week. The Friday discipline reaches back over what shipped rather than over what should have.

What shipped shares one shape. Monday built cost attribution on Vertex: billing export to BigQuery, per-model token accounting, an alert that fires when a budget threshold breaks. Tuesday built enforcement on AWS: Budgets Actions that attach a deny policy when spend crosses a line, Cost Anomaly Detection that watches the shape rather than the total, Service Control Policies that make the expensive thing unavailable rather than merely discouraged. Wednesday built identity on Kubernetes: a ServiceAccount, an RBAC binding, and a bound projected token scoped to one audience with a short life, so a leaked token buys nothing outside its blast radius.

Three vendors, three uniforms, one idea. Each lesson builds a thing whose only job is to say no.

Today's Ops lesson happens to build a guardrail of exactly this shape at a different layer: a gate that can only refuse, never grant. Three cloud lessons and one market lesson converged on the same architecture in the same week without coordinating. That is worth noticing.

§ IIWeek in Review

Monday — Vertex AI cost observability and attribution

The Google lesson's core move was making spend legible before making it enforceable. Billing export to BigQuery turns an invoice into a queryable table. Per-model token accounting attributes cost to the specific model, the specific caller, and the specific request class rather than to a single line item labeled "Vertex AI." Budget alerts fire on threshold breach.

The lesson's discipline was that attribution precedes enforcement. You cannot cap what you cannot attribute, because a cap on an unattributed total either strangles the whole platform or protects nothing.

Tuesday — AWS cost governance and budget enforcement

The AWS lesson supplied the enforcement Monday's attribution made possible. Budgets Actions attach an IAM deny policy when spend crosses a threshold, converting an alert into a stop. Cost Anomaly Detection reads the shape of spend rather than its level, catching a runaway before the monthly total notices. Service Control Policies sit above the account and remove the ability entirely, so the guardrail does not depend on the workload's own good behavior.

The lesson's discipline was the ladder: alert, then throttle, then deny. Each rung is a different answer to how much you trust the thing you are constraining.

Wednesday — Kubernetes workload identity and bound service account tokens

The CKA half established the identity plumbing: a ServiceAccount is an identity, a Role names permissions, a RoleBinding joins them. The CKS half hardened it: the legacy long-lived Secret-mounted token is replaced by a bound projected token issued through the TokenRequest API, scoped to a specific audience, tied to the pod's lifetime, and expiring on a short clock.

The lesson's discipline was blast radius. A token that is valid forever, for every audience, is a credential that turns any single compromise into a full compromise. Audience scoping and short expiry make the same leak local and temporary.

Thursday — did not ship

The agentic-AI lesson for Week 9 does not exist. Named here rather than papered over, because a reach-back synthesis that invents a fourth leg is exactly the failure today's Ops lesson spent 2,500 words describing.

§ IIITerraform as Connective Tissue

Brikman's module chapter opens on the argument the whole week needs (Terraform: Up and Running, 3ed, Ch 4, "Module Basics," pp. 201–202): a module is a container for resources used together, and its value is that the collection becomes one reusable, versioned unit rather than a pattern people retype. He makes the point about environments, where the copy-paste of a whole staging stack into production is the thing modules eliminate.

Apply that to guardrails and something sharper falls out. A guardrail is exactly the kind of resource that gets retyped, forgotten, and drifted, because it is nobody's feature. The budget alarm ships with the workload the first time, because someone remembered. The second workload ships without it, because that someone was on vacation. By the fourth workload, the guardrail is a convention rather than a fact, and a convention is what an invoice discovers.

Put the guardrail inside the module and the arithmetic changes. The workload module and the guardrail module compose in the root, and there is no way to provision the workload without provisioning the thing that constrains it, because the root module wires both in one apply.

The composition, per vendor

The Vertex piece declares the billing export dataset, the budget, and its threshold rules as resources. The attribution Monday taught is a google_billing_budget with a filter on service and label, which means the label the workload carries is what makes the budget attributable. That label is passed from the workload module's output into the budget module's input, so the dependency is a real graph edge rather than a naming convention someone honors.

The AWS piece declares the budget, the action, the anomaly monitor, and the SCP. Tuesday's ladder becomes three resources at three scopes: aws_budgets_budget with a notification for the alert rung, aws_budgets_budget_action with an IAM policy ARN for the throttle rung, and an aws_organizations_policy of type SERVICE_CONTROL_POLICY for the deny rung. The SCP is the interesting one, because it lives at a different account scope than the workload and therefore in a different state file.

The Kubernetes piece declares the ServiceAccount, the Role, the RoleBinding, and the pod spec's projected-token volume with its audience and expiration. Wednesday's bound token is a kubernetes_service_account plus a volume projection in the deployment, and Terraform's job is guaranteeing the binding exists before the workload that assumes it.

The state-isolation point the SCP forces

Brikman's state-isolation treatment (Ch 3, "State File Isolation," pp. 161–162) argues that isolating state by environment and by component limits the blast radius of a mistake: a botched apply against one state file cannot damage what another state file owns.

The SCP makes this concrete in a way the week's other resources do not. An SCP applies to a whole organizational unit and is managed by the account that owns the organization, not by the team deploying the workload. Putting the SCP in the workload's state file gives the workload's apply permission to remove its own ceiling. The guardrail becomes editable by the thing it guards, which is not a guardrail.

So the SCP lives in a separate state file with separate credentials, and the workload module reads it through a data source rather than owning it. The workload can see its ceiling. It cannot raise it.

That is the same shape as today's Rust lesson, where Claim<Unattributable> has no method that returns a claim. The refusal is enforced by structure rather than by policy, and the thing being constrained has no lever to reach.

Why this takes longer than it looks

Brikman on production-grade infrastructure (Ch 8, pp. 422–423) makes the point that shipping the happy path is a small fraction of the work, and that the gap between a demo and production is filled with exactly the unglamorous surface this week describes.

Week 9's three lessons are that gap, itemized. Nobody demos a budget alarm. Nobody's roadmap has a line item for audience-scoped tokens. Each one is invisible while it works and expensive on the day it was missing.

§ IVWorked Example

A root module that provisions one model-serving workload with all three of the week's guardrails wired in. The workload's own definition is elided; the guardrails are the lesson.

Start with the SCP, in its own state, owned by the organization account. The workload never sees this file.

resource "aws_organizations_policy" "no_expensive_instances" {
  name = "deny-p5-outside-approved-ou"
  type = "SERVICE_CONTROL_POLICY"

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Deny"
      Action   = ["ec2:RunInstances"]
      Resource = "arn:aws:ec2:*:*:instance/*"
      Condition = {
        StringLike = { "ec2:InstanceType" = ["p5.*"] }
      }
    }]
  })
}

The workload's root module reads that policy rather than owning it. A data source is a read, and a read cannot delete.

data "aws_organizations_policy" "ceiling" {
  policy_id = var.scp_policy_id
}

Now the workload module, which emits the two outputs every guardrail downstream needs: the label that makes its cost attributable, and the service account that carries its identity.

module "serving" {
  source = "../modules/model-serving"

  model_id   = var.model_id
  cost_label = "workload=${var.workload_name}"
  namespace  = var.namespace
}

The Vertex budget module takes the label as an input. The reference is what makes the graph create the workload first and the budget second.

module "vertex_budget" {
  source = "../modules/vertex-budget"

  billing_account   = var.billing_account
  budget_amount_usd = var.monthly_cap_usd
  filter_label      = module.serving.cost_label
  threshold_rules   = [0.5, 0.9, 1.0]
  alert_topic       = local.on_threshold
}

The alert target is derived from environment rather than set independently, which keeps a dev-environment threshold breach off a human's pager.

locals {
  on_threshold = var.env == "prod" ? google_pubsub_topic.pager.id : google_pubsub_topic.log_only.id
}

The AWS enforcement module implements Tuesday's ladder. The deny rung attaches a policy at a spend threshold, and the anomaly monitor watches shape independently of the level.

module "aws_enforcement" {
  source = "../modules/aws-cost-guardrail"

  budget_amount_usd  = var.monthly_cap_usd
  alert_threshold    = 0.5
  throttle_threshold = 0.9
  deny_policy_arn    = aws_iam_policy.freeze_inference.arn
  cost_filter_tag    = module.serving.cost_label
  anomaly_monitor    = true
}

The Kubernetes identity module takes the namespace from the workload and issues the bound token Wednesday's lesson taught.

module "workload_identity" {
  source = "../modules/k8s-bound-identity"

  namespace            = module.serving.namespace
  service_account_name = "${var.workload_name}-sa"
  audience             = "vertex-inference"
  token_expiry_seconds = 3600
  role_rules = [{
    api_groups = [""]
    resources  = ["configmaps"]
    verbs      = ["get", "list"]
  }]
}

The pod spec projects that token rather than mounting a Secret. The audience is what makes the token useless anywhere else.

resource "kubernetes_deployment" "serving" {
  metadata {
    name      = var.workload_name
    namespace = module.serving.namespace
  }

  spec {
    template {
      spec {
        service_account_name = module.workload_identity.service_account_name

        volume {
          name = "vertex-token"
          projected {
            sources {
              service_account_token {
                path               = "token"
                audience           = "vertex-inference"
                expiration_seconds = 3600
              }
            }
          }
        }
      }
    }
  }
}

Read what the graph now guarantees. The workload cannot exist without a budget filtered on its own label, because the budget module consumes an output the workload produces. It cannot exist without the AWS ladder, for the same reason. Its pod cannot start without the ServiceAccount, because the deployment references it. And it cannot raise its own ceiling, because the SCP lives in a state file its credentials cannot write.

One apply. Four refusals, all provisioned.

§ VPractice Scenarios

§ VIConnection to Today's Ops and Dev Lessons

Three layers, one architecture, and they arrived in the same cycle without planning it.

The Ops lesson builds an Attribution Gate whose only verdict is refuse, and whose stated weakness is that a procedure can be skipped. The Rust lesson removes the skip by making the unverified claim a type the sizer cannot accept. This lesson removes the skip at a third layer: a guardrail that is a resource in the same apply as the workload cannot be forgotten, and a ceiling in a state file the workload cannot write cannot be raised by the thing it constrains.

The shared principle is worth stating plainly, because it is the week's real lesson. A constraint enforced by memory is not a constraint. Move it into a structure that has no lever: a type with no method, a module the root wires unconditionally, a state file with different credentials. Then the constraint holds on the day everyone is tired.

§ VIIClosing

Week 9 shipped three lessons and lost one to a clock nobody has fixed yet. The three that shipped all describe machinery that says no, and Terraform is where that machinery stops being a plan and becomes an account fact.

Attribute before you cap. Ladder the enforcement from alert to throttle to deny. Scope the token to one audience and one hour. Then put every one of those in a module the workload's root wires without asking, and put the ceiling in a state file the workload cannot reach.

Read your own root module and find the guardrail that is documented but not declared. That one is the next invoice.