The Kubernetes Admission Chain — Mutation, Validation, and the Gate Before the Ledger

The loops repair the world the ledger describes. The admission chain decides what the ledger is allowed to say.

Where does cluster policy live? Not in the loops that repair the world. In the gate that decides which claims are allowed to enter it.

§ IFrame

Thursday's lesson gave you the control plane as a room of reconciliation loops, each one reading desired state from etcd and pulling the world toward it. That picture had a deliberate gap. It treated every claim in etcd as legitimate, the way a carpenter treats a work order as signed. Today closes the gap. Between a kubectl apply and the write to etcd stands a fixed chain of gates, and everything the cluster will ever refuse to run is refused there, before the ledger records it.

This is the admission chain. Master it and three things that look separate become one mechanism: the built-in quota and limit enforcement you meet in week one, the Pod Security Admission the CKS drills, and the custom webhooks every serious platform team eventually writes. All three are the same move at different depths. A request arrives, authenticated and authorized, and the chain asks a final question the earlier stages never asked: should this object, in this form, be allowed to exist?

The reconciliation lesson taught that desired state is a claim, not a command. Today's corollary: a claim must earn its place in the ledger. The loops downstream never re-litigate that decision. Once a spec rests in etcd, kubelets will fight to make it real. So the admission chain is where a cluster's judgment lives, and it runs exactly once per write.

§ IIFoundations

Walk the path of a single request. kubectl apply sends a Pod spec to the API server. Authentication answers who are you and dies if it cannot. Authorization answers may this identity perform this verb on this resource, the RBAC question the workload-identity lesson walked two weeks ago. Both gates examine the requester. Neither examines the object. A fully authenticated, fully authorized user can still submit a Pod that mounts the host filesystem and runs as root, and authn and authz will wave it through, because inspecting the payload was never their job.

Admission is the stage that reads the payload. Burns and his coauthors, in the policy-and-governance chapter that grounds this lesson, define it plainly: admission controllers sit in the request path after authorization and before the write, and they are the only stage that can both inspect and change what is being written. Poulton's API-security chapter makes the same cut from the other side. Authentication and authorization are about the subject. Admission is about the object.

The chain has a fixed internal order, and the order is the doctrine. Mutating admission runs first. Then schema validation. Then validating admission. Last, and only if every gate passed, the write to etcd. Name it mutate-then-validate: whatever the mutators did to the object, the validators see the finished form, so no mutation can smuggle a change past the final check. Get the order backward in your head and half of admission behavior stops making sense; keep it, and the design is obvious. You inspect the object you are actually about to store.

One more distinction and the foundations are set. Gates come in two builds. In-tree admission plugins are compiled into the API server: NamespaceLifecycle refuses writes into a namespace being deleted, LimitRanger fills in default resource limits, ResourceQuota counts what a namespace is consuming and refuses the write that would exceed it, and Pod Security Admission holds the pod-security floor the cert lesson drills today. Dynamic admission is the extension point: MutatingWebhookConfiguration and ValidatingWebhookConfiguration objects that tell the API server to call your HTTPS endpoint with the object and wait for a verdict. The first build ships with the cluster. The second is where your judgment enters the chain, and it is what today's Dev lesson has you write in Python.

§ IIIMechanism

Consider the machinery of a webhook call, because its operational costs all trace back to this shape. When a matching request reaches the webhook stage, the API server serializes the object into an AdmissionReview, POSTs it to the registered endpoint over TLS, and blocks the request until the webhook answers or the timeout expires. The response carries three things that matter: the echoed request UID, an allowed verdict, and, for mutating webhooks, a patch. That is the whole protocol. Everything else is configuration about when it fires and what happens when it fails.

When it fires is scope. A webhook configuration matches on operations, resources, and namespaces. Scope discipline is the first thing that separates a production admission setup from an outage generator: match the narrowest set of resources you can defend, and exclude kube-system unless you have a specific reason not to. A webhook that gates all Pod creation everywhere has put itself between the control plane and its own repair tools.

What happens when it fails is the deeper question, and it deserves its own name: the two failure postures. failurePolicy: Ignore means the gate fails open. The webhook is down, the API server shrugs, the write proceeds unexamined. failurePolicy: Fail means the gate fails closed. The webhook is down and every matching write is refused until it returns. Fail-open trades enforcement for availability; an attacker who can crash your webhook has disabled your policy. Fail-closed trades availability for enforcement; an operator who deploys a broken webhook has frozen every matching write in the cluster, sometimes including the write that would fix the webhook. Neither posture is correct in general. The discipline is to choose per gate, out loud: security-critical validators fail closed with tight scope, convenience mutators fail open. Write the choice into the configuration and the runbook both.

There is a newer road that avoids the network hop entirely. ValidatingAdmissionPolicy puts a CEL expression inside the API server itself: no webhook deployment, no TLS, no failure posture to agonize over, because there is no remote endpoint to fail. Where a rule is expressible in CEL against the object alone, prefer it. Reserve webhooks for verdicts that need computation or context CEL cannot reach. The best gate is the one with the fewest moving parts.

Cost closes the mechanism. Every webhook you register adds a synchronous network call inside the API server's write path. Latency here is not like latency in a workload; it is latency on the front door of the entire cluster, felt by every controller and every deploy pipeline at once. Set timeoutSeconds low, single digits. Watch the apiserver_admission_webhook_admission_duration_seconds metric the way you watch etcd health. A slow gate is a cluster-wide brake.

§ IVWorked Example

Ship a real gate the disciplined way. The rule: no image from outside registry.hedronite.internal runs in the trading namespace. The wrong move is to write the policy and enforce it in one afternoon. The right move is a rollout ladder.

First, express the rule in CEL, since it needs nothing but the object:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: internal-registry-only
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  - expression: "object.spec.containers.all(c, c.image.startsWith('registry.hedronite.internal/'))"
    message: "trading namespace admits internal-registry images only"

Second, bind it in audit mode before enforce mode. A ValidatingAdmissionPolicyBinding with validationActions: [Audit, Warn] scoped to the trading namespace turns every would-be violation into an audit-log entry and a client warning while refusing nothing. Run it that way for a week. The audit log now tells you the truth about what enforcement will break: the forgotten cron job pulling from Docker Hub, the debug DaemonSet someone applies by hand. Fix those while nothing is on fire.

Third, flip the binding to validationActions: [Deny]. Nothing changes on flip day, because the ladder already surfaced every violator. That quiet flip is the signature of the method. An admission rollout that surprises anyone was run backward.

Now the counterfactual that keeps the failure postures honest. Suppose the same rule had shipped as a webhook with failurePolicy: Fail scoped to all namespaces, and the webhook's Deployment died on a Saturday. Every Pod create in the cluster now fails, including the replacement webhook Pod itself if it matches the scope, and the on-call engineer is learning about admission ordering at 03:00. Scope exclusions and the namespace selector are not polish. They are the difference between a gate and a trap. Examine the blast radius of every gate before the cluster does it for you.

§ VConnection to Prior Lessons

Thursday's reconciliation lesson is the direct foundation: the loops act on the ledger, and today named the gate that decides what enters the ledger. Hold both and the control plane's full shape appears. Judgment at the door, persistence in the loops. The supply-chain lesson from three weeks back now snaps into place as one instance of this chain, image-signature verification being a validating webhook with a specific question. And the workload-identity lesson supplied the stages upstream: bound tokens answer who, RBAC answers may, admission answers should. Three questions, three stages, one request path.

§ VIConnection to Today's Dev and Cert Lessons

Today's Dev lesson builds the other end of the wire: a FastAPI webhook server that speaks the AdmissionReview contract, echoes the UID, and returns verdicts and patches, with the fail-closed decision made in code rather than in YAML. Today's cert lesson stands inside the chain at its most examined link, Pod Security Admission, the in-tree gate that holds the pod-security floor per namespace. Ops names the chain. Dev builds a link. Cert drills the link the exam weighs heaviest.

§ VIIClosing

The admission chain is the cluster's judgment, exercised once per write, in fixed order: mutate, validate schema, validate policy, then and only then the ledger. In-tree plugins hold the common floor. CEL policies hold the rules that need no outside context. Webhooks hold your judgment, at the price of a synchronous call on the cluster's front door and a failure posture you must choose with eyes open. Roll every gate up the audit-warn-enforce ladder, scope it to the narrowest defensible match, and read its latency metric as front-door telemetry.

Before the next K8s day, find one rule your cluster enforces by convention and convert it to a ValidatingAdmissionPolicy in audit mode. Read the audit log after a day. The gap between what the convention claims and what the audit records is the measure of how much judgment your cluster has been outsourcing to hope. Examine it well.

Paired lessons → Dev Polyglot-Dev/Python/2026-07-26-building-a-kubernetes-admission-webhook-in-python-... · Cert Cert-Prep/CNCF/2026-07-26-cks-pod-security-standards-...