Building a Kubernetes Admission Webhook in Python — FastAPI, the AdmissionReview Contract, and the Fail-Closed Decision

The controller is a night watchman walking rounds. The webhook is a doorman with a queue forming behind every hesitation.

What does the API server want from your webhook? One JSON document in, one JSON document out, and the request's own uid quoted back. Everything else is your judgment.

§ IFrame

Thursday you wrote a controller: a loop that watches the ledger and repairs the world after the fact. Today you write the other kind of Kubernetes program, the one that stands at the door. Today's Ops lesson walked the admission chain from the outside; this lesson builds one of its links, a validating-plus-mutating webhook server in FastAPI that enforces a single rule: containers must not run as root.

The two programs could not be more different in temperament, and the difference teaches the platform. A controller is patient. It can retry forever, because level-triggered reconciliation forgives a missed event. A webhook has no such luxury. The API server is standing in the doorway holding someone's kubectl apply, blocking, waiting for your verdict, and your timeoutSeconds budget is single digits. The controller you wrote Thursday is a night watchman walking rounds. The webhook is a doorman with a queue forming behind every hesitation. Write it accordingly: fast, deterministic, and boring.

§ IILanguage Idiom

The contract first. The API server POSTs an AdmissionReview JSON document whose request field carries a uid, the operation, and the full object under review. Your response is another AdmissionReview, and it must contain two things or the request dies: allowed, and the same uid echoed back. Name that second requirement the uid echo. The echo exists so a confused or malicious webhook cannot answer a question it was not asked; the API server matches verdict to request by uid and discards a mismatch. Forget the echo and every admission attempt fails with a cryptic error while your handler returns 200 the whole time. It is the first thing to check when a webhook misbehaves, and the last thing most people look at.

Pydantic gives Python the right idiom for a contract like this. Model the fragment of AdmissionReview you touch, let FastAPI parse and reject malformed payloads at the framework boundary, and keep your handler working with typed attributes rather than nested dict access.

from fastapi import FastAPI
from pydantic import BaseModel, Field

class AdmissionRequest(BaseModel):
    uid: str
    operation: str
    object: dict = Field(default_factory=dict)

class AdmissionReview(BaseModel):
    apiVersion: str = "admission.k8s.io/v1"
    kind: str = "AdmissionReview"
    request: AdmissionRequest | None = None
    response: dict | None = None

app = FastAPI()

The response helper is where the uid echo becomes structural instead of remembered. Build every verdict through one function and the echo cannot be forgotten, the same move the exception-hierarchy lesson made on Friday: put the invariant in the type, not in the caller's discipline.

def verdict(uid: str, allowed: bool, message: str = "") -> dict:
    response: dict = {"uid": uid, "allowed": allowed}
    if not allowed:
        response["status"] = {"code": 403, "message": message}
    return {
        "apiVersion": "admission.k8s.io/v1",
        "kind": "AdmissionReview",
        "response": response,
    }

§ IIICode Worked Example

The validating endpoint reads the Pod spec and refuses root. The rule mirrors what the restricted Pod Security profile demands, which is the cert lesson's territory today; here it is one explicit check you can read in ten seconds.

@app.post("/validate")
def validate(review: AdmissionReview) -> dict:
    req = review.request
    if req is None:
        return verdict("", False, "empty admission request")
    spec = req.object.get("spec", {})
    pod_sc = spec.get("securityContext", {})
    for container in spec.get("containers", []):
        sc = {**pod_sc, **container.get("securityContext", {})}
        if not sc.get("runAsNonRoot", False):
            name = container.get("name", "unnamed")
            return verdict(req.uid, False, f"container {name} must set runAsNonRoot")
    return verdict(req.uid, True)

Three details carry the lesson. The merge expression {**pod_sc, **container.get(...)} implements Kubernetes precedence in one line: container-level securityContext overrides pod-level, so the check reads the effective value, the same resolution rule the CKS expects you to know. The refusal message names the container, because an admission denial is an error message someone reads at deploy time, and a denial that does not say which container fails is a support ticket. And the handler raises nothing. Friday's lesson taught exceptions as the error API between layers; a webhook inverts that instinct. An unhandled exception becomes an HTTP 500, and a 500 hands the decision to the failurePolicy in the webhook configuration. The gate must render its own verdicts, in-band, every time it possibly can. Reserve the crash for the case where you have no uid to echo.

Mutation is the same contract plus a patch. A mutating webhook that fills in a missing runAsNonRoot returns a base64-encoded JSONPatch, and the API server applies it before validation runs, the mutate-then-validate order from the Ops lesson doing its work.

import base64, json

@app.post("/mutate")
def mutate(review: AdmissionReview) -> dict:
    req = review.request
    if req is None:
        return verdict("", False, "empty admission request")
    spec = req.object.get("spec", {})
    if "securityContext" in spec and "runAsNonRoot" in spec["securityContext"]:
        return verdict(req.uid, True)
    patch = [{"op": "add", "path": "/spec/securityContext",
              "value": {"runAsNonRoot": True}}]
    out = verdict(req.uid, True)
    out["response"]["patchType"] = "JSONPatch"
    out["response"]["patch"] = base64.b64encode(json.dumps(patch).encode()).decode()
    return out

One caution travels with every mutating webhook: the API server may call it more than once for the same object when other mutators also fire, so the patch must be idempotent. This handler checks before it adds. Run it twice and the second pass is a no-op. That is the same idempotent-reconcile discipline Thursday's controller lesson coined, surfacing at the door instead of in the loop.

Serving is the unglamorous half. The API server refuses to talk to a webhook over plain HTTP, so uvicorn serves with ssl_keyfile and ssl_certfile, and the ValidatingWebhookConfiguration carries the CA bundle that signed them. In production, cert-manager injects both. The webhook configuration for this server sets failurePolicy: Fail, scoped to labeled namespaces only, with timeoutSeconds: 5: fail-closed because the rule is a security floor, tightly scoped because the Ops lesson's Saturday-outage counterfactual is what an unscoped fail-closed gate does to a cluster.

Test the gate the way the Terraform lesson tested a plan: assert on the verdict artifact, no cluster required.

from fastapi.testclient import TestClient

def review_for(containers: list[dict]) -> dict:
    return {"apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview",
            "request": {"uid": "test-uid-1", "operation": "CREATE",
                        "object": {"spec": {"containers": containers}}}}

def test_root_pod_denied():
    client = TestClient(app)
    body = review_for([{"name": "trader", "securityContext": {}}])
    resp = client.post("/validate", json=body).json()
    assert resp["response"]["uid"] == "test-uid-1"
    assert resp["response"]["allowed"] is False

def test_nonroot_pod_admitted():
    client = TestClient(app)
    body = review_for([{"name": "trader",
                        "securityContext": {"runAsNonRoot": True}}])
    resp = client.post("/validate", json=body).json()
    assert resp["response"]["allowed"] is True

The first assertion in the deny test is the uid echo. Test it explicitly. A webhook that passes every policy test and fails the echo is a webhook that fails in production only, which is the most expensive place to learn the contract.

§ IVConnection to Today's Ops Lesson

The Ops lesson named the chain: mutate, validate schema, validate policy, then the ledger. This server is one link of that chain seen from inside, and every configuration knob the Ops lesson treated as doctrine became a line of code here. The failure posture became the decision to render verdicts in-band instead of raising. The synchronous front-door cost became the doorman temperament: no external calls, no retries, single-digit timeout. The mutate-then-validate order became the reason the mutating handler can trust validation to catch what it declines to fix. Read the two lessons against each other and the YAML stops being configuration and becomes a description of code you have now written.

§ VPrior-Lesson Reach

Three Python lessons stand under this one. Thursday's kopf controller is the temperament contrast: watch-loop patience against doorman immediacy, and its idempotent-reconcile rule reappearing as the idempotent patch. The Terraform testing lesson supplies the method here applied to verdicts: the plan was a testable artifact, and so is an AdmissionReview response; assert on the artifact, not the cloud. Friday's exception-model lesson gave the raise-versus-return judgment this lesson inverts deliberately: exceptions are the error API between your own layers, but at a protocol boundary owned by someone else's failurePolicy, the in-band verdict wins.

§ VIClosing

A webhook is forty lines of Python wearing a cluster's trust. The contract is small: parse the review, judge the object, echo the uid, answer fast. The discipline is smaller still and harder to keep: no side quests in the hot path, idempotent patches, verdicts in-band, the crash reserved for the unanswerable. Build the verdict helper first so the echo is structural. Test the echo before you test the policy.

Take the server above and add one more rule to /validate: refuse any container whose image tag is latest. Write the deny test first, watch it fail, make it pass. The rule is trivial. The point is the motion: contract, verdict, echo, test. Run the motion until it bores you, because boring is what the doorman owes the door.

Paired lessons → Ops 01-Earth-DevOps/Synthesis-Lessons/2026-07-26-the-kubernetes-admission-chain-... · Cert Cert-Prep/CNCF/2026-07-26-cks-pod-security-standards-...