Writing a Kubernetes Controller in Python — the Watch, the Work Queue, and Level-Triggered Reconciliation

An operator is the oldest program Kubernetes runs, written for a noun you invented. Observe, compare, act.

§ IFrame

The Ops lesson this morning named a loop the control plane already runs for the resources Kubernetes ships with. Now you write one of your own. That is the entire idea of an operator: teach the cluster to keep something alive that it did not previously understand, using the same three verbs its own controllers use. Observe, compare, act.

Python is a fine language to learn this in, and not only because the kubernetes client library is mature. The reconciliation pattern is small enough that a language which stays out of your way lets you see the pattern itself rather than the plumbing around it. The Python operator framework kopf, short for Kubernetes Operator Pythonic Framework, handles the watch machinery, the retry logic, and the event stream, and hands you a plain function to fill in. What you write inside that function is the loop. Everything else is scaffolding you get for free.

This lesson builds one operator end to end. A custom resource that declares a wish, a handler that reconciles that wish against reality, and the level-triggered discipline the Ops lesson insisted on, expressed as Python you could run. By the close you will have written the same loop the control plane runs, in about forty lines, and you will understand why every line that looks like a shortcut is actually the trap.

§ IIFoundations — A Custom Resource Is a New Kind of Claim

Yesterday's Ops lesson said a Deployment is a claim: three replicas, always. A custom resource is a claim about something Kubernetes has never heard of. You invent the noun, and then you invent the controller that honors it.

Define the noun with a CustomResourceDefinition. Suppose your team keeps running the same chore by hand: whenever someone creates a project, they also create a namespace, a quota, and a default network policy for it. You want a single object that means all of that. Call it a Tenant.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: tenants.hedronite.io
spec:
  group: hedronite.io
  scope: Cluster
  names:
    kind: Tenant
    plural: tenants
    singular: tenant
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                displayName: { type: string }
                cpuQuota:    { type: string }

Apply that and the API server now accepts Tenant objects and stores them in etcd, exactly as it does for built-in kinds. A user can write kind: Tenant with a displayName and a cpuQuota, and the object will persist. But nothing happens. The claim exists and no one is honoring it. Burns and his coauthors are precise about this in the chapter on extending Kubernetes: a custom resource without a controller is inert data. The CRD gives you a new noun the cluster can store; it does not give you a new behavior. The behavior is the controller you are about to write.

That split is worth holding onto. The CRD is the schema of the claim. The controller is the loop that closes the gap between the claim and the world. They are two separate things you build, and beginners routinely ship the first, see the object save, and wonder why the namespace never appears.

§ IIIMechanism — The Watch, the Handler, and the Work Queue

A controller needs to know when a Tenant object appears, changes, or is deleted. The naive approach is to poll: list all Tenants every few seconds and diff. That works and it is wasteful, and at scale it hammers the API server. The mechanism Kubernetes provides instead is the watch, a long-lived streaming connection to the API server that pushes changes as they happen.

Raw watches are fiddly. Connections drop, resource versions expire, and you must resync the full list on reconnect so you do not miss anything while disconnected. Production controllers wrap the watch in an informer, which maintains a local cache of the objects and a work queue of items needing reconciliation. When an object changes, the informer updates its cache and enqueues the object's key. Worker tasks pull keys off the queue and call your reconcile function. The queue matters for a reason that connects straight back to the Ops lesson: it deduplicates. If the same object changes five times before you get to it, the key sits in the queue once, and your handler runs against the latest state, not five times against five stale snapshots. The work queue is what makes the loop level-triggered in practice.

kopf gives you all of that. You decorate a Python function, name the resource it handles, and kopf runs the watch, the cache, the queue, the retries, and the backoff. Your job is the body of one function.

import kopf
from kubernetes import client

@kopf.on.create('hedronite.io', 'v1', 'tenants')
@kopf.on.update('hedronite.io', 'v1', 'tenants')
@kopf.on.resume('hedronite.io', 'v1', 'tenants')
def reconcile_tenant(spec, name, logger, **kwargs):
    display_name = spec.get('displayName', name)
    cpu_quota = spec.get('cpuQuota', '4')
    ns_name = f'tenant-{name}'

    ensure_namespace(ns_name, display_name, logger)
    ensure_quota(ns_name, cpu_quota, logger)

    return {'namespace': ns_name, 'quota': cpu_quota}

Read the three decorators, because they are the whole edge-versus-level lesson in code. on.create fires when a Tenant is first made. on.update fires when its spec changes. on.resume fires for every existing Tenant when the operator itself starts up. That third one is the one people forget, and forgetting it is the edge-triggered bug the Ops lesson warned about. Without on.resume, an operator that restarts never re-checks the Tenants that already exist. It waits for the next change, and if a namespace was deleted while the operator was down, the gap never closes. With on.resume, the operator re-reconciles every existing claim on startup, which is precisely how the built-in controllers behave: read the full current state on cold start and correct every gap.

§ IVWorked Example — The Reconcile That Survives Its Own Restart

Fill in the two helpers, and write them the way a level-triggered handler has to be written, because this is where the discipline lives.

def ensure_namespace(ns_name, display_name, logger):
    core = client.CoreV1Api()
    body = client.V1Namespace(
        metadata=client.V1ObjectMeta(
            name=ns_name,
            labels={'hedronite.io/tenant': display_name},
        )
    )
    try:
        core.create_namespace(body)
        logger.info(f'created namespace {ns_name}')
    except client.ApiException as e:
        if e.status == 409:            # already exists
            core.patch_namespace(ns_name, body)
            logger.info(f'namespace {ns_name} already present; patched')
        else:
            raise

The shape of that function is the shape of every correct reconcile. It does not ask was this namespace just created or is this a create event or an update event. It asks should this namespace exist in this form, and it makes that true. Create if absent, patch if present, and treat the already-exists error not as a failure but as a normal branch. This is what people mean by writing an idempotent reconcile: run it once or run it a hundred times against the same claim, and the end state is identical.

That property is what lets the handler survive restarts, missed events, and duplicate deliveries, all three of which will happen in production. Kopf may call your handler again after a transient error. The informer may deliver the same key twice. The operator may crash and resume and re-reconcile every Tenant through on.resume. In every one of those cases, an idempotent handler does the right thing, because it never assumes it is running for the first time. It reads the world, compares it to the claim, and closes the gap, which is exactly the loop the control plane runs.

§ VStatus, Finalizers, and Telling the Truth Back

A built-in controller reports what it actually did, and yours should too. When kopf's handler returns a dictionary, kopf writes it into the resource's status subresource. That is how kubectl get tenant can show a NAMESPACE column reflecting reality rather than intent. Status is the observed half of the loop written back where a human, or another controller, can read it. Never store observed state in spec; spec is the claim the user owns, and status is the report the controller owns. Crossing that line is how you get two writers fighting over one field.

Deletion needs its own care, and kopf handles it with a finalizer. If a Tenant owns resources outside its own namespace, deleting the Tenant object should not orphan them. A finalizer is a marker on the object that tells the API server do not actually delete this until I say so. Kopf adds one automatically when you register an on.delete handler, runs your cleanup, then removes the finalizer so the delete completes.

@kopf.on.delete('hedronite.io', 'v1', 'tenants')
def cleanup_tenant(name, logger, **kwargs):
    core = client.CoreV1Api()
    ns_name = f'tenant-{name}'
    try:
        core.delete_namespace(ns_name)
        logger.info(f'deleted namespace {ns_name}')
    except client.ApiException as e:
        if e.status != 404:            # already gone is fine
            raise

The same idempotence discipline applies to teardown. Deleting something already deleted is not an error, so a 404 is a normal branch, not a crash. A finalizer that raises forever on a resource that is already gone will wedge the delete permanently, leaving an object stuck in Terminating that no kubectl delete can clear. Write cleanup as level-triggered as you write creation: make the world match the claim, and here the claim is this should not exist.

§ VIConnection to Today's Ops and Cert Lessons

The Ops lesson gave you the reconciliation loop as the control plane runs it, across the API server, etcd, the controllers, the scheduler, and the kubelet. This lesson put that same loop in your hands: a custom resource is a claim, a controller is the loop that honors it, and level-triggered idempotence is what makes the loop survive the failures that define real operations. The cert lesson closes the triangle from the other direction. When the CKA exam hands you a cluster where a workload will not come up, the fastest path to the fault is knowing that some loop stopped turning, and being able to name which component owns it. You have now written a loop, watched it recover from its own restart, and seen why the shortcut is the bug. That is the same knowledge the exam tests, learned by building it.

§ VIIClosing

An operator is not a new kind of program. It is the oldest program Kubernetes runs, written for a noun you invented. Define the claim as a custom resource, write a handler that reads the world and makes it match, register on.resume so a restart re-checks everything, and catch the already-exists and already-gone cases so running twice is safe. Do that and you have a controller indistinguishable in behavior from the ones shipped in the control plane, because it is the same three verbs on the same clock.

Take the Tenant operator and kill its pod mid-reconcile. Watch it come back, resume every existing Tenant, and close whatever gaps opened while it was down, without a single lost claim. If it does that, you wrote it level-triggered. If it does not, find the on.create you used where you needed on.resume, and the loop will start turning again.

Paired lessons → Ops 01-Earth-DevOps/Synthesis-Lessons/2026-07-23-the-kubernetes-reconciliation-loop-... · Cert Cert-Prep/CNCF/2026-07-23-cka-cluster-architecture-...