The Kubernetes Reconciliation Loop — Desired State and the Control Plane

A Terraform run ends. A Kubernetes cluster never does. It spends its life closing the gap between what you claimed and what is true.

§ IFrame

A Terraform run has a beginning and an end. You type apply, the tool changes the world, the tool stops. Kubernetes has no such stop. You hand the cluster a description of what you want, and the cluster spends the rest of its life trying to make the world match that description, checking again the moment it succeeds, and starting over the instant anything drifts. Yesterday's Terraform lesson opened on the state file, the record the tool keeps of what it built. Kubernetes keeps a record too, but the record is not the interesting part. The loop that reads the record and acts on it, without ever being asked twice, is the whole machine.

This is the concept the K8s track has to open on, because every other Kubernetes idea is a special case of it. A Deployment is a loop that keeps a set number of Pods alive. A Service is a loop that keeps a routing table pointed at healthy Pods. An operator you write yourself, which the Dev lesson does later today, is a loop you taught to keep something of your own alive. Learn the loop once, at the grain the control plane runs it, and the rest of Kubernetes stops being a pile of resource types and becomes one pattern wearing many names.

The trio today holds to that single idea. This Ops lesson names the loop and the control-plane parts that run it. The Dev lesson writes a loop of your own in Python. The cert lesson drills the same architecture in the shape the CKA exam asks it, where the fastest way to fix a broken cluster is to know which loop stopped turning.

§ IIFoundations — Desired State Is a Claim, Not a Command

Start with the sentence that separates Kubernetes from every tool that came before it. You do not tell Kubernetes what to do. You tell it what should be true.

When you run kubectl apply -f deployment.yaml, you are not issuing an order to start three containers. You are filing a claim: there should be three replicas of this Pod running, always. The word always is the whole difference. An imperative command runs once and is done. A claim is a standing condition the cluster is now responsible for, and it will act on that responsibility for as long as the claim exists, whether the threat to it comes one second from now or one month from now.

Burns and his coauthors frame this in the opening chapter as declarative configuration paired with online self-healing, and the pairing is the point. Declarative configuration alone is just a wish. What makes the wish operational is a system that is online, watching, and self-healing: a system that notices the gap between what you claimed and what is running, and closes it without a human in the path. Take the two apart and neither works. A declaration nobody enforces is a comment. An enforcer with nothing declared has nothing to enforce.

Two consequences follow immediately, and both matter for everything downstream. The first is that Kubernetes is convergent, not procedural. It does not care how the current world came to be wrong; it cares only about the distance between now and the claim, and it moves to shrink that distance. Kill a Pod by hand and the cluster does not file an incident. It notices the replica count fell below the claim and starts a replacement, calmly, because closing that gap is the only thing it was ever doing. The second consequence is that drift is expected, not exceptional. Where Terraform detects drift when you run plan, Kubernetes is detecting drift continuously, as its resting state, and correcting it before you would ever think to look.

§ IIIMechanism — The Control Plane Is a Room of Loops

The reconciliation loop is not one program. It is a pattern that several separate programs each run, and those programs together are the control plane. Name the parts by the job each one does in the loop, and the architecture assembles itself.

The API server is the front door and the single source of truth. Every read and every write in the cluster goes through kube-apiserver. It is the only component that talks to the cluster's memory, and it is the only one anything else talks to. When you kubectl apply, your YAML lands here and nowhere else. When a controller wants to know the current state, it asks here. The API server does not decide anything about your workload. It validates, it stores, and it tells everyone who is watching that something changed. Its power is that it is the sole authority on what the cluster believes.

etcd is the memory. The API server writes desired state and observed state into etcd, a distributed key-value store that is the one stateful component in the control plane. Everything else can be restarted from nothing; etcd is the thing you back up, because etcd is the cluster's record of reality. Lose it without a backup and you have lost the claims themselves.

The controller manager is the room full of reconcilers. kube-controller-manager runs the built-in controllers, and each controller runs the same loop for its own resource. The Deployment controller watches Deployments and manages ReplicaSets. The ReplicaSet controller watches ReplicaSets and manages Pods. The Node controller watches whether nodes are still reporting in. Each one does exactly three things, over and over: observe the current state through the API server, compare it to the desired state, and act to close the difference. Observe, compare, act. That triad is the loop, and the controller manager is a room where dozens of copies of it turn at once.

The scheduler is the placement reconciler. kube-scheduler has one narrow job inside the loop. It watches for Pods that have been created but not yet assigned to a node, and for each one it decides which node should run it. It weighs resource requests against available capacity, honors affinity and taint rules, picks a node, and writes that binding back through the API server. The scheduler never starts a container. It only answers the question where, and then hands the Pod on.

The kubelet is the hands on each node. Every worker node runs a kubelet, and the kubelet is where claims finally become processes. It watches the API server for Pods bound to its own node, and for each one it drives the container runtime to pull images and start containers, then reports back what is actually running. The kubelet closes the loop at the far end: the control plane decided what should exist, and the kubelet is the part that makes it exist and tells the truth about whether it did.

Read those five parts as one sentence and you have the cluster. A claim enters through the API server, rests in etcd, is turned into Pods by a controller, placed on a node by the scheduler, and run by a kubelet, while every one of those components keeps watching for the gap to reopen.

§ IVWorked Example — Killing a Pod and Watching the Room Respond

Take a Deployment claiming three replicas. All three Pods are running, spread across two nodes, and the cluster is quiet. Quiet, for Kubernetes, does not mean idle. It means every loop has looked, found no gap, and is about to look again.

Now delete one Pod by hand with kubectl delete pod. Watch what happens, named component by named component, because this single event runs the entire architecture in about a second.

# the claim
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3          # <- the standing condition: three, always

# the event
$ kubectl delete pod web-7d9f-abc12
# reality is now 2; the claim still says 3; a gap exists

The API server receives the delete, removes the Pod object from etcd, and notifies its watchers that the set of Pods changed. The ReplicaSet controller is one of those watchers. It observes that its ReplicaSet claims three Pods and only two now exist, compares, finds a gap of one, and acts by creating a new Pod object through the API server. That new Pod is created with no node assigned. The scheduler is watching for exactly that. It sees an unscheduled Pod, weighs the two nodes against the Pod's resource requests, picks one, and writes the binding back through the API server. The kubelet on the chosen node is watching for Pods bound to it. It sees the new assignment, tells the container runtime to pull the image and start the container, and reports the Pod as running.

No human was involved after the delete. No alert fired, because from the cluster's point of view nothing went wrong. A gap opened between the claim and reality, and the loops that exist to close gaps closed it. This is what people mean when they say Kubernetes is self-healing, and it is worth being precise about the mechanism rather than the marketing. The cluster does not heal because it is smart. It heals because every component is a loop whose resting behavior is to erase the difference between what you claimed and what is true.

§ VThe Loop Is Level-Triggered, Not Edge-Triggered

Here is the property that makes the loop durable, and the one that trips up engineers coming from event-driven systems. Kubernetes reconciliation is level-triggered, not edge-triggered. Coin that distinction and hold it, because the Dev lesson builds directly on it.

An edge-triggered system responds to the event of a change. The Pod was deleted, so fire the deletion handler. If that handler misses the event, because the controller was restarting, or the message was dropped, the response never happens and the system is now permanently wrong. A level-triggered system responds to the current state regardless of what events led to it. It does not ask what just changed. It asks what is true right now, and does it match the claim. If it matches, do nothing. If it does not, act, no matter how the mismatch arose or how many events were missed getting there.

Kubernetes controllers work the second way, and this is why they survive their own failures. Restart the controller manager mid-incident and nothing is lost. The moment it comes back, every controller re-observes the full current state, compares it to the desired state, and closes whatever gaps exist, exactly as if it had been watching the whole time. There is no backlog of missed events to replay, because the loop never depended on events for correctness. Events only tell it when to look sooner. The claim in etcd, compared against reality, is always enough to decide the right action from a cold start.

This is the deepest reason the reconciliation model holds under pressure. A system that needs every event to land is fragile in exactly the moments that matter, when components are crashing and messages are being lost. A system that only needs to read the current state and the claim is at its calmest precisely then, because a crash costs it nothing but a little time.

§ VIConnection to Today's Dev Lesson

The control plane runs this loop for the resources Kubernetes ships with. The moment you have a resource of your own, a database, a certificate, a tenant, that no built-in controller understands, you write the loop yourself. That is what an operator is, and that is where today's Python lesson goes. It defines a custom resource, then writes a controller that watches for that resource and reconciles it: observe the claim, compare it to what exists, act to close the gap, and do it level-triggered so a restart loses nothing. The Ops discipline here is to see the loop the control plane already runs. The Dev discipline is to run one of your own by the same rules, because a controller you write and a controller Kubernetes ships are the same three verbs turning on the same clock.

§ VIIClosing

Kubernetes is one idea repeated until it looks like many. You file a claim about what should be true, and a room of loops spends its life closing the distance between that claim and the world, checking again the moment it wins, correcting drift as its resting state rather than its alarm. The API server holds the front door, etcd holds the memory, the controllers hold the reconcilers, the scheduler holds placement, and the kubelet holds the hands that make it real. Learn to name which loop owns which gap and you can reason about any behavior the cluster shows you, including the ones the cert lesson will ask you to fix under time.

Open kubectl get events on any cluster you run and read it as a log of loops closing gaps. When a workload will not come up, the question is never what broke. It is which loop stopped turning, and where in the chain from API server to kubelet did the claim fail to become a process. Find the stalled loop and you have found the fault.

Paired lessons → Dev Polyglot-Dev/Python/2026-07-23-writing-a-kubernetes-controller-in-python-... · Cert Cert-Prep/CNCF/2026-07-23-cka-cluster-architecture-...