Python for Service Discovery — the Watch API, EndpointSlices, the registry

Every service mesh is watching the API for the same thing. This lesson builds the smallest version of that watch.

LIST to establish state. WATCH from that resourceVersion. Reconnect on 410 with a fresh LIST. That is the whole discipline the Watch API rewards.

§ IFrame

Every service mesh and every smart client-side load balancer is doing the same thing under the covers: watching the Kubernetes API for changes to a Service's EndpointSlice, maintaining a local cache of the current pod IPs, and routing traffic to those IPs directly. The Ops lesson stopped at what the cluster hides behind a Service; this lesson lets a Python program see through the abstraction, subscribe to endpoint changes, and build a live registry the caller can use.

The reason to do this at all is that ClusterIP + kube-proxy is not always what you want. A client-side load balancer sees every backend individually and can implement gRPC-style per-request load balancing that ClusterIP's per-connection random routing cannot. A metrics scraper wants a list of pods to scrape, not one Service to hit. A service-mesh sidecar wants the full endpoint picture to make its own routing calls. All three lift the Service's private mechanism, the endpoint list, back up to a client that owns its own routing. This lesson builds the smallest version of that lift, and names the two disciplines the Kubernetes Watch API demands from every client that talks to it.

§ IILanguage Idiom

The Kubernetes API server offers three read modes: LIST returns everything at once; GET returns one object; WATCH opens a long-lived HTTP stream that pushes ADDED, MODIFIED, and DELETED events as objects change. A production client uses LIST-then-WATCH: LIST first to establish the current state and get a resourceVersion, then WATCH from that resourceVersion so no event is missed between the two calls. Miss that pattern and the client either has stale state (WATCH without LIST) or misses events during the LIST (LIST without WATCH continuation).

The Watch API is a stream, and Python maps it to an iterator. The official kubernetes Python client exposes watch.Watch().stream(...) which returns an iterator of event dicts. Read one event, process it, loop. This is Ramalho's generator pattern from the iterator chapter that grounds this lesson: the stream is not a fixed collection but a long-running producer, and treating it as an iterable is what keeps the client's memory footprint constant even when the cluster produces thousands of endpoint events over a day.

Reconnection is not optional. Watch streams end for many reasons: the API server rotates, the resourceVersion ages out of the etcd compaction window (after 5 minutes by default), a network hop drops the connection. A production client catches the disconnect, LISTs again to re-establish state and get a fresh resourceVersion, and WATCHes again from there. This is the single failure mode most first-cut watch clients ignore; the second mode is not catching the "410 Gone" that the API server returns when the resourceVersion has aged out, which requires a full re-LIST rather than a simple reconnect.

§ IIICode Worked Example

The specimen is a ServiceRegistry class that, given a Service name, maintains a live set of the current pod IPs behind it and exposes an iterator over changes.

from kubernetes import client, config, watch
from threading import RLock

class ServiceRegistry:
    def __init__(self, namespace: str, service: str):
        config.load_kube_config()
        self.api = client.DiscoveryV1Api()
        self.namespace = namespace
        self.service = service
        self._addresses: set[str] = set()
        self._lock = RLock()

    def _label_selector(self) -> str:
        return f"kubernetes.io/service-name={self.service}"

    def snapshot(self) -> set[str]:
        with self._lock:
            return set(self._addresses)

Two invariants live in the constructor. First, the DiscoveryV1Api is the versioned client for EndpointSlices; do not reach for the older CoreV1Api Endpoints endpoint unless you are targeting a cluster that predates 1.21. Second, the lock protects the address set because the watch loop runs in its own thread and callers on the main thread read the snapshot concurrently.

The bootstrap and the watch loop follow the LIST-then-WATCH pattern the API server rewards:

def _list(self) -> str:
    resp = self.api.list_namespaced_endpoint_slice(
        namespace=self.namespace,
        label_selector=self._label_selector(),
    )
    addrs = set()
    for slice_obj in resp.items:
        for ep in (slice_obj.endpoints or []):
            if ep.conditions and ep.conditions.ready:
                addrs.update(ep.addresses or [])
    with self._lock:
        self._addresses = addrs
    return resp.metadata.resource_version

def _watch(self, resource_version: str):
    w = watch.Watch()
    stream = w.stream(
        self.api.list_namespaced_endpoint_slice,
        namespace=self.namespace,
        label_selector=self._label_selector(),
        resource_version=resource_version,
        timeout_seconds=600,
    )
    for event in stream:
        self._apply(event["type"], event["object"])

The list call returns a resource_version on the collection's metadata; that opaque string tells the API server "start the watch from immediately after this snapshot" when passed as resource_version to the stream call. The timeout_seconds=600 is a client-side upper bound on how long any single stream can run before we cycle it, defense against silent stalls the server end sometimes produces.

The apply step is where events become state changes:

def _apply(self, event_type: str, slice_obj) -> None:
    ready = {addr
             for ep in (slice_obj.endpoints or [])
             if ep.conditions and ep.conditions.ready
             for addr in (ep.addresses or [])}
    with self._lock:
        if event_type == "DELETED":
            self._addresses -= ready
        else:
            other_slices_addrs = self._addresses - self._prior_addrs_of(slice_obj)
            self._addresses = other_slices_addrs | ready

DELETED removes the slice's ready addresses; ADDED and MODIFIED replace the slice's contribution with the current ready set, computed from ep.conditions.ready. The one subtle case is that a slice's contents can move (a pod becoming NotReady drops from the slice), so MODIFIED cannot be treated as "add these"; it must be "replace this slice's contribution." A production version tracks the per-slice contribution in a dict and diffs on each event; the shape above is the smallest working version.

The outer loop handles reconnection, catching the 410 Gone that means the resourceVersion aged out and demands a re-LIST:

def run_forever(self) -> None:
    while True:
        try:
            rv = self._list()
            self._watch(rv)
        except client.exceptions.ApiException as err:
            if err.status == 410:
                continue
            raise

continue re-enters the loop and re-LISTs. Any other API exception propagates loud, per the discipline yesterday's lesson named at the fleet altitude and Friday's lesson named at the language altitude: catch what you named, let the rest fly.

§ IVConnection to Today's Ops Lesson

The Ops lesson taught that a Service is a stable name over churning pods, and that the endpoints controller keeps the EndpointSlice in sync as pods come and go. This lesson is that same picture, subscribed to from the outside. Where the Ops lesson used kube-proxy as the machinery that reads the slices and programs the data path, this lesson makes the Python client be the machinery. The two answer the same question, "who is behind this Service right now," from two altitudes: kube-proxy answers with iptables rules, the Python client answers with a set of IPs a smart client-side balancer can call. Every service mesh works by inserting a Python or Go equivalent of ServiceRegistry between the app and the network, and every custom autoscaler that scales on endpoint count is doing the same watch under the covers.

§ VPrior-Lesson Reach

The controller lesson from six days ago walked the same Watch pattern for custom resources; the kopf framework hides most of it, but the level-triggered mechanic is identical. The admission webhook from three days ago is the other client role, one that the API server calls into rather than one that watches the API server. Together the three lessons, controller, admission, service registry, trace the full picture of Python's Kubernetes surface: react to state (controller), gate a change (admission), read the endpoint set (registry). Yesterday's structured-concurrency lesson supplies the shape the run loop grows into when the registry serves not one Service but many; each Service becomes a task in an asyncio.TaskGroup, and the whole set can be cancelled cleanly with a single scope exit.

§ VIClosing

The Kubernetes Watch API is a stream. LIST to establish state, WATCH from that resourceVersion, and reconnect with a fresh LIST on 410 Gone. Read events as iterator items rather than as batch responses; keep memory constant; hold a lock over any mutable state the watch thread writes and the caller reads. The failure mode is never that the cluster was flaky. It is a watch client without reconnection logic, a client that used LIST-only and got stale, or one that treated MODIFIED as "add" instead of "replace this slice's contribution" and grew a set of ghost IPs.

Pick a Service in a cluster you can watch. Write a fifty-line script that lists its EndpointSlice, opens a watch stream from that resource-version, and prints every ADDED/MODIFIED/DELETED event. Delete one of its pods and watch the events arrive. When you can reason about the sequence, the shape a service mesh assumes has become concrete.