Vertex AI Pipelines — KFP Components, the DAG Contract, Caching, and the Retry Policy

Read a pipeline definition and predict which steps re-run, which retry, and which failure kills the run. Most of the automation questions are secretly this one.

What you wrote by hand in asyncio yesterday, you declare to Vertex today. The exam rewards knowing it is the same shape.

§ IFrame

The Professional Machine Learning Engineer exam has a center of gravity, and it is not model architecture. It is the question of what happens to a model after the notebook: who retrains it, on what trigger, with what data, verified how, deployed by whom. Google's answer across the blueprint's automation objectives is Vertex AI Pipelines, and the exam returns to it from a dozen angles. A candidate who can read a pipeline definition and predict which steps re-run, which retry, and which failure kills the run holds the piece most of the automation questions are secretly about.

This is the first PMLE-emphasis lesson of the Python track. Friday's cert lesson drilled PCAP exception mechanics, what one Python program does when a step fails. Today scales that same question past the process boundary: a training workflow whose steps are containers, whose orchestrator is a managed service, and whose failure discipline is declared in Python but enforced by Google. The trio theme runs straight through: many steps, bounded, deadlined, graded, here as a managed DAG.

§ IIDomain Foundations — What a Pipeline Actually Is

A Vertex AI Pipeline is a directed acyclic graph of containerized steps, written in Python with the Kubeflow Pipelines (KFP) SDK, compiled to a spec file, and executed serverlessly by Vertex. Each piece of that sentence carries exam weight.

Components are the steps. A component is a self-contained unit of work: a container image, a command, typed inputs and outputs. The KFP DSL's @dsl.component decorator builds one from a plain Python function, packaging the function body into an image at compile time. Inputs and outputs come in two kinds, and the distinction recurs across questions: parameters are small values passed by value (strings, numbers, booleans), while artifacts are files and datasets passed by reference through Cloud Storage, with types like Dataset, Model, and Metrics that carry metadata.

The DAG is inferred, never drawn. You do not declare edges. When one step consumes another step's output, that data dependency is the edge, and steps with no dependency between them run in parallel automatically. The graph is the data flow made visible. This is the single most-tested structural fact: given a pipeline, the exam asks what can run concurrently, and the answer is always read off the outputs each step consumes.

Execution is managed. The compiled spec goes to Vertex AI Pipelines, which schedules each step as its own container run, records every execution in Vertex ML Metadata, and charges per run rather than per idle cluster. Gift and Deza's GCP chapter frames the platform choice this way: the operational posture of GCP's ML stack is to hand Google the undifferentiated operations so the team's effort lands on the model. When an exam scenario says minimize operational overhead, managed Vertex Pipelines beats self-hosted Kubeflow on GKE, and the justification is exactly that sentence.

§ IIIPMLE Flavor — Caching and the Retry Policy

Two per-step controls turn a correct pipeline into an efficient and resilient one, and both are decided in the Python definition.

Execution caching. Vertex caches each step keyed on the component's spec and its exact inputs. Re-run the pipeline and any step whose code and inputs are unchanged is skipped, its cached outputs handed to its consumers. This is what makes iterating on the tail of a pipeline cheap: change only the evaluation logic and the extract and train steps resolve from cache in seconds. The trap the exam sets is the step whose inputs look unchanged but whose world has moved, an extract step reading a BigQuery table that receives fresh rows nightly. Its parameters are identical every night, so with caching left on, the pipeline happily retrains on stale data forever. The rule: caching is for deterministic steps; any step that reads external mutable state disables it with set_caching_options(False).

The retry policy. A step can declare set_retry(num_retries=3, backoff_duration="60s", backoff_factor=2.0), and the platform re-runs it on failure with exponential backoff. The discipline for when comes straight from Friday's PCAP lesson, lifted one level: retry is for transient faults, the preempted worker, the momentary service error, the network reset. A schema mismatch or a bug in the component body fails identically on every attempt, and three retries only make the run three times slower to report it. The platform's retry policy is an except clause the infrastructure runs for you; like any except, it must be scoped to failures that repetition can actually cure.

Failure semantics complete the picture. A step that exhausts its retries fails, its downstream steps never start, and the run is marked failed. Cleanup and notification that must happen regardless ride in a dsl.ExitHandler, the pipeline's finally.

§ IVWorked Scenario — The Nightly Retrain

The canonical PMLE pipeline, and today's specimen: extract training data from BigQuery, validate it, train, evaluate, and deploy only if the new model clears a quality bar.

from kfp import dsl

@dsl.pipeline(name="nightly-retrain")
def nightly(project: str, threshold: float):
    extract = extract_op(project=project)
    extract.set_caching_options(False)

    check = validate_op(dataset=extract.outputs["dataset"])

    train = train_op(dataset=check.outputs["clean"])
    train.set_retry(num_retries=3, backoff_duration="60s", backoff_factor=2.0)

    metrics = evaluate_op(model=train.outputs["model"],
                          dataset=check.outputs["clean"])

    with dsl.If(metrics.outputs["auc"] >= threshold):
        deploy_op(model=train.outputs["model"])

Read it as an examiner would. The DAG falls out of the outputs: extract feeds validate, validate feeds train, train and validate together feed evaluate, and nothing here runs in parallel because each step consumes its predecessor. Caching is off on extract because BigQuery is mutable external state; it stays on everywhere else, so a re-run after an evaluation-code change resolves extract, validate, and train from cache and spends money only where something changed. The retry policy sits on train alone, because training is the step exposed to transient infrastructure weather, preemptions and capacity. Validate carries no retry on purpose: bad data fails deterministically, and the correct response is the PCAP one, fail loud now. The dsl.If gates deployment on a Metrics artifact, so a model that cannot beat the threshold ends the run as a recorded no-deploy rather than a bad rollout. Géron's Vertex chapter picks up the story at the far end of this graph, where the blessed model becomes a prediction endpoint.

One operational habit completes the scenario: the run is scheduled, nightly by cron trigger, and every execution lands in Vertex ML Metadata, so any served model traces back to the exact run, data, and metrics that produced it. Lineage is not a bonus feature; it is the audit trail the rest of the MLOps lifecycle stands on.

§ VConnection to Today's Ops and Dev Lessons

Hold the three altitudes of today's trio in one view. The Ops lesson ran two hundred concurrent probes inside one Python process, choosing the survey posture: every task ends in a ledger row, the batch is graded at the end. The Dev lesson supplied the opposite posture, the TaskGroup unit where the first failure cancels the siblings. The pipeline is both, one level up. Within a run it is the TaskGroup posture, a failed step stops its downstream and fails the unit. Across steps it carries the Ops lesson's disciplines as platform features: the retry policy is backoff-with-a-ceiling declared instead of coded, caching is the memo of work already done, and ML Metadata is the ledger. What you wrote by hand in asyncio yesterday, you declare to Vertex today. The exam rewards knowing it is the same shape.

§ VIPractice Questions

Answer each before reading the resolutions.

§ VIIQuestion Resolutions

§ VIIIClosing

A Vertex pipeline is a failure-routing discipline compiled to a graph: caching for the work already done, retries for the faults repetition cures, loud stops for the ones it cannot, a conditional gate before anything reaches production, and a metadata trail from every endpoint back to the run that made it. Learn to read the graph the way §IV read it and the automation questions become bookkeeping.

Sketch your own nightly-retrain DAG on paper: five steps, mark which get caching, which get retries, and where the quality gate sits. Then defend each mark aloud in one sentence. The marks you cannot defend are the exam questions you would miss.

Paired lessons → Ops 01-Earth-DevOps/Synthesis-Lessons/2026-07-27-concurrent-fleet-automation-... · Dev Polyglot-Dev/Python/2026-07-27-pythons-structured-concurrency-...