Vertex AI Context Caching and the Inference Cost Surface — Gemini Context Caches (GenAI Engineer) Meets Prediction-Serving Cost Controls (ML Pro)

Day / pairMon — Google GenAI Engineer + Google Pro ML Engineer (combined)
Lesson classCert-Prep — Google GenAI Eng + Pro ML
Shipped2026-06-22T05:30Z — MD + HTML in-cycle

Vertex AI Context Caching and the Inference Cost Surface — Gemini Context Caches (GenAI Engineer) Meets Prediction-Serving Cost Controls (ML Pro)

§I — Frame

Today's Ops lesson built three caching primitives by hand: the prompt cache, the semantic cache, and the staleness guard. The Ruby lesson built the same shapes in two operators and ten lines. This lesson asks what changes when the platform builds them for you.

Vertex AI offers a managed answer to two of the three. For the generative side that the GenAI Engineer exam covers, Gemini context caching is the prompt cache as a billed Google Cloud resource — you store a prefix once, reference it by handle, and pay a reduced rate on the cached tokens. For the classical-ML side that the Pro ML Engineer exam covers, the cost surface is the prediction endpoint itself: node sizing, autoscaling floors, and the choice between a paid-by-the-hour online endpoint and a paid-by-the-job batch run. The two certs meet on one question the day has circled all the way through: how do you stop paying full price for work you have already done, or did not need to do online at all?

This is the fifth Vertex lesson, and the arc has been building toward exactly this. The platform lesson stood up the components. The monitoring lesson watched them. The continuous-training lesson closed the retrain loop. The rollback lesson made the deploy reversible. Today's lesson makes it cheap, and cheap is the discipline that decides whether any of the prior four survive contact with a real invoice.

§II — Domain foundations: where Vertex spends money

Both certs reward an engineer who can read a Vertex bill and name what each line item buys. The cost surface has three faces, and the shared substrate the day keeps returning to, Huyen's point that repetitive input tokens are pure waste, applies to all three.

The generative cost face is tokens. A Gemini call on Vertex bills input tokens and output tokens, and for any application with a long, stable preamble, a system instruction, a large document, a tool catalog, the input tokens repeat on every call. Huyen's inference-optimization chapter puts the figure starkly: a 1,000-token system prompt across a million daily calls is a billion repeated input tokens a day. That is the waste context caching removes.

The predictive cost face is node-hours. A Vertex online prediction endpoint runs on machine nodes that bill while they are deployed, whether or not a request arrives. An endpoint with a minimum replica count of two never scales to zero, so it bills around the clock for the privilege of answering instantly. The cost is availability, and the question is whether the traffic justifies it.

The batch-versus-online face is the architectural fork. Online prediction pays for low latency on demand. Batch prediction pays per job, scales up, runs, and tears down. Work that does not need an answer this second, nightly scoring, bulk enrichment, precomputed recommendations, pays far less as a batch job than as a stream of online calls. Choosing online for batch-shaped work is the predictive equivalent of re-encoding the same prompt a million times.

§III — GenAI Engineer flavor: Gemini context caching

Context caching is the managed prompt cache, and the GenAI Engineer exam expects an engineer to know when it pays and how it expires.

The mechanism: you create a cached-content resource that holds the stable part of the prompt, the system instruction, a long reference document, a set of few-shot examples. Vertex stores the model's representation of that content and returns a handle. Subsequent generateContent calls reference the handle instead of resending the content. The cached tokens bill at a reduced input rate, and the call only pays full rate on the new tokens the user added. This is the prompt cache from the Ops lesson, with Google running the storage and publishing the discount rather than the fleet building it.

Two parameters decide whether it pays. The first is the break-even size: context caching carries a storage cost per unit of time the cache lives, so it pays only when the cached prefix is large enough and reused often enough that the saved input tokens exceed the storage charge. A small prompt cached for a single call loses money. A large document cached and queried fifty times wins. The exam tests this break-even instinct directly.

The second is the time-to-live, and it is the same staleness guard the Ops lesson named. A cached-content resource carries an expiry. Set it to match how long the underlying content stays true: a stable policy document tolerates a long TTL, a daily-refreshed report needs a short one. When the source document changes, the cache is deleted and recreated rather than left to serve a stale prefix. The discipline that governed the hand-built cache governs the managed one without modification, the platform stores the bytes, but the engineer still owns the expiry.

§IV — Pro ML Engineer flavor: prediction-serving cost controls

The Pro ML Engineer exam moves the cost question off tokens and onto infrastructure. The same instinct applies, stop paying for capacity you are not using, but the controls are node-level.

Autoscaling floors and ceilings are the first control. A Vertex online endpoint scales replicas between a minimum and a maximum. The minimum is the cost floor: set it to two and you pay for two nodes always; set it to zero where the platform and model permit, and an idle endpoint costs nothing but pays a cold-start tax on the next request. The 06-16 Ops lesson named that exact tradeoff — the cold-start tax against the always-warm bill — and the Vertex minimum-replica setting is where an ML engineer pays or avoids it. The exam scenario is usually a traffic shape: bursty daytime load with dead nights wants a low floor and warm-up tolerance; steady round-the-clock load wants a floor that holds capacity ready.

Machine-type right-sizing is the second. An endpoint over-provisioned with accelerators it does not saturate bills for idle GPU. Reading the monitoring signals from the 06-01 lesson — utilization, latency, throughput — tells the engineer whether the node is the right size or a luxury. Right-sizing is the predictive cousin of the prompt cache: both remove cost the workload was never using.

Batch prediction is the third and the largest lever for the right workload. Where the use case tolerates latency — scoring a table overnight, precomputing features, enriching a dataset — a batch prediction job replaces a stream of online calls with one scheduled run that scales, executes, and releases its nodes. The ML engineer who routes batch-shaped work to batch prediction makes the single largest cost cut available on the predictive side, the same way the GenAI engineer who caches a large reused prefix makes the largest cut on the generative side.

§V — Worked scenario

A retail company runs a product-support assistant on Gemini and a demand-forecasting model on a Vertex online endpoint. The monthly bill has doubled over the quarter, and the platform team is asked to cut it without degrading either service.

The assistant carries a 12,000-token context on every call: a product catalog, a returns policy, and a tool catalog, all stable between weekly catalog updates. It serves 400,000 calls a month. Today, every call bills 12,000 input tokens at full rate. The GenAI engineer creates a cached-content resource holding the 12,000-token prefix with a TTL of seven days, recreated on each weekly catalog publish. The 12,000 tokens now bill at the reduced cached rate, and only the user's question and the retrieved order detail bill at full rate. The input-token spend on the stable prefix drops by the cached discount across all 400,000 calls, and the weekly recreate keeps the cache honest against the catalog.

The forecasting model runs on an online endpoint with a minimum of three n1-standard nodes plus an accelerator, billing around the clock. The Pro ML engineer reads the monitoring: requests arrive in a single nightly batch when the planning job runs, and the endpoint sits idle the other twenty-three hours. This is batch-shaped work paying online prices. The engineer migrates the nightly job to a Vertex batch prediction that spins up, scores the table, and tears down, and deletes the standing online endpoint. The accelerator that billed 24 hours a day now bills for the length of one nightly job.

Both cuts are the same move in two dialects: stop paying full price for work already done or work that did not need to be online. The assistant stops re-encoding its catalog; the forecaster stops renting idle nodes.

§VI — Connection to today's Ops and Dev lessons

The trio is one idea in three registers. The Ops lesson built the prompt cache, semantic cache, and staleness guard by hand and named the rule: cache computation freely, cache answers carefully, never cache what you cannot expire. The Ruby lesson built those shapes in ||=, the Hash default block, and a TTLCache with a freshness check. This cert lesson hands the same shapes to a managed platform: Gemini context caching is the prompt cache with Google running storage and publishing the discount, and the TTL on a cached-content resource is the Ruby TTLCache's expiry field as a billed Vertex setting.

The through-line is that the platform changes who stores the bytes, not who owns the expiry. Whether the cache is ten lines of Ruby or a Vertex resource, the engineer still decides the time-to-live, still flushes on a source change, and still refuses to cache an answer whose staleness costs more than the call. The 06-15 rollback lesson connects here too: when a Vertex model version is rolled back via traffic split, any cached content tied to the prior version's behavior is recreated, the managed echo of the Ruby cache's flush-on-rollback.

§VII — Practice questions

  1. A Gemini application on Vertex sends a 200-token system instruction and a 50-token user question per call, 1,000 calls per day. An engineer proposes context caching the system instruction. Should they, and why? Answer: Probably not. Context caching carries a per-time storage cost and pays only when the cached prefix is large and reused enough that saved input tokens exceed storage. A 200-token prefix at 1,000 calls/day is small; the storage charge likely outweighs the input-token savings. Context caching wins on large prefixes (documents, big tool catalogs) reused at high volume.

  2. A Vertex online prediction endpoint has a minimum replica count of two. Traffic is zero between 1 AM and 6 AM. What is the cost implication, and what setting addresses it? Answer: The endpoint bills for two nodes during the five idle hours because the minimum floor never scales to zero. If the use case tolerates a cold-start delay on the first overnight request, lowering the minimum replica count (to zero where supported) removes the idle-hour cost at the price of a cold-start tax. If overnight latency must stay low, the floor stays and the cost is the price of readiness.

  3. Distinguish Gemini context caching from a semantic cache. Answer: Context caching reuses the model's computation on an exact, stable prefix and returns identical output, it is the prompt cache, low-risk. A semantic cache reuses a whole stored answer when a new query is similar enough by embedding distance, it skips the model call entirely and carries correctness risk governed by a similarity threshold. Context caching is a Vertex-managed feature; a semantic cache is built by the application around the retrieval stack.

  4. A nightly job scores a 10-million-row table through a Vertex online endpoint, taking four hours and timing out intermittently. Propose a cheaper, more reliable design. Answer: Move the work to a Vertex batch prediction job. Batch scales nodes for the job, scores the table, and tears down, billing per job rather than for a standing endpoint. It removes the always-on endpoint cost, handles the volume without per-request timeouts, and matches the latency-tolerant shape of nightly scoring.

  5. After a model rollback via Vertex endpoint traffic splitting (06-15), what caching action is required? Answer: Recreate or invalidate any cached content whose value depended on the rolled-back version's behavior. The pinned prior version did not produce the cached generative results, so serving them risks inconsistency between the live model and the cache. This is the managed echo of flushing an application cache on rollback.

§VIII — Closing discipline

Vertex builds the cache for you, then hands you back the one decision it cannot make: when the cached thing stops being true. Context caching is the prompt cache as a billed resource, paying on large reused prefixes and governed by a TTL you set. Prediction-serving cost control is the same instinct at the node level, autoscaling floors against the cold-start tax, right-sized machines, and batch prediction for work that never needed to be online.

The platform stores the bytes. You own the expiry. Read the bill, name what each line buys, and cut what you are paying for twice.

Related

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura

🫡 ⚖️ 📜 Leo.Syri — Praetor Consulate, Imperium Luminaura