Vertex AI Agent Builder and the RAG Grounding Discipline — Vector Search Datastores, Citation Injection, and the Hallucination-Reduction Surface

§I — Frame

The 06-22 lesson addressed the cost surface: how to reduce what you pay per Vertex AI call when context is repeated. This lesson addresses the quality surface: how to reduce wrong answers by anchoring the model's responses in retrieved facts, with citations the user can verify.

A language model generates the statistically most likely continuation of its prompt. When the prompt carries no grounding facts, the model draws from its training distribution — which may be months or years stale, incomplete for your domain, or simply wrong in ways that sound convincing. Grounding replaces statistical confidence with factual reference. The model is told: here are documents from a trusted source; generate only what these documents support; mark every claim with the source it came from.

Vertex AI Agent Builder is Google's managed surface for RAG on GCP. It provides the datastore, the retrieval pipeline, and the grounding integration in one managed offering. Both the GenAI Engineer cert and the ML Pro cert examine this surface: GenAI Engineer on the API surface and RAG configuration, ML Pro on pipeline integration, evaluation, and datastore lifecycle management.

Huyen's Chapter 6 frames RAG as two independent problems: retrieval quality (are the right documents selected?) and generation quality (does the model use them faithfully?). The two problems require separate measurement and separate optimization (AI Engineering, Ch. 6, pp. 285–290). Agent Builder provides primitives for both.

§II — Domain Foundations: The RAG Pattern on GCP

Retrieval-Augmented Generation has three steps. First, the user query is embedded into a vector. Second, the vector is compared against a pre-indexed datastore of document embeddings and the top-K most similar chunks are retrieved. Third, those chunks are injected into the model prompt alongside the original query, and the model generates a response grounded in the retrieved text.

On GCP, Agent Builder abstracts the second step. The developer creates a datastore, populates it with documents (from Cloud Storage, BigQuery, or a web crawl), and Agent Builder handles the chunking, embedding, and indexing. The underlying index is Vertex AI Vector Search — Google's managed approximate nearest-neighbor service. The embedding model defaults to text-embedding-004; custom embedding models can be substituted for domain-specific vocabularies.

A grounding call in Agent Builder exposes the RAG pattern through the Gemini API's grounding parameter rather than as a separate retrieval step. The developer passes the datastore ID in the grounding configuration; the API handles retrieval and injection transparently. The response object returns both the generated text and a list of groundingChunks — the specific document segments the model cited.

§III — GenAI Engineer Flavor: Configuring Grounding

Four configuration decisions govern grounding quality.

Datastore type. Agent Builder supports structured datastores (BigQuery tables with schema-keyed retrieval) and unstructured datastores (documents chunked and embedded). Unstructured datastores are the starting point for most GenAI applications; structured datastores add SQL-style filtering before the vector search, useful for date-range or category constraints.

Chunk size. Large chunks (512–1024 tokens) preserve surrounding context but may retrieve irrelevant content. Small chunks (128–256 tokens) retrieve precise passages but may miss broader context. The chunking strategy is set at datastore creation time; changing it requires rebuilding the index.

Dynamic retrieval threshold. Agent Builder exposes a dynamicRetrievalConfig.dynamicThreshold between 0.0 and 1.0. Below the threshold, the model answers from training data; above it, grounding fires. A threshold of 0.0 grounds every call; 1.0 grounds nothing. Setting it between 0.5 and 0.7 grounds domain-specific questions while letting the model handle factual questions it can answer reliably from training.

Citation injection. When grounding fires, the response includes a groundingMetadata field with groundingChunks and groundingSupport — a mapping from claim segments in the response to the chunks that support them. The application renders these citations to the user. A grounded response without rendered citations gives users no way to verify the claims; rendering citations is the completion of the grounding contract.

§IV — ML Pro Flavor: Pipeline Integration and Evaluation

Datastore refresh pipelines. A datastore is a snapshot of its source documents at indexing time. When sources change — new policies published, old docs superseded — the datastore must be refreshed or it serves stale grounding. Agent Builder supports incremental and full re-indexing via the Data Store API. The ML Pro engineer wires a Vertex AI Pipeline trigger: when a Cloud Storage event fires (new document uploaded), a pipeline job runs the indexing step and updates the datastore without taking the grounding endpoint offline. Incremental re-indexing is preferred for large datastores; full re-indexing is reserved for schema changes or embedding model swaps.

Grounded response evaluation. A grounded agent should be evaluated on three metrics separately: retrieval recall (did the top-K chunks contain the information needed?), grounding faithfulness (does the response contain only claims supported by the retrieved chunks?), and citation accuracy (do the groundingSupport pointers map correctly to the segments that support each claim?). Vertex AI Evaluation Service provides answer_correctness and groundedness metrics against offline eval sets using the same API surface as online serving.

Latency budget and the context window. Retrieval adds latency and tokens. Five retrieved chunks of 400 tokens each consume 2,000 tokens of budget before the query arrives. The connection to today's Ops lesson is exact: retrieved chunks are high-priority content in the context budget — grounding evidence ranks above old conversation turns in the eviction order. The budget disciplines from the Ops lesson govern how much room remains for retrieval.

§V — Worked Example

A legal research assistant built on Agent Builder grounds responses in 50,000 regulatory documents stored in Cloud Storage. Chunk size: 512 tokens with 64-token overlap. The team observes two problems after launch.

First: 12% of responses include claims not found in any retrieved document. The team runs a faithfulness evaluation against a 500-query eval set and finds dynamicThreshold is set to 0.3 — too low. The model answers from training data for many queries rather than from the datastore. Raising the threshold to 0.65 and enabling citation rendering drops the confabulation rate to 2.1%.

Second: latency for complex queries exceeds the target because top-K retrieval returns 10 chunks, filling the token budget before retrieval completes. The team reduces top-K to 5 and adds a reranker pass (Vertex AI Rank API) that scores the 5 chunks for relevance before injection. The reranker adds 80ms but drops the average chunk count per response from 5 to 3.2, cutting 720 tokens from the average context. Total latency drops because generation is faster over a shorter context.

§VI — Connection to Today's Ops and Dev Lessons

The Ops lesson established that retrieved documents occupy a specific slot in the context budget hierarchy: higher priority than old conversation turns, lower than the system prompt. A grounded Agent Builder response carries the retrieved chunks in that slot. The priority-truncation eviction order applies directly: if the context budget is tight, oldest conversation turns truncate before grounding chunks, because the grounding evidence is what makes the response trustworthy.

The Go lesson's Consume function fires before context assembly. When the grounding call retrieves 5 chunks, the middleware consumes their token count from the budget before injection. If the budget cannot accommodate the chunks, the gate fires — in this case, the gate function might reduce top-K from 5 to 3 rather than failing the call. Budget enforcement and grounding interact at the context-assembly layer.

§VII — Practice Questions

§VIII — Closing

Cost and quality are the two sides of every Vertex AI serving decision. The 06-22 lesson addressed cost: how to pay less for what the model already knows. This lesson addresses quality: how to make the model answer from what you know, not from what it remembers.

The datastore is the source of truth. The retrieval pipeline is the mechanism that surfaces it. The citation is the proof that the mechanism worked. Without the citation rendered, the grounding discipline is incomplete: the user cannot verify, and the engineer cannot measure.

Build the eval before the threshold. Set the threshold against the eval. Render every citation.

Examine well.

Related

• Prior arc: Vertex AI Context Caching (2026-06-22) — cost surface; grounding is the quality surface
• Domain hub: Certifications (Cross-References)
• Grounding tome: AI Engineering — Chip Huyen (Ch. 6 RAG and Agents, pp. 285–296)