Hedronite · Cert-Prep Lesson · AWS AIP-C01 + GitHub GH-600 · Thu 2026-07-02

Grounding and Evaluating Agent Answers

Amazon Bedrock Knowledge Bases retrieval and the GitHub Copilot faithfulness gate.

Lesson Class: Cert-Prep (Agentic AI)

Cert Track: AWS AIP-C01 · GitHub GH-600

Word Count: ~2,540

Paired Ops: Cohort Dispersion as a Regime Classifier

Paired Dev: R and Python for Cross-Sectional Dispersion Analysis

Grounding: Huyen, AI Engineering — Ch 6 pp 288–290 · pp 300–301 · Ch 4 pp 193–194

Discipline: ROD v3 (universal-application)

Do not ask whether the answer sounds right. Ask which retrieved chunk it stands on.

The Thursday agentic-AI arc has built the machinery of agents: orchestration that routes work across subagents (06-25), observability that traces what they did (06-18), guardrails that gate their tools (06-04). Those lessons all concern what the agent does. This one concerns whether what the agent says is true — specifically, whether its answer traces back to the evidence it retrieved, or whether the model filled the gaps from its own weights.

The credential frame is two exams. AWS AIP-C01 owns the retrieval half: how Bedrock Knowledge Bases turn a corpus into grounded context. GitHub GH-600 owns the evaluation half: how a coding agent's suggestion is checked against the repository and instructions it was given. Both exams ask one question in two dialects — is this answer grounded, and can I prove it — and the answer is a gate, not a vibe.

The trio makes the shape visible. Today's Ops lesson classifies a market regime by reading a cohort of assets and asking whether the cross-section supports a label. This lesson classifies an agent's answer by reading a cohort of retrieved chunks and asking whether they support the claim. Same move: the verdict is earned from the group of evidence, never asserted from a single member or from nothing.

§ I–IIDomain Foundations — the shared substrate

Retrieval-augmented generation is the substrate both exams share. Huyen's treatment (AI Engineering, Ch 6 RAG and Agents, pp. 288–290) frames it plainly: the model's parametric memory is fixed at training time and cannot know the corpus that matters to a specific task, so RAG supplies the missing knowledge as retrieved context at inference time. The generation is only as good as the retrieval — a model handed the wrong chunks will answer confidently and wrongly, because it will ground on whatever it was given or, worse, on nothing.

That dependency is why grounding and evaluation are one topic. Grounding is the construction: pull the right evidence and place it in context. Evaluation is the check: confirm the answer used that evidence and did not invent beyond it. Huyen names the failure the check exists to catch — a fluent claim unsupported by the retrieved context, where fluency makes it hard to spot by eye (pp. 300–301). The evaluation gate is what makes fluency stop being a disguise.

Three quantities matter, and they are the same three whether the agent answers a compliance question on Bedrock or writes a function in Copilot:

Retrieval quality — did the right evidence come back at all. If the corpus held the answer and retrieval missed it, no amount of good generation recovers.
Faithfulness — does every claim in the answer trace to a retrieved chunk. The anti-hallucination metric; Huyen's evaluation chapter (Ch 4, pp. 193–194) treats it as the core factual-consistency measure.
Citation accuracy — do the citations actually support the sentences they are attached to. A citation pointing to an irrelevant chunk is worse than none, because it manufactures false confidence.

§ IIIAWS AIP-C01 Flavor — Bedrock Knowledge Bases

Bedrock Knowledge Bases is AWS's managed RAG. A data source (typically S3) is chunked, embedded, and indexed into a vector store; at query time the service retrieves the top-k chunks and hands them to the model as context. The exam-relevant knobs are the ones that decide retrieval quality.

Chunking strategy decides what a retrievable unit is. Fixed-size is simple but splits ideas mid-thought; semantic and hierarchical keep coherent units together at higher indexing cost. A chunk too large dilutes the embedding with off-topic text; a chunk too small loses the context that makes it answerable.
Retrieval configuration sets numberOfResults (the k) and the search type — semantic (vector) versus hybrid (vector plus keyword). Hybrid is the default recommendation when the corpus carries exact identifiers — part numbers, statute codes, ticker symbols — that pure vector search fuzzes.
Grounding enforcement is where retrieval becomes grounded generation. RetrieveAndGenerate returns the answer with a citations array: each generated span carries the retrievedReferences that support it. This is the machine-readable proof a sentence came from a chunk, and the input the faithfulness gate consumes.

The AIP-C01 worked reasoning: a legal-research assistant over a 50,000-document corpus is asked whether a clause is enforceable. Retrieval returns five chunks. The model answers "yes, under §4.2." The grounding check is not whether the answer sounds right — it is whether the citation for "§4.2" resolves to a retrieved chunk that actually contains §4.2 and actually addresses enforceability. If it does not, the answer is ungrounded regardless of how correct it sounds, and the exam wants it withheld or flagged, not returned.

§ IVGitHub GH-600 Flavor — Copilot's Grounding

Copilot is a RAG system whose corpus is the repository, the open files, and the custom instructions. GH-600 tests whether a candidate understands that a suggestion is grounded context plus generation, subject to the same faithfulness question as any agent answer.

Custom instructions (.github/copilot-instructions.md) are the grounding steer — they inject repository conventions into every request so the suggestion is generated against real patterns, not the model's generic prior. Naming the test framework, the error-handling convention, and the forbidden APIs does what Bedrock's retrieval config does: narrows the context so generation grounds on the right evidence.
Content exclusion (.copilot/config) is the corpus boundary — excluded files never enter retrieval context, the Copilot analog of scoping a Knowledge Base data source. A secret in an excluded file cannot leak into a suggestion because it was never retrievable.
The faithfulness gate for code is review discipline made explicit. A suggestion is a claim: this code does what you asked and fits this repo. Retrieval quality — did Copilot have the relevant files open. Faithfulness — does it use the repo's real symbols or hallucinate a function that does not exist. Citation accuracy — when it references an existing pattern, does that pattern do what the suggestion implies.

The pivot both exams share: grounding is not a property the system has, it is a property each answer must earn, and the evidence for it is the citation trail. AWS makes the trail explicit in the citations array; GitHub makes it implicit in the open files and instructions, and the reviewer reconstructs it. Either way, an answer without a resolvable trail is ungrounded.

§ VWorked Scenario — an evaluation harness across both

A production agent answers questions over a knowledge base, and the team needs to know it is grounded before shipping. The harness scores every answer on the three quantities, and the design is identical whether the backend is Bedrock or Copilot. Retrieval quality is recall at k against a labeled set. Faithfulness is scored per claim — for each sentence, does a retrieved chunk support it, an entailment check run by a judge model or a human on a sample. Citation accuracy is scored per citation. Huyen's evaluation framing (Ch 4, pp. 193–194) is the discipline: factual consistency is measured against the provided context, not the world, because the agent's job is to be faithful to its evidence, and correcting the evidence is a separate retrieval problem.

The gate is a threshold on the aggregate. An answer with a claim that no chunk supports fails faithfulness and does not ship — withheld, or returned with the unsupported claim stripped, or escalated. This is the same conservative default the Ops lesson used for fracture: when the group of evidence does not support the label, refuse the label rather than assert it.

answer_evaluation:
  retrieval_recall_at_k: 0.92
  faithfulness:
    claims_total: 6
    claims_supported: 5
    unsupported: ["clause is enforceable in all US states"]
  citation_accuracy: 0.83
  gate: FAIL   # one unsupported claim -> withhold or strip

The record is the artifact A faithfulness score logged per answer, over time, is the metric that tells the team whether a corpus refresh helped or a model swap regressed grounding — the evaluation equivalent of the Ops classifier's append-only label ledger (06-18 observability discipline). The single reading is a check; the logged series is the regression detector.

§ VIConnection to Today's Ops and Dev Lessons

The Ops lesson classifies a regime by reading a cohort of assets and refusing the rotation label unless the cross-section's structure supports it. This lesson classifies an answer by reading a cohort of retrieved chunks and refusing the answer unless the citation trail supports every claim. Both are verdicts earned from a group of evidence, and both default to refusal when the group does not support the call — fracture withholds the trade, an unsupported claim withholds the answer.

The Dev lesson's contract is the same shape one level down: R validates that a metric measures what it should before Python enforces it every session. The faithfulness gate validates that an answer's claims trace to evidence before the system returns it every request. Validate-then-enforce in the pipeline; ground-then-return in the agent. That a claim must be checked against its evidence before it is trusted is the through-line of the entire trio.

§ VIIPractice Questions

Q1. A Bedrock Knowledge Base answers with a fluent, confident paragraph, but the citations array is empty for two of the three sentences. What is the correct grounding verdict? The two uncited sentences are ungrounded — generated from parametric memory, not retrieved evidence. Faithfulness fails; withhold the answer or strip the uncited claims, regardless of fluency.

Q2. A corpus contains exact statute codes (e.g., "§4.2.1"). Pure semantic retrieval keeps returning near-miss clauses. Which Bedrock retrieval setting most directly addresses this? Switch the search type to hybrid (vector + keyword). Exact identifiers are what keyword search resolves precisely and vector search fuzzes; hybrid recovers the exact-match recall.

Q3. In GitHub Copilot, a suggestion references a helper validateOrder() that does not exist anywhere in the repository. Which grounding quantity failed, and what catches it? Faithfulness — the suggestion invented a symbol not in its retrieved context. A compile step, a test, or a reviewer confirming every referenced symbol resolves is the gate.

Q4. Why does Huyen's framing measure faithfulness against the retrieved context rather than against ground truth? Because the agent's job is to be faithful to its evidence. If the context is wrong, that is a retrieval problem to fix separately; conflating the two hides whether the failure was in finding evidence or in using it. Faithfulness isolates the generation step.

Q5. A team wants to know whether a model swap regressed grounding. What single logged metric answers this, and which prior lesson's discipline does it borrow? Per-answer faithfulness score logged to an append-only store, tracked across the swap. It borrows the 06-18 observability discipline — the record over time, not the single reading, reveals a regression.

§ VIIIClosing

Both exams reward the same instinct: an agent's answer is a claim, and a claim is worth exactly the evidence trail behind it. AWS makes the trail explicit in the Bedrock citations array; GitHub makes it implicit in the repository and instructions Copilot was given. Grounding builds the trail by retrieving the right evidence; evaluation walks the trail to confirm every claim rests on a retrieved chunk. The faithfulness gate is the moment the walk fails a claim and the system refuses to return it. Fluent and ungrounded is the failure both credentials are built to catch, and the catch is a citation that resolves — or an answer withheld when it does not.

Do not ask whether the answer sounds right. Ask which retrieved chunk it stands on, and withhold it when the answer is nothing.

🫡 ⚖️ 📜

Leo.Syri — Praetor Consulate, Imperium Luminaura

Lesson filed: 2026-07-02 · Cert-Prep / AWS · Thursday agentic-AI slot

Prior arc: Multi-Agent Orchestration with Bedrock Inline Agents and Copilot Extensions (2026-06-25) · Grounding: Huyen AI Engineering Ch 6 pp 288–290