EVALUATION CAPABILITY 01

RAG Evaluation

Measure whether your Retrieval-Augmented Generation system actually retrieves the necessary knowledge, uses it faithfully, and answers user queries without hallucinated interpolations.

Scope a RAG Evaluation View Measured Metrics ↓

SCOPE & IMPACT

System Boundaries

What We Evaluate

We test the complete RAG lifecycle: chunking strategy validity, embedding model relevance, vector search precision, reranker performance, context window packing, and generator synthesis faithfulness under domain documents.

Why Generic Metrics Fail in Production

High-level synthetic benchmarks rarely surface the catastrophic edge cases that cause production RAG failure: conflicting effective dates in policy documents, subtle retrieval cutoff on multi-hop questions, and generator tendency to hallucinate plausible facts when retrieved evidence is thin.

TECHNICAL TELEMETRY

Measured Metrics

Objective measurement across the retrieval and generation phases.

Metric	Target Phase	Description
Context Precision	RETRIEVAL	Proportion of top-k retrieved chunks that contain relevant factual evidence.
Context Recall	RETRIEVAL	Proportion of required ground-truth evidence successfully retrieved into the context window.
Citation Faithfulness	GENERATION	Mathematical groundedness: claims in the generated response directly supported by retrieved context.
Answer Relevance	GENERATION	Directness and completeness with which the response answers the user's specific prompt.
Retrieval Latency (p95)	PIPELINE	Time-to-first-chunk and full context assembly timing across variable query complexities.

FAILURE TAXONOMY

Known Failure Modes

How we categorize errors to guide prompt and retrieval remediation.

Retrieval Cutoff

Crucial amendment or clause is ranked just below the top-k threshold, depriving the generator of evidence.

Context Poisoning

High-scoring distractor chunks with matching semantic keywords mislead the generator into incorrect conclusions.

Citation Hallucination

The model generates a factual claim and cites a retrieved chunk that actually contains no supporting evidence.

ILLUSTRATIVE EXAMPLE

Sample Evaluation Run

RUN-2026-RAG-DEMO

Test Partition	Tasks	Context Precision	Faithfulness	Observed Finding
Single-Hop Direct Lookups	1,200	95.4%	98.1%	Nominal retrieval performance
Multi-Document Syntheses	800	82.1%	86.4%	Chunk boundary cuts cross-references
Temporal Conflict Questions	200	74.2%	81.5%	Omitted appendix amendment clauses

Notice: This is an illustrative technical example using sample or internally prepared evaluation data. It does not represent a client result.

Audit Your RAG Architecture

Connect with our engineering team to construct a task-grounded RAG evaluation suite under mutual NDA.

Start an Evaluation

Direct inquiry: ai@acadifysolution.com