# RAG systems that retrieve the right thing — and prove it

> Design, build, and evaluate retrieval-augmented generation systems: chunking and indexing strategy, hybrid and reranked retrieval, grounded answer generation, and retrieval evaluation harnesses.

Source: https://parsemystatement.com/services/rag-development
Updated: 2026-09-17

Nearly every failing RAG system fails at retrieval, not generation — and almost none of them are measuring retrieval separately, so nobody knows. I build RAG systems where ingestion, chunking, retrieval, and generation are each independently evaluated, so when quality drops you know exactly which stage caused it.

**At a glance**

- Retrieval measured on its own: recall@k, MRR, nDCG — before touching prompts
- Document-aware ingestion, because most RAG corpora start as PDFs
- Hybrid search with reranking, not naive top-k cosine similarity
- Grounded answers with citations and refusal behaviour when evidence is missing

## Why RAG pilots stall

A RAG demo takes an afternoon: embed some documents, do a similarity search, stuff the results into a prompt. It works impressively on the ten questions you thought of while building it, and then it goes to real users and the answers get vague, confidently wrong, or cite the wrong page.

The debugging then goes in the wrong direction. Teams tune prompts, swap models, raise the temperature, add more context — all downstream of the actual problem, which is almost always that the correct passage never made it into the context window at all. No prompt can fix a retrieval miss.

The discipline that fixes this is unglamorous: evaluate retrieval independently of generation. Build a question set with known correct source passages. Measure recall@k. Get that number high first. Only then is it worth tuning anything about the prompt.

## What I build

### Document-aware ingestion

Most knowledge bases arrive as PDFs, scans, and exported wiki pages. Getting clean, structure-preserving text — tables intact, headers attached to their sections — is where retrieval quality is won or lost. This is the same extraction work as my OCR practice, pointed at a different outcome.

### Chunking strategy

Chunk size, overlap, and boundary rules tuned against your evaluation set rather than copied from a tutorial. Structural chunking that respects section and table boundaries, with parent-document retrieval where small chunks retrieve but large context answers.

### Hybrid retrieval

Dense vectors for semantic similarity plus BM25 for exact terms — identifiers, product codes, names — that embeddings reliably lose. Fused and then reranked with a cross-encoder, which is usually the single largest quality jump available.

### Metadata and filtering

Per-chunk metadata for date, source, department, and permission scope, so retrieval can be filtered before ranking. This is also how multi-tenant and access-controlled RAG stays correct.

### Grounded generation

Answers with inline citations to source passages, and explicit refusal when retrieved evidence does not support an answer. A system that says 'not found in the documents' is worth far more than one that guesses plausibly.

### Evaluation harness

Separate scoring for retrieval (recall@k, MRR, nDCG) and answers (groundedness, correctness, citation accuracy), runnable in CI so a model or index change cannot silently regress quality.

## How the build runs

### 1. Question set and ground truth

Real questions from real users, each annotated with the passages that actually answer them. Fifty to a hundred is usually enough to drive every decision that follows. This comes before any index is built.

### 2. Ingestion and indexing

Document extraction, chunking, embedding, and index construction — with the chunking strategy chosen by scoring two or three options against the question set rather than by preference.

### 3. Retrieval tuning

Hybrid weighting, reranking, and filter design, iterated until recall@k plateaus. This is where most of the quality comes from and where most of the engagement time goes.

### 4. Generation and guardrails

Answer synthesis with citations, refusal behaviour, and prompt-injection handling for untrusted document content — which matters a great deal once your corpus includes anything user-submitted.

### 5. Production hardening

Incremental re-indexing as documents change, cost and latency budgets, caching, observability on retrieval quality in production, and a feedback path that turns user corrections into new evaluation cases.

## Common failure modes and what actually fixes them

| Symptom | Usual root cause | Fix |
| --- | --- | --- |
| Answers are vague and generic | Correct passage never retrieved | Measure recall@k; fix chunking and add reranking |
| Exact IDs or codes are never found | Pure vector search loses rare tokens | Add BM25 and fuse the result sets |
| Confident but wrong answers | No grounding constraint or refusal path | Citation-required generation with explicit refusal |
| Good on old docs, wrong on new | Stale index, no incremental re-ingestion | Change-detection pipeline with scheduled re-indexing |
| Quality dropped after a model change | No regression testing on retrieval | Evaluation harness in CI with a quality gate |
| Users see documents they should not | Permissions applied after retrieval | Permission-scoped metadata filtering before ranking |

## Engagement options

### RAG audit — from $1,400 (1–2 weeks)

For teams with a RAG system that underperforms and no clear picture of why.

- Evaluation set built from your real questions
- Retrieval scored independently of generation
- Root-cause analysis with ranked fixes by impact
- Cost and latency profile
- Written remediation plan

### RAG build — from $3,500 (4–10 weeks)

A production retrieval system over your corpus, with evaluation built in from the start.

- Document ingestion including PDFs and scans
- Chunking, hybrid retrieval, and reranking tuned to your data
- Grounded generation with citations and refusal
- Evaluation harness wired into CI
- Deployment, observability, and handover

### Ongoing RAG engineering — from $2,200 / month (Rolling, 3-month minimum)

Continuous quality work as your corpus, questions, and models change.

- New sources onboarded
- Evaluation set grown from production feedback
- Model and embedding migrations handled
- Monthly quality and cost reporting

## Frequently asked questions

### Which vector database should we use?

For most corpora it matters far less than teams expect. Postgres with pgvector handles a surprising amount of production RAG perfectly well and removes an entire piece of infrastructure. Dedicated vector databases earn their place at large scale or with demanding filtering requirements. I will recommend based on your corpus size, filtering needs, and what your team already operates — not on what is currently fashionable.

### Can this run entirely on self-hosted models?

Yes — embeddings, reranker, and generation model can all be open-weight and self-hosted. That pairs naturally with the self-hosted vision-language model service if your documents also cannot leave your network.

### Do we need fine-tuning?

Usually not, and it is rarely the first thing to try. Retrieval quality and prompt structure account for most of the achievable gain; fine-tuning is worth considering for domain-specific output formatting or a specialised reranker once retrieval is already strong. I will tell you if your case is one of the exceptions.

### How do you handle documents with tables and figures?

With the same structure-preserving extraction I use for document parsing — tables kept as structured markup rather than flattened into prose, and figures captioned via a vision model so they become retrievable. Corpora heavy in tables are exactly where naive ingestion pipelines lose the most.

### What about prompt injection from documents in the corpus?

Treated as a real threat, not a footnote. Retrieved content is delimited and handled as untrusted data, tool-calling surfaces are constrained, and instruction-like content in documents is neutralised. This matters most when any part of the corpus is user-submitted.

## Start by measuring retrieval

If your RAG system is underperforming, the audit will tell you within two weeks whether the problem is ingestion, chunking, retrieval, or generation — with numbers rather than intuition.

Contact: adarsh@parsemystatement.com · https://parsemystatement.com/contact

## Related

- https://parsemystatement.com/services/document-data-extraction
- https://parsemystatement.com/services/self-hosted-vision-language-models
- https://parsemystatement.com/services/document-ai-consulting
- https://parsemystatement.com/services/custom-ocr-development