RAG systems
RAG systems that retrieve the right thing — and prove it
Nearly every failing RAG system fails at retrieval, not generation — and almost none of them are measuring retrieval separately, so nobody knows. I build RAG systems where ingestion, chunking, retrieval, and generation are each independently evaluated, so when quality drops you know exactly which stage caused it.
- Retrieval measured on its own: recall@k, MRR, nDCG — before touching prompts
- Document-aware ingestion, because most RAG corpora start as PDFs
- Hybrid search with reranking, not naive top-k cosine similarity
- Grounded answers with citations and refusal behaviour when evidence is missing
Why RAG pilots stall
A RAG demo takes an afternoon: embed some documents, do a similarity search, stuff the results into a prompt. It works impressively on the ten questions you thought of while building it, and then it goes to real users and the answers get vague, confidently wrong, or cite the wrong page.
The debugging then goes in the wrong direction. Teams tune prompts, swap models, raise the temperature, add more context — all downstream of the actual problem, which is almost always that the correct passage never made it into the context window at all. No prompt can fix a retrieval miss.
The discipline that fixes this is unglamorous: evaluate retrieval independently of generation. Build a question set with known correct source passages. Measure recall@k. Get that number high first. Only then is it worth tuning anything about the prompt.
What I build
Document-aware ingestion
Most knowledge bases arrive as PDFs, scans, and exported wiki pages. Getting clean, structure-preserving text — tables intact, headers attached to their sections — is where retrieval quality is won or lost. This is the same extraction work as my OCR practice, pointed at a different outcome.
Chunking strategy
Chunk size, overlap, and boundary rules tuned against your evaluation set rather than copied from a tutorial. Structural chunking that respects section and table boundaries, with parent-document retrieval where small chunks retrieve but large context answers.
Hybrid retrieval
Dense vectors for semantic similarity plus BM25 for exact terms — identifiers, product codes, names — that embeddings reliably lose. Fused and then reranked with a cross-encoder, which is usually the single largest quality jump available.
Metadata and filtering
Per-chunk metadata for date, source, department, and permission scope, so retrieval can be filtered before ranking. This is also how multi-tenant and access-controlled RAG stays correct.
Grounded generation
Answers with inline citations to source passages, and explicit refusal when retrieved evidence does not support an answer. A system that says 'not found in the documents' is worth far more than one that guesses plausibly.
Evaluation harness
Separate scoring for retrieval (recall@k, MRR, nDCG) and answers (groundedness, correctness, citation accuracy), runnable in CI so a model or index change cannot silently regress quality.
How the build runs
1. Question set and ground truth
Real questions from real users, each annotated with the passages that actually answer them. Fifty to a hundred is usually enough to drive every decision that follows. This comes before any index is built.
2. Ingestion and indexing
Document extraction, chunking, embedding, and index construction — with the chunking strategy chosen by scoring two or three options against the question set rather than by preference.
3. Retrieval tuning
Hybrid weighting, reranking, and filter design, iterated until recall@k plateaus. This is where most of the quality comes from and where most of the engagement time goes.
4. Generation and guardrails
Answer synthesis with citations, refusal behaviour, and prompt-injection handling for untrusted document content — which matters a great deal once your corpus includes anything user-submitted.
5. Production hardening
Incremental re-indexing as documents change, cost and latency budgets, caching, observability on retrieval quality in production, and a feedback path that turns user corrections into new evaluation cases.
Common failure modes and what actually fixes them
| Symptom | Usual root cause | Fix |
|---|---|---|
| Answers are vague and generic | Correct passage never retrieved | Measure recall@k; fix chunking and add reranking |
| Exact IDs or codes are never found | Pure vector search loses rare tokens | Add BM25 and fuse the result sets |
| Confident but wrong answers | No grounding constraint or refusal path | Citation-required generation with explicit refusal |
| Good on old docs, wrong on new | Stale index, no incremental re-ingestion | Change-detection pipeline with scheduled re-indexing |
| Quality dropped after a model change | No regression testing on retrieval | Evaluation harness in CI with a quality gate |
| Users see documents they should not | Permissions applied after retrieval | Permission-scoped metadata filtering before ranking |
Engagement options
RAG audit
from $1,400
1–2 weeks
For teams with a RAG system that underperforms and no clear picture of why.
- Evaluation set built from your real questions
- Retrieval scored independently of generation
- Root-cause analysis with ranked fixes by impact
- Cost and latency profile
- Written remediation plan
RAG build
from $3,500
4–10 weeks
A production retrieval system over your corpus, with evaluation built in from the start.
- Document ingestion including PDFs and scans
- Chunking, hybrid retrieval, and reranking tuned to your data
- Grounded generation with citations and refusal
- Evaluation harness wired into CI
- Deployment, observability, and handover
Ongoing RAG engineering
from $2,200 / month
Rolling, 3-month minimum
Continuous quality work as your corpus, questions, and models change.
- New sources onboarded
- Evaluation set grown from production feedback
- Model and embedding migrations handled
- Monthly quality and cost reporting
Frequently asked questions
Which vector database should we use?
For most corpora it matters far less than teams expect. Postgres with pgvector handles a surprising amount of production RAG perfectly well and removes an entire piece of infrastructure. Dedicated vector databases earn their place at large scale or with demanding filtering requirements. I will recommend based on your corpus size, filtering needs, and what your team already operates — not on what is currently fashionable.
Can this run entirely on self-hosted models?
Yes — embeddings, reranker, and generation model can all be open-weight and self-hosted. That pairs naturally with the self-hosted vision-language model service if your documents also cannot leave your network.
Do we need fine-tuning?
Usually not, and it is rarely the first thing to try. Retrieval quality and prompt structure account for most of the achievable gain; fine-tuning is worth considering for domain-specific output formatting or a specialised reranker once retrieval is already strong. I will tell you if your case is one of the exceptions.
How do you handle documents with tables and figures?
With the same structure-preserving extraction I use for document parsing — tables kept as structured markup rather than flattened into prose, and figures captioned via a vision model so they become retrievable. Corpora heavy in tables are exactly where naive ingestion pipelines lose the most.
What about prompt injection from documents in the corpus?
Treated as a real threat, not a footnote. Retrieved content is delimited and handled as untrusted data, tool-calling surfaces are constrained, and instruction-like content in documents is neutralised. This matters most when any part of the corpus is user-submitted.
Start by measuring retrieval
If your RAG system is underperforming, the audit will tell you within two weeks whether the problem is ingestion, chunking, retrieval, or generation — with numbers rather than intuition.
Related
Document parsing & extraction
Document parsing and data extraction, built as production infrastructure
Turn unstructured PDFs, scans, and emails into validated structured data. Custom document parsing pipelines with schema design, confidence scoring, human review queues, and measurable accuracy.
Self-hosted VLM deployment
Deploy vision-language models on your own hardware
Set up open-weight vision-language and OCR models on your own GPUs or private cloud — Qwen3-VL, InternVL, and similar. Sizing, quantization, vLLM serving, throughput tuning, and accuracy validation.
Advisory
Document AI consulting: architecture, build-versus-buy, and accuracy strategy
Independent consulting on document AI architecture, model and vendor selection, accuracy measurement, cost modelling, and build-versus-buy decisions — from an engineer who ships extraction systems.
Custom OCR engineering
Custom OCR development for documents that off-the-shelf OCR gets wrong
Custom OCR pipeline development for scanned, photographed, and low-quality documents — layout-aware extraction, confidence scoring, vision-model escalation, and ground-truth accuracy benchmarks.
This page is also available as markdown for AI agents: /services/rag-development.md · index at /llms.txt. Canonical URL: https://parsemystatement.com/services/rag-development.