RAG quality is a data problem, not a model problem
A pattern we see constantly: a team builds a RAG system, the answers are mediocre, and the first instinct is to upgrade the model. It rarely helps, because the model was never the bottleneck. It was answering as well as anyone could given what retrieval handed it.
If the context is wrong, no model can save you. Here's where we actually look when RAG underperforms.
Chunking is a design decision, not a default
Most frameworks split documents every N tokens and call it a day. But a chunk is a promise: "this span of text is self-contained enough to be useful out of context." Fixed-size splitting breaks that promise constantly — half a table here, a sentence orphaned from its heading there.
We chunk along the document's own structure — sections, headings, list boundaries — and we attach breadcrumbs (document title, section path) to every chunk so the model knows where a fact came from.
Your corpus is dirtier than you think
Before tuning anything, read 50 random chunks from your index. In almost every engagement this exercise finds:
- boilerplate (headers, footers, legal disclaimers) drowning out signal,
- stale duplicates of documents that were updated elsewhere,
- scanned PDFs that OCR'd into soup,
- content that contradicts itself across versions.
A retrieval system faithfully surfaces whatever you indexed. Garbage in, confidently-worded garbage out.
Retrieval needs its own eval
Separate the two questions: did we retrieve the right context? and did the model use it well? You can't fix what you can't isolate.
| Question | Metric | Fix lives in |
|---|---|---|
| Right documents found? | recall@k on a labeled set | chunking, embeddings, hybrid search |
| Right context used? | answer faithfulness | prompting, reranking, context budget |
A labeled retrieval set of even 40 queries transforms the conversation from "the AI is wrong sometimes" to "recall dropped because half the HR docs were never re-indexed after the migration."
The unglamorous conclusion
The best RAG improvements we've shipped were data engineering: better ingestion, deduplication, structure-aware chunking, and a re-indexing pipeline that keeps the corpus fresh. The model got smarter because we finally stopped feeding it noise.
Buy the shovel before the bigger model.