Everything You Know About RAG
Is Half True.

A meeting room. Someone says it: "We're using RAG now, so the AI answers from our documents." Heads nod. Coffee is sipped. Nobody pushes back.

I've been in that meeting. You probably have too.

And nobody in that room is wrong. That's the problem.

RAG, short for Retrieval-Augmented Generation, is a way of building AI systems in which the model looks up relevant material, most often your own documents, before it answers. A student allowed to bring the textbook into the exam.

Except the textbook doesn't take the exam. The student does.

Almost everything people believe about RAG is like this: half true, with a true half strong enough that nobody goes looking for the rest. But the missing half is where the value lives. It's the difference between a system that impresses in the demo and one your team still trusts six months in.

Five beliefs. Both halves of each.

Half-truth 1: "RAG stops the AI making things up"

Start with the biggest one.

The true half. RAG sharply reduces hallucination: the model's habit of producing confident, fluent, wrong answers. Instead of answering from everything it half-remembers, the model is handed passages from your library and told to answer from those. The result is what the industry calls grounding: an answer anchored to material you can actually inspect. It's a big part of why RAG has become the default for serious document work, and the reputation is earned.

The missing half. Reduced is not eliminated.

The step doing the work is retrieval: before answering, the system searches the library and pulls out the passages that look most relevant. Picture a library assistant sprinting to the shelves.

Now watch the assistant stumble, three ways. They grab the wrong page and answer beautifully from it. They grab the right page and skim it badly. Or, the strange one: they grab the right page, set it on the desk, and answer from memory anyway. In early research that traced RAG answers back to their causes, this ignored-evidence pattern showed up more often than failed retrieval, at least in the settings studied so far.

The homework was done. It just wasn't consulted.

How big is the remainder? The best numbers we have come from law. When Stanford researchers tested the RAG-based research tools sold to lawyers, roughly one answer in five still contained invented content. For some of the tools tested, one in three.

Law isn't health economics. But it's the nearest domain with published numbers: regulated, citation-driven, no room for invention. Read it as a proxy, not a prediction. Nobody has run our version of that study yet.

And the surviving errors share a feature: they arrive with a source attached, dressed exactly like the answers that don't need catching. The citation is what makes you relax.

Hold that thought. It comes back in half-truth five.

RAG turns "the AI makes things up" from a constant problem into a manageable one. Managing it is still part of the job.

Half-truth 2: "The AI learns from you"

A training session, almost word for word: "I corrected how it commented on my document. Next time it'll comment the way I like."

It won't. Tomorrow it meets you for the first time. Again.

The half that's true: the system really does get better as you invest in it. Better documents in, better answers out. Sharper instructions, sharper behaviour. Over months, a well-tended system becomes visibly more useful. From the outside, that looks exactly like learning.

But in the standard setup, the model itself never changes. A RAG system is a goldfish with a filing cabinet: spectacular at looking things up, remembers nothing. Query ten thousand is answered by the same model as query one.

What people are picturing has a name: fine-tuning, giving the model additional training so that the model itself permanently changes. RAG is letting the student bring the book into the exam. Fine-tuning is sending the student back to school. Different projects, different costs, and a remarkable share of vendor-meeting confusion comes down to not knowing which one is on the table.

Here's the useful side of the missing half. Because the behaviour lives in the setup, it lives somewhere you can reach. The instructions are a document you can read and edit. The library is a folder you can curate. Nothing important is hiding inside the model. Want consistency? Write it down once, in the setup.

Feed the filing cabinet. Not the goldfish.

The system does improve. The improvement lives in the library and the instructions, not in the model's memory of you.

Half-truth 3: "The AI reads your documents"

"The answer is RIGHT THERE on page 47."

Every RAG user says this eventually. Usually out loud. Usually alone.

And they're half right to be annoyed. The system really does answer from your documents; that's the whole point, and when it finds the right passage it can be startlingly precise, down to the exact figure in the exact table.

But "your documents" means selected passages from your documents. Files in a RAG system are pre-cut into searchable pieces. The industry calls this chunking, and for once the jargon is honest: your 200-page report is literally in chunks. Ask a question and retrieval fetches maybe five or ten of them. The model answers from those.

It's like asking a colleague about a report they've never opened while handing them three flagged pages. The answer might be excellent. It comes from the flagged pages.

So: page 47. The AI didn't miss it. It never saw it.

Once you know that, the fix is usually simple. Ask more specifically. Point it at the section. Check which passages it was given. The mystery evaporates.

Where the cuts fall matters too. Split a recipe in the wrong place and the ingredients end up separated from the instructions; the AI happily tells you to bake for forty minutes without knowing what you're baking. In evidence work the same split looks like this: the number in a table, the caveat two pages later. Keeping those pairs together takes deliberate care.

Who takes that care? That's the next belief.

The AI answers from your documents, passage by passage. Understanding which passages it saw is half of understanding any answer it gives.

Half-truth 4: "RAG is something you set up"

Same model. Same documents. Two teams. One builds a system the whole department leans on. The other builds one that quietly falls out of use by spring.

What happened?

The half that's true: RAG really is something you set up, and the setup is where quality gets decided. Of the five beliefs, this one is closest to fully true. What "set up" hides is the tense. It isn't a weekend and a checklist. It's decisions that keep needing to be made: which documents go in, which get culled, where the chunks are cut, what the standing instructions say.

Nobody demos these parts. Every demo is the model being eloquent.

And the part that needs the most ongoing attention is the library, because the system is completely loyal to it. Feed it the 2019 version of a guideline and it quotes the 2019 version with total sincerity. RAG has no idea your folder named FINAL_v3_old_DO_NOT_USE is radioactive. It retrieves what's there, not what's current. A shared drive holding every draft since the beginning of recorded time is not yet a knowledge base. Curating it into one is real work, and it never fully stops.

Curate the shelf.

The model is the engine. The decisions around it are the vehicle. Two teams with the same engine can build very different vehicles.

Half-truth 5: "Citations mean you can trust the answer"

This one is my favourite, because the true half is so close to fully true.

Citations are the feature that makes these systems fit for serious work. An answer that points to its source is worth far more than one that doesn't.

But remember where half-truth one left you: the errors that survive come dressed in sources too. So a citation can't be a licence to stop reading. What is it, then?

Speed.

Verifying a claim buried in a long document used to mean an afternoon of digging. With the exact passage a click away, it can be the work of a moment. That, more than anything, is what RAG changes: not the end of checking, but a drastic drop in what checking costs.

And for this field, checking the source isn't even a new discipline. It's the discipline. RAG just made it fast.

Citations don't end verification. They make it cheap. The teams that click them get the full value of the system.

The other half

Put the five together and they reduce to three habits.

Verify, because the errors that survive look exactly like the answers that don't need catching, and the citations have made checking cheap. Invest in the setup, because the library and the instructions are where the system's behaviour actually lives, and where it quietly goes stale if ignored. And know what the system saw, because an answer is only ever as good as the passages behind it.

(One footnote for the road: newer systems with much larger working memory can sometimes swallow whole documents at once, so half-truth three is slowly becoming more true. Slowly.)

None of this is a case against RAG. It's the case for taking it seriously. The teams getting real value aren't the ones who believed the half-truths hardest. They're the ones who learned the other half and built it into how they work.

Everything you know about RAG is half true. The other half is where the value is.