Retrieval augmented generation: what RAG is and how it works

An AI model cannot answer questions about your company's refund policy, because your refund policy was not in its training data. There are three ways to fix that. Retrieval augmented generation is the one that works for most teams, and it is the only one of the three you can update by editing a document.

This guide covers the pipeline, the one decision that determines whether your system works or quietly returns nonsense, and an honest comparison against the alternatives.

What retrieval augmented generation is

RAG is a pattern where a search step runs before the generation step: the system finds relevant passages in your documents, pastes them into the prompt, and asks the model to answer using only those passages.

The model is not modified in any way. Nothing is trained. The improvement comes entirely from changing what is in the prompt — which is why you can fix a wrong answer by editing a source document and re-indexing, rather than by retraining anything.

That property is the whole reason RAG became the default. It makes AI answers auditable: you can show which document a claim came from, and you can correct it the same afternoon.

The five stages of a RAG pipeline

1. Ingest. Collect the source documents and extract clean text — PDFs, help centre pages, policy documents, support tickets. Extraction quality sets the ceiling for everything after it.
2. Chunk. Split each document into passages small enough to retrieve precisely. This is where most systems are won or lost; the next section covers it properly.
3. Embed and index. Convert each chunk into a vector — a numeric representation of meaning — and store it. Semantically similar text ends up close together in that space, which is what lets the search match "how do I get my money back" to a passage titled "Refunds".
4. Retrieve. At question time, embed the question the same way and pull the closest chunks. Most systems retrieve between three and ten.
5. Generate. Put the retrieved chunks into the prompt with an instruction to answer only from them, and to say so if the answer is not present.

Stage five deserves an explicit instruction, not a hope. Without "if the passages do not contain the answer, say that they do not", the model will fall back on general knowledge and produce an answer that looks sourced but is not.

Chunking: the decision that determines retrieval quality

Chunking is splitting documents into pieces before indexing, and it gets less attention than it deserves relative to how much it decides.

Chunks that are too large retrieve a page of text to answer one sentence's worth of question. The relevant line is in there, surrounded by noise that dilutes both the embedding and the model's attention.

Chunks that are too small lose the context that made them meaningful. A chunk reading "This does not apply to enterprise customers" is useless when the sentence explaining what "this" refers to ended up in a different chunk.

Practical starting points, to be tested rather than trusted:

• Size: roughly 200–500 words per chunk suits most prose documentation. Start in the middle and adjust based on what retrieval actually returns.
• Overlap: carry 10–15% of each chunk into the next so a sentence spanning a boundary survives in at least one piece.
• Split on structure, not character count. Break at headings and paragraph boundaries. A split in the middle of a sentence damages both chunks.
• Keep the heading with the body. Prefixing each chunk with its document title and section heading is a small change that noticeably improves retrieval, because it restores context the split removed.
• Tables need separate handling. A table cut in half is unreadable to the retriever and to the model. Keep them intact or convert them into sentences.

The fastest way to debug a RAG system is to look at the retrieved chunks before looking at the answer. If the right passage was never retrieved, the prompt is not the problem and no amount of prompt engineering will fix it.

RAG vs fine-tuning vs a long context window

These solve different problems, and picking the wrong one wastes a great deal of time.

	RAG	Fine-tuning	Long context
Best for	Facts that change, and answers that need sources	Style, format and task behaviour	One-off questions about a specific document
Updating	Edit the document, re-index	Retrain	Paste the new version
Can cite sources	Yes	No	Partly
Setup cost	Moderate	High	None
Per-question cost	Low — only relevant chunks are sent	Low	High — the whole document is sent every time
Scales to	Large document sets	N/A	Whatever fits the window

A useful rule: if the requirement is "the model should know something", that is RAG. If it is "the model should behave a certain way", that is fine-tuning. And if you have one document and one question, just paste the document — building a pipeline for that is engineering theatre.

Preparing your source data

Most RAG problems in practice are data problems wearing a technical costume.

Content exported from spreadsheets — product catalogues, FAQ sheets, pricing tables, support macros — is a common starting corpus, and it needs converting into structured records before it is chunked. Our CSV to JSON converter turns a header row and its data rows into an array of objects, so each row becomes a record you can attach metadata to rather than a line of undifferentiated text.

Once converted, check the structure parses cleanly in the JSON formatter — its colour-coded tree makes it obvious when a column has shifted, which is exactly the sort of silent corruption that produces confidently wrong retrieval later.

Two more preparation steps that repay the effort:

• Attach metadata to every chunk — source document, section, and a last-updated date. It lets you filter retrieval and show provenance in the answer.
• Remove superseded documents. Two versions of a policy in the index means the retriever will sometimes find the old one, and the answer will be confidently out of date.

Where RAG still fails

• Questions that need the whole corpus. "What are the main themes across these 400 tickets?" cannot be answered by retrieving five chunks. Summarisation at scale is a different pipeline.
• Contradictions in your own documents. If two files disagree, retrieval surfaces one of them and the answer inherits the conflict without flagging it.
• Vague questions. Retrieval quality depends on the question having enough signal to match against. "Tell me about the process" retrieves nearly at random.
• Keyword-exact lookups. Vector search matches meaning, so an exact product code or error number can be missed. Combining vector search with plain keyword search — hybrid retrieval — is the standard fix.
• Stale indexes. A document edited but not re-indexed is invisible to the system. Re-indexing has to be part of the publishing workflow, not a manual afterthought.
• It reduces invention, it does not eliminate it. A model given relevant passages can still blend them with prior knowledge. Requiring citations and spot-checking them is the only reliable guard.

Questions people ask about RAG

Does RAG stop AI from making things up?

It reduces it substantially, because the model answers from supplied text rather than memory. It does not eliminate it. Require the answer to cite which passage it used, and check those citations on anything that matters.

Is RAG the same as fine-tuning?

No, and they are not alternatives so much as answers to different questions. RAG changes what the model is given; fine-tuning changes how the model behaves. Teams often end up using both.

Do I need a vector database?

Not to start. With a few hundred chunks, an in-memory search is entirely adequate and much easier to debug. Add dedicated infrastructure when scale or latency actually requires it, not before.

What kinds of documents work best?

Well-structured prose with clear headings — policies, documentation, help articles, FAQs. Scanned PDFs without a text layer need OCR first, and heavily designed layouts often extract badly, which corrupts everything downstream.

How many chunks should I retrieve per question?

Three to five is a sensible default. More increases the chance the right passage is included, and also increases cost and the chance the model latches onto an irrelevant one. Tune it by inspecting retrievals, not by guessing.

Is this only worth it for large companies?

No. The pattern scales down cleanly — a support team with fifty help articles gets a genuinely useful assistant. What determines the value is whether your documents are good, not how many of them there are.

If you are building your first one

Start with twenty documents, not two thousand. Chunk at around 300 words with heading prefixes, retrieve five chunks, and — before you touch the prompt — read what the retriever actually returns for ten real questions. That exercise tells you more about your system than any amount of tuning downstream.