Explained · Engineering
RAG (retrieval-augmented generation)
Instead of hoping the model memorised the right facts, you hand it the right documents at question time. The system searches your files, pastes the best passages into the prompt, and the model answers from those. It is why a chatbot can cite your company wiki without being retrained on it.
The problem it solves
A model's knowledge is frozen at training time. Your company wiki, this morning's prices, the contract you signed last week — none of it exists inside the weights, and retraining the model every time your documents change would be slow, expensive, and never quite up to date. But there is one thing you control completely at question time: the prompt. RAG is the engineering trick built on that fact — if the model cannot already know your documents, hand it the right ones at the moment you ask.
What actually happens
Ahead of time, your documents are split into chunks, and each chunk is converted into an embedding — a long list of numbers that places its meaning as a point in space, where passages about similar things sit near each other. When a question comes in, it is converted the same way, the system finds the stored chunks nearest to it, and the best few are pasted into the prompt alongside the question. The model then answers by reading what it was handed. Nothing about the model itself changed — it is an open-book exam where a very fast librarian slides the right pages across the desk.
Where it breaks
The weak link is rarely the model — it is the search. If retrieval pulls the wrong passages, the model answers fluently from the wrong pages, and a confident answer with a citation is more dangerous than an honest "I don't know". Chunking adds its own failures: split a document in the wrong place and the answer straddles two chunks, neither of which ranks well on its own. And similarity search finds passages that resemble the question — ask something that requires reading everything, like "summarise what changed this quarter", and fetching the five nearest paragraphs quietly misses the point. When a RAG system disappoints, look at what was retrieved before blaming the model.
RAG or fine-tuning?
A useful rule: retrieval is for knowledge, fine-tuning is for behaviour. Facts that change, private data, anything you need traced back to a source — that is RAG, because you can update the library without touching the model. Tone, format, house style — that is fine-tuning, because behaviour has to live in the weights. The two combine happily. And although ever-larger context windows tempt you to skip the search and paste everything in, every pasted page is input tokens you pay for on every single question — retrieval exists to keep the open book short.