What is RAG, and when should you use it instead of sending the whole document in the prompt?

The interviewer wants to know whether you understand RAG as a process with steps, not just a fashionable name, and whether you can weigh the trade-offs.

RAG, short for retrieval-augmented generation, has three steps. One: split the documents into small passages, done once in advance. Two: when a question comes in, find the few most relevant passages, either by counting keywords or by comparing meaning. Three: put those passages into the prompt and only then call the model, with an instruction to answer based only on those passages.

Use RAG when:

1. The documents are long enough that sending all of them is expensive. Everything in the prompt is sent again on every call, so cost and latency grow with the length of the document, while RAG sends only a few passages.
2. The documents change often. You edit the document and update the index, instead of rewriting the prompt.
3. You need to know what the answer is based on. Each passage has a name, so you can make the model cite its source and check it afterwards.

Don't use RAG when the document is only a few pages. Sending the whole document is simpler and removes one place that can go wrong.

An important point to make: RAG adds one more place for errors, the search step. If that step doesn't bring back the right passages, the model may answer incompletely or guess, while the answer still sounds perfectly fluent. So when RAG answers wrongly, the first thing to do is look at which passages were sent, not to edit the prompt right away.

Bonus points: give a concrete example, such as a customer asking "how long until I get my money back" while the document says "refund period". Keyword search misses this because the two phrasings share no words, and that is why people move to searching by meaning with embeddings.