Retrieval-augmented generation (RAG)

Retrieval-augmented generation, or RAG, pairs a language model with a search step that runs before generation. Instead of answering from memory alone, the system pulls relevant passages out of an external document store and drops them into the prompt. The model then writes its answer grounded in that retrieved text.

A typical setup splits source documents into chunks of a few hundred tokens. An embedding model converts each chunk into a vector, and those vectors go into a database. At query time the system embeds the question, pulls the 3 to 10 closest chunks by similarity, and places them ahead of the question in the prompt. Facebook AI researchers named the technique in a 2020 paper, and it is now the standard way to ground a model in documents it never saw during training.

Running RAG locally means keeping two models resident instead of one, an embedding model for search and a language model for the answer. The vector index sits alongside them. Size the GPU for the language model as usual. Retrieval adds CPU and disk load more than GPU load once the index is built.

Sources

Last verified August 29, 2026.