What Is RAG (Retrieval-Augmented Generation)?

Why does RAG exist?#

RAG exists because a language model's built-in knowledge is fixed at training time, hard to update and hard to trace to a source. By looking up current documents at the moment of asking, a RAG system answers from material you control and can point to the passage behind each answer, which makes the output easier to check.

The research paper that introduced the term, published on arXiv in 2020, set out the problem clearly. Large pre-trained language models store factual knowledge in their parameters, but their ability to access and precisely use that knowledge is limited. The authors also described two open problems: providing provenance (a source) for the model's decisions, and updating its world knowledge.

For a workplace, those problems are practical ones. A general assistant does not know your leave policy, your latest price list or the procedure your team agreed last month. Even when it sounds confident, you cannot see where its answer came from. Confident answers that are not supported by any source are called hallucinations, and they are the main risk RAG is designed to reduce.

How does RAG work, step by step?#

A RAG system works in two phases. First, you prepare your documents so they can be searched quickly. Then, for every question, the system retrieves the most relevant passages, adds them to the prompt with clear instructions, and asks the model to answer using only that material, ideally with a note of which passage it used.

1. Prepare the documents. Collect the approved documents and split them into short passages, a step called chunking. Each chunk is usually turned into an embedding, a list of numbers that represents its meaning, and stored in a vector database or similar index.
2. Retrieve. When someone asks a question, the system turns the question into an embedding too and finds the chunks whose meaning is closest. This is semantic search: it matches meaning rather than exact words.
3. Augment the prompt. The retrieved chunks are added to the prompt, along with instructions such as “answer only from these passages”.
4. Generate. The model writes an answer based on the question and the passages it was given.
5. Show the source. A good system returns the answer with the passages or document names it relied on, so the reader can check.

Anthropic's documentation describes features built for this last step. Its citations feature returns the exact passages that support each claim, and its search result blocks let the model cite your own content with the source and title you provide, which the documentation describes as suited to RAG applications. Other providers offer similar features, but check each provider's own documentation.

What did the original research paper propose?#

The 2020 paper proposed combining two kinds of memory: the knowledge stored inside a pre-trained language model, and an external index of documents the model could search. The authors used a dense vector index of Wikipedia, accessed with a pre-trained neural retriever, and fine-tuned the combined system on knowledge-intensive tasks such as open-domain question answering.

The paper called these two kinds of memory parametric (inside the model's parameters) and non-parametric (the searchable index). It compared two versions: one that uses the same retrieved passages for the whole answer, and one that can draw on different passages for each part of the answer. For language generation tasks, the authors report that their RAG models produced more specific, diverse and factual language than a comparable model without retrieval.

Today's RAG tools are usually simpler to set up than the research system. Many do not retrain the model at all: they retrieve passages and place them in the prompt. The core idea is the same, though. Keep knowledge in documents you can read, update and cite, and let the model do the writing.

How does RAG compare with asking a model directly?#

Compared with asking a model directly, RAG changes where the knowledge comes from. A plain model answers from what it learned in training. A RAG system answers from passages it retrieved from your documents just before answering. That difference affects how you update the system, how you check answers and where things go wrong.

What does a worked Singapore example look like?#

Imagine a small property management firm in Toa Payoh whose officers often ask the same questions about the firm's own procedures, such as how to log a defect or who approves a contractor quote. This illustrative example shows how the firm might plan a simple internal RAG assistant, without claiming any results.

A sensible plan might look like this:

• Pick the documents. The operations lead chooses the current procedures handbook and a set of approved forms guides. Old drafts and personal notes are left out, so the assistant cannot retrieve outdated rules.
• Remove personal data. Resident names, unit numbers and phone numbers do not belong in a procedures assistant. The team keeps personal data out of the document set unless the tools have been approved for it.
• Write clear instructions. The prompt tells the model to answer only from the retrieved passages, name the section it used, and say “The handbook does not cover this, please ask the operations lead” when nothing relevant is found.
• Test with real questions. Officers write down questions they actually ask. For each one, the team checks two things: did the system retrieve the right section, and did the answer match it?
• Keep a person responsible. Answers about money, safety or contracts are checked by a supervisor before anyone acts on them.

The assistant will only be as good as the handbook. If two sections contradict each other, retrieval may surface either one, so tidying the source documents is part of the project.

How do you make RAG answers trustworthy?#

You make RAG answers trustworthy by controlling the documents, telling the model to stay within the retrieved material, showing the source of every answer and testing retrieval and answers separately. None of these steps makes a system perfect, but together they make errors easier to spot before they reach a customer or colleague.

Anthropic's guidance on reducing hallucinations lists techniques that fit RAG well: give the model permission to say it does not know, ask it to use direct quotes as grounding, have it cite a source for each claim, and tell it to use only the documents provided rather than its general knowledge. The same guidance notes that these techniques reduce hallucinations but do not eliminate them, and that critical information should always be validated.

In practice this means building evaluation into the project from the start: a fixed list of test questions, the answer you expect and the passage that should support it. Our guide on testing AI-generated outputs shows how to set up that routine.

Common mistakes#

The most common RAG mistakes are indexing outdated or conflicting documents, splitting text in unhelpful places, testing only the final answers and not what was retrieved, letting the model fall back on general knowledge, and putting confidential or personal data into tools that have not been approved for it.

• Messy source documents. Duplicates, old versions and contradictions all end up in answers. Clean the document set first.
• Poor chunking. Chunks that cut a rule in half, or lump several topics together, lead to partial or muddled answers.
• Checking only the answer. A fluent answer can rest on the wrong passage. Look at what was retrieved as well.
• No “I don't know” path. Without one, the model may fill gaps from general knowledge and sound just as confident.
• Forgetting to re-index. When a policy changes, the index must be updated, or the assistant keeps quoting the old version.
• Unapproved data handling. Check where documents and questions are stored before any confidential or personal data goes in.

What to learn next#

Once you understand RAG, the next steps are learning how retrieval finds the right passages, and deciding when RAG is the right tool at all. After that, the most useful skill is testing: checking that a RAG assistant retrieves the right material and answers from it before anyone relies on it.

• Read how embeddings and vector databases power the retrieval step.
• Compare prompting, RAG and fine-tuning to choose the right approach.
• Browse more guides in AI for builders, or follow the AI learning pathway.
• To build and test a simple AI app yourself, see GenAI Builder Sprint, a 2-day classroom course for adults with no previous AI or programming experience needed, although formal entry requirements apply.

Sources#

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv, accessed 8 November 2026
2. Search results, Anthropic (Claude Platform Docs), accessed 8 November 2026
3. Citations, Anthropic (Claude Platform Docs), accessed 8 November 2026
4. Reduce hallucinations, Anthropic (Claude Platform Docs), accessed 8 November 2026
5. Embeddings, Anthropic (Claude Platform Docs), accessed 8 November 2026

Frequently asked questions#

What is RAG in simple terms?+

Retrieval-augmented generation (RAG) is a way of building AI tools where the system first looks up relevant passages in a trusted set of documents, then gives them to a language model to write its answer. The model answers from your material instead of relying only on what it learned in training.

Where did the idea of RAG come from?+

The term comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, published on arXiv. It combined a pre-trained language model with a searchable index of Wikipedia passages, found by a neural retriever.

Why not just ask a language model directly?+

A language model stores knowledge in its parameters, which makes that knowledge hard to update and hard to trace to a source. The 2020 paper describes providing provenance for decisions and updating world knowledge as open problems. RAG keeps the knowledge in documents you control.

What are the steps in a RAG system?+

Prepare the documents by splitting them into chunks and indexing them, retrieve the chunks most relevant to the question, add them to the prompt with clear instructions, generate the answer, and show which passages the answer used.

Does RAG stop AI hallucination?+

No. It reduces the risk by giving the model relevant material and asking it to stay within that material, but techniques like this do not eliminate hallucinations. Test the system with real questions and check critical answers.

What are the common mistakes when building RAG?+

Indexing outdated or conflicting documents, splitting text in unhelpful places, testing only the answers and not what was retrieved, letting the model fall back on general knowledge, and putting confidential or personal data into tools that are not approved for it.