What Is RAG? Retrieval-Augmented Generation Explained

RAG, short for retrieval-augmented generation, is a way to make an AI model answer from documents it looks up at the moment you ask, instead of relying only on what it absorbed during training. If you have ever dropped a handbook into a chat app and asked it questions, a RAG-style pipeline can be what runs underneath.

Three things are worth knowing before the detail:

• It's a pattern, not a product. Nobody sells "RAG" as one thing. Search tools, support bots and chat apps each build their own version.
• It has three steps. Retrieve the relevant passages, add them to the prompt, then let the model write the answer.
• It lowers the odds of a made-up answer; it doesn't remove them. The answer is only as good as the passages that were fetched.

The rest of this explains how RAG works, how it differs from a long context window and from fine-tuning, where you already meet it, and how it fails.

What is RAG, and where does the name come from?

The term comes from a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and eleven co-authors, accepted at NeurIPS 2020. The paper paired a language model's built-in knowledge with a searchable index of Wikipedia, so the model could consult a source instead of answering from memory alone.

Today the name is used more loosely. It usually means the same idea in an everyday pipeline: Anthropic describes RAG as a method that retrieves relevant information from a knowledge base and appends it to the user's prompt. AWS puts the benefit as extending a model to an organization's own knowledge base without retraining the model.

That is the whole appeal. A model's training data stops at a cutoff date and never included your private files. RAG gives it a way to read them at question time.

How RAG works: retrieve, augment, generate

Picture a company handbook and a staff chatbot. An employee asks, "How much parental leave do I get?" The model was never trained on your handbook, so left alone it would guess. With RAG, three things happen in order.

The model never "learns" the handbook. It reads the relevant pages fresh each time, then forgets them. That has two useful consequences. When the handbook changes, you update the documents and the next answer reflects it. And because the answer came from specific passages, the system can show you which ones, so you can check.

Google Cloud's explainer makes the same point from the hallucination side: providing facts to the model as part of the prompt can mitigate gen AI hallucinations, provided the most relevant facts are the ones supplied. That condition is where most of the difficulty lives, and it's why the next section matters.

Embeddings and vector search, in two short paragraphs

Retrieval is the hard step. Your documents are first cut into small chunks. Anthropic's primer says these are usually no more than a few hundred tokens each. An embedding model then turns each chunk into a list of numbers, a vector, that encodes its meaning, and the vectors go into a vector database. Think of coordinates on a map: chunks with similar meaning land near each other. When you ask a question, it's converted the same way, and the system fetches the chunks that sit closest to it.

The payoff is that matching works on meaning, not exact words. OpenAI's retrieval guide describes this as semantic search, which surfaces similar results even when they share few or no keywords. A question about "time off after a baby" can find a passage titled "Parental leave policy." But meaning-based search can miss exact strings, such as a product code or an error number, which is why Anthropic notes that many systems pair embeddings with an older keyword-ranking method, and why OpenAI's file search tool uses both semantic and keyword search. Anthropic also describes an optional reranking pass, which re-scores the first batch of results so only the best ones reach the model.

RAG vs a long context window vs fine-tuning

These three get confused because they all aim at "make the model know more." They work in different places.

Versus a long context window. The context window is the text a model can read in one go. If your material fits, you can skip retrieval entirely: Anthropic's own post says that a knowledge base small enough to fit in the prompt can simply be included whole, with no RAG needed, and that you need something more scalable as it grows. Google makes the matching point, recommending RAG when you need more information than fits in the window, or to cut the tokens you send and save time and cost. And bigger isn't automatically better. Anthropic's context window docs warn that as token count grows, accuracy and recall degrade, which they call context rot. Our context window explainer covers the mechanics.

Versus fine-tuning. Fine-tuning changes the model by training it on examples. OpenAI's fine-tuning documentation says it can make a model format responses consistently or handle novel inputs. That's a behavior fix more than a knowledge fix. AWS makes the contrast directly: retraining a foundation model on domain-specific information is expensive, and RAG is the more cost-effective way to introduce new data. One date-sensitive note: as of October 2026, that same OpenAI page says the company is winding down its fine-tuning platform and has closed it to new users, so check what your vendor actually offers before planning around it.

The three combine. A model can be fine-tuned for tone, fed retrieved passages for facts, and given a large window to hold them. Most non-technical readers will only ever use the middle one, because the product decides it for them.

Where you already meet RAG

You rarely build RAG. You meet it inside products:

• Chat apps with project or knowledge features. Claude's help center explains that when a project's knowledge approaches the context window limit, Claude automatically enables RAG mode, searching your uploaded documents instead of loading everything at once. If the knowledge later shrinks below the threshold, it can switch back to loading everything.
• "Chat with your files" tools built on developer APIs. OpenAI's file search lets a model retrieve information from previously uploaded files in a vector store, and the model decides when to call it.
• Support bots and help-center assistants. Anthropic's own example is a customer support chatbot that needs knowledge about the specific business it serves.
• Company-wide search. Tools that answer questions across Slack, drives and wikis are the largest version of the idea. AI-driven enterprise search covers what separates them, and permissions turn out to matter more than the model.

Where RAG goes wrong

A RAG system fails in the retrieval step far more often than in the writing step. The usual causes:

1. The right passage never gets fetched. Google's explainer is blunt: if the retrieved information is irrelevant, the generation can be grounded but off-topic or incorrect. Anthropic shows one cause. A chunk that says "revenue grew 3% over the previous quarter" doesn't say which company or period, so it's hard to retrieve for the right question. Anthropic's Contextual Retrieval method adds a short explanation to each chunk before indexing to fix this.
2. Too few or too many passages. OpenAI notes that limiting the number of results cuts token use and latency but can reduce answer quality. Anthropic notes that more retrieved information can distract the model, so there's a limit.
3. A stale index. AWS points out that if the external data goes out of date, you have to update both the documents and their embeddings, either continuously or in batches.
4. Questions that need the whole document. A system that answers from a handful of fetched passages is a poor fit for "summarize everything in this file." That's a case for the context window instead.
5. Permissions and messy sources. AWS notes that developers can restrict sensitive information retrieval to different authorization levels, which only helps if someone sets it up, and a search over contradictory documents produces a confident answer built on the wrong one.

RAG reduces hallucinations; it does not end them. Our guide to AI hallucinations covers why a model can still state something false with the right passage in front of it.

How to tell if a product uses RAG

Vendors rarely say "RAG" on the box. Signs that retrieval is running:

• Answers cite your own files. AWS lists source attribution as one benefit of RAG, and OpenAI's file search returns annotations that reference the files used.
• You can see a search step. Claude's help center says that in RAG-enabled projects you'll see Claude using a project knowledge search tool when it needs to find something in your uploads.
• It accepts more material than fits in the model's context window. That usually means a search layer sits between your files and the model.
• The docs use the vocabulary. "Vector store," "knowledge base," "grounding" and "semantic search" are all terms in the vendor pages above.

A quick test: ask something that only appears deep in one of your documents, then ask something that appears in none of them. A well-built system finds the first and says it can't find the second. If it confidently answers both, treat the citations with suspicion.

Where this leaves a non-developer

Building a real RAG pipeline means choosing chunk sizes, an embedding model and a vector store, which is engineering work. Using one is easy: upload files into a chat app and ask.

The gap appears when you see someone's polished "chat with my documents" setup online and can't reproduce it. That's the gap Taku is aimed at: mirroring an AI setup someone already got working into a desktop workspace, so you run it instead of rebuilding it. Browse what's there in the free app library. Taku is in Beta, and the Mac app is available now.

FAQ

What does RAG stand for?

Retrieval-augmented generation. The system retrieves relevant information, augments the prompt with it, and the model generates an answer from both.

Is RAG the same as fine-tuning?

No. RAG changes what the model reads for each question and leaves the model untouched. Fine-tuning changes the model through extra training. RAG suits facts that change or need sources; fine-tuning suits consistent format or style.

Does RAG stop AI hallucinations?

It helps, because the model has real passages to work from, but it doesn't guarantee accuracy. If retrieval fetches the wrong passage, the answer can be grounded in the wrong material.

Do I need a vector database to use RAG?

Not as a user. Products that do RAG handle it for you. If you're building your own and your material is small enough to fit in the model's context window, you may not need retrieval at all.

Is RAG still needed now that context windows are huge?

For small sets of material, often not. For large or changing collections, a search step still helps, since vendors note that more text in the window can reduce accuracy and that sending fewer tokens saves time and cost.

Can RAG leak documents someone shouldn't see?

Yes, if retrieval isn't filtered by who is asking. Permission-aware retrieval is a design requirement, so ask how access is enforced before connecting sensitive sources.