This article is part of our AI Agents for Business hub. Want help applying it? Talk to eSEOspace.
RAG, or retrieval-augmented generation, is a way to make an AI answer from your own documents. The system finds the relevant passages first, then hands them to the model to write the reply.
It is how an agent can know your return policy or your price list. This guide is part of our AI agents for business hub. It was checked against vendor and OWASP documentation on 9 November 2026.
Key Takeaways
- RAG finds relevant text in your content and gives it to a language model along with the question.
- Your documents are not trained into the model, which is why updates can show up without retraining.
- Quality depends heavily on how your documents are prepared and split for search.
- Access control is a serious business risk, because the AI must not retrieve what a user may not see.
- RAG does not stop prompt injection, and a poisoned document can skew answers.
The simple version
Picture an employee taking an open-book test. They do not memorize your binder. They look up the right page, then answer from it.
Microsoft, which sells a search product for this purpose, defines it in one line: "Retrieval-augmented generation (RAG) is a pattern that extends LLM capabilities by grounding responses in your proprietary content." An LLM, or large language model, is the AI that reads and writes text.
The term comes from a 2020 research paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," by Patrick Lewis and colleagues.
How it works, step by step
There are two stages. One happens ahead of time, and one happens each time someone asks a question.
Getting your content ready
- Your documents are collected from places such as shared drives, databases and web pages.
- Long documents are cut into smaller pieces, called chunks. Microsoft says to use chunking so that portions can be matched independently.
- Each chunk is converted into a list of numbers, called an embedding, that represents its meaning.
- The chunks and embeddings go into a searchable index.
Answering a question
- The person's question goes to the search index.
- The system finds the closest chunks.
- Those chunks go to the model with the question.
- The model writes an answer based on them, ideally with a citation.
Why not send every document each time? Microsoft's page gives the reason: "LLMs accept limited token inputs." A token is a small piece of text, so the system has to pick what matters.
Microsoft also recommends hybrid search, which combines keyword and meaning-based search. Exact words matter for a part number, and meaning matters when a question uses different wording from the document.
RAG compared with the alternatives
| Approach | What it does | Good fit | Watch out for |
|---|---|---|---|
| Plain chatbot | Answers from what the model already knows | General questions | Knows nothing about your business |
| RAG | Looks up your content, then answers | Policies, manuals, product details | Depends on content quality and permissions |
| Connected tools | Calls a live system for current data or actions | Order status, account changes | Needs tight permissions |
RAG suits content that reads like documents. Live data, such as today's stock level, is better fetched by a connected tool. Our comparison of MCP and RAG explains how the two fit together.
Anthropic, which sells AI models, describes the building block of agents as a model enhanced with retrieval, tools and memory. So RAG is often one part of an agent, not a replacement for one.
What actually goes wrong
Bad source material. Microsoft's page says: "RAG quality depends on how you prepare content for retrieval." Outdated or contradictory documents will produce outdated or contradictory answers.
Permissions that do not carry over. This is the one to take seriously. Microsoft puts it this way: "Opening private content to LLMs requires granular access control." Its example is finance data that should stay with the finance team, even when an executive asks the chatbot.
OWASP, a security community, lists this as LLM08, Vector and Embedding Weaknesses. It names unauthorized access, leaks between users in shared databases, and data poisoning.
It says: "Implement fine-grained access controls and permission-aware vector and embedding stores." It also recommends validating sources and keeping detailed retrieval logs.
Prompt injection through documents. Text inside a retrieved file can carry instructions. OWASP's prompt injection entry gives the example of a tampered document in a RAG repository that skews answers. It also says RAG does not fully mitigate prompt injection.
Too much trust in the answer. RAG gives the model better material, but it does not guarantee a correct answer. Show citations so a person can check the source.
What it costs to run
Prices vary by how you build it. Here are three published examples, checked on 9 November 2026.
| Vendor | Item | Listed price |
|---|---|---|
| OpenAI | File search tool calls (Responses API only) | $2.50 per 1,000 calls |
| OpenAI | File search storage | $0.10 per GB per day, first 1 GB free |
| Microsoft Copilot Studio (standard harness) | Tenant graph grounding | 10 Copilot Credits per message |
Those come from OpenAI's pricing page and Microsoft's billing rates. Microsoft says a tenant-graph-grounded reply could use 12 credits, once 2 credits for the generative answer are added. Microsoft 365 Copilot licensed users may see no charge, with exceptions.
These are examples, not a full estimate. You also pay for the model that writes the answer, and for the work of preparing your content.
Questions to ask before you connect your documents
- Which documents will it read, and who decided that?
- Does it respect the permissions people already have on those files?
- How does a change to a document reach the index, and how long does that take?
- Will answers show which document they came from?
- Who can add content to the knowledge base?
- Is retrieval logged, and who reviews the log?
If you already read our older article on RAG systems for CRM, this guide is the general version.
Next step
Pick one small set of clean documents with simple permissions, and test real questions against it. If you want a RAG system built into an agent or a CRM, see our AI agent development service.
Frequently asked questions
Does RAG train the AI on my data?
RAG works by retrieving your content when a question is asked, not by training the model on it. Check each vendor's data terms separately, because what they store or log is a different question.
Is RAG the same as fine-tuning?
No. Fine-tuning changes the model itself. RAG leaves the model alone and gives it relevant text for each question.
Can RAG stop AI from making things up?
It can reduce the problem by giving the model real material, but it cannot eliminate it. A model can still misread a source, and the source can be wrong. Citations let a person check.
Do I need a vector database?
You need some kind of searchable index. Microsoft recommends combining keyword and vector search, so a vector store is one common part. Some vendors include this in the product.
How is RAG different from an AI agent?
RAG is a way of finding information. An agent decides what steps to take and uses tools to take them. An agent can use RAG as one of its tools.
