Tech Explained: RAG (Retrieval-Augmented Generation)
Ask ChatGPT or Claude a question about your own business, your products, your processes or your policies, and it either makes something up, says it doesn't have that information, or gives a generic answer that ignores your situation.
That follows from how these models are built. They were trained on data up to a certain point in time, and they have no knowledge of anything outside that training, including everything that makes your business unique.
Retrieval-Augmented Generation, RAG, is the most practical solution to this problem. It's the technology that allows AI systems to answer questions using your own documents, data and knowledge, accurately, in real time, without needing to retrain the model from scratch.
The name is clumsy, but it describes the mechanism in the order the work happens.
Look it up first
The system finds the passages in your own documents that relate to the question
Add it to the question
Those passages are handed to the model together with what was asked
Answer from what it read
The model writes the answer out of the material it was given, not out of memory
Everything else in this article is detail on those three words. The model doesn't change, the question doesn't change, and nothing gets retrained. What changes is that the model is holding the right pages when it starts writing.
Start with the problem
To understand why RAG matters, it helps to understand exactly what's missing without it.
- General world knowledge
- Reasoning and language ability
- Frozen at the training cutoff
- Internal documents and processes
- Current, proprietary data
- Recent events and decisions
Large language models are trained on vast amounts of text, books, websites, articles, code. In the process they develop impressive general knowledge and reasoning ability. But that training happens at a fixed point in time, after which the model's knowledge is frozen.
The left column is impressive and entirely generic. The right column is the part that actually decides whether an answer is useful to you, and none of it was ever in the training data.
That mismatch shows up as three concrete problems for businesses.
The knowledge cutoff problem. The model doesn't know about anything that happened after its training data was collected. Ask it about last quarter's results, a recent policy change or a new product launch, and it has no idea. The gap grows quietly until someone notices that the tool is confidently describing a world that has moved on.
The internal knowledge problem. The model has never seen your employee handbook, your client contracts, your product documentation or your internal processes. That knowledge exists nowhere in its training data, and no amount of clever prompting will conjure it up.
The hallucination problem. When an LLM doesn't know something, it doesn't always say so. Sometimes it fills the gap with a plausible-sounding answer that happens to be wrong. That's a nuisance in a chat window and a significant reliability risk in a business process, because a wrong answer in the house style is much harder to spot than an obvious blank.
RAG addresses all three, and it addresses them in the same way: by making sure the model is reading before it writes.
The library analogy
Suppose you hire an experienced consultant. They have deep general expertise, they've read widely, they reason well, they communicate clearly. But they've never worked in your industry, and they've never seen your internal documents.
You have two options.
Option A. Send them on a six-month training programme where they read every document your company has ever produced. That is expensive and slow, and they would have to repeat it every time something changed.
Option B. Give them access to your filing cabinet. When you ask a question, they look up the relevant documents first, read them, and then answer, combining your specific information with their general expertise.
RAG is Option B. Instead of retraining the model on your data, RAG gives the model the ability to retrieve relevant information at the moment it's needed, and use that information to construct an accurate, grounded answer. The consultant is no cleverer than before, but stops guessing about your business.
How RAG works
The mechanics are more straightforward than the name suggests. Think of the whole pipeline as a narrowing: it starts with everything you own and ends with a single answer built from the handful of passages that were actually relevant.
Everything you want the AI to know
PDFs, Word files, emails, web pages, database records
Chunks in a vector database
Each piece converted into an embedding that captures its meaning
The question, converted the same way
Compared against the index by meaning rather than by keyword
The top matching chunks
Only the passages that relate to what was asked
One grounded answer
Written from the retrieved material, alongside the original question
There are four stages, and almost every design decision in a RAG project sits inside one of them.
Stage 1: ingest and index. Your documents, PDFs, Word files, emails, web pages, database records, whatever you have, are broken into chunks and converted into numerical representations called embeddings. These embeddings capture the meaning of each chunk in a way that allows similar content to be found quickly. They're stored in a vector database, a specialised database designed for this kind of semantic search. In effect you build an index of everything the AI should be able to reference.
Stage 2: receive the question. A user asks a question. Before the LLM tries to answer anything, the system first converts the question into the same numerical format as the indexed documents, so that question and documents can be compared on the same terms.
Stage 3: retrieve relevant content. The system searches the vector database for the chunks of content closest in meaning to the question, rather than the ones that share its keywords. The top results are retrieved. This is the retrieval part of Retrieval-Augmented Generation, and it's the stage that most often decides whether the final answer is any good.
Stage 4: generate a grounded answer. The retrieved content is passed to the LLM alongside the original question. The model now has both its general reasoning ability and the specific, relevant information it needs to answer accurately. It generates a response grounded in your actual documents. This is the augmented generation part.
The model is the last link in the chain rather than the clever bit in the middle. By the time it starts writing, the work that decides the quality of the answer is done.
A concrete example
Take a question that every support and sales team gets asked, and give it to the same model twice.
The question: what is our refund policy for enterprise customers?
Without RAG, the LLM has no access to your terms and conditions. It either says it doesn't know, or it generates a plausible-sounding but entirely fabricated refund policy. Neither is useful, and the second is worse than the first, because it reads exactly like the real thing.
With RAG, the system retrieves the relevant section from your enterprise terms and conditions document. The model reads it and answers from it, along the lines of: enterprise customers are entitled to a full refund within 30 days of contract signature, and a pro-rated refund for unused months if they cancel with 60 days' notice after the initial term, according to the agreement last updated in March 2024. Those terms are an illustration rather than anyone's real policy. The point is where they came from.
The model is the same in both cases. The only difference is the information it had when it wrote the answer.
What makes RAG different from fine-tuning
When businesses first encounter the "how do we make AI work with our own data" problem, they often hear about two approaches: RAG and fine-tuning. They're solving related but different problems, and the third option, simply pasting context into the prompt, is worth putting next to them.
| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| What it does | Gives context in the prompt | Retrieves relevant docs at query time | Retrains the model on your data |
| Best for | Simple, contained tasks | Answering questions from large document sets | Changing how the model behaves or speaks |
| Knowledge updates | Manual, every time | Automatic, update your documents | Requires retraining |
| Cost | Very low | Low to medium | High |
| Accuracy on your data | Limited by context window | High | High |
| Hallucination risk | Higher | Significantly reduced | Reduced |
Read the knowledge updates row on its own and you have most of the argument. Prompting means somebody pastes the right document in every single time. Fine-tuning means the knowledge is baked in, so changing it means retraining. RAG means you update the document and the system is current from the next question onwards.
For most business knowledge problems, RAG is the right approach. Fine-tuning is better suited to changing the model's style, tone or domain-specific reasoning, not to giving it access to your documents. They are not rivals, and a system can use both, but only one of them answers the question "why doesn't it know our own policies".
Where businesses are using RAG today
The pattern that keeps coming back is the same: a large body of written material, a team that needs answers out of it faster than reading allows, and a question that is asked in slightly different words every time.
- Internal knowledge bases. An assistant that answers employee questions from HR policies, IT documentation, process guides and internal wikis. No more searching through SharePoint or asking a colleague who might also not know.
- Sales enablement. Sales teams asking natural language questions about products, pricing, competitive positioning or contract terms, and getting accurate answers instantly rather than hunting through a folder of PDFs.
- Customer support. Support agents or chatbots answering customer queries by retrieving the relevant section of your product documentation, FAQ or knowledge base, which reduces handle time and improves accuracy.
- Legal and compliance. Searching across large volumes of contracts, regulatory documents or compliance policies to find relevant clauses, identify obligations or flag potential issues, without reading every document manually.
- Research and due diligence. Analysts asking questions across a large corpus of reports, filings or research documents, surfacing what's relevant rather than reading everything sequentially.
- Financial and operational reporting. Connecting RAG to live data sources, databases, dashboards and reporting tools, so that business leaders can ask plain-language questions about performance and get data-grounded answers.
Limitations
RAG has clear limits, and knowing them is what separates a system people trust from one they quietly stop using.
Garbage in, garbage out
RAG is only as good as the documents it indexes. If your internal documentation is incomplete, outdated or contradictory, the AI will faithfully retrieve and repeat those problems.
Good RAG requires good document hygiene.
The chunking problem
Documents are broken into chunks for indexing. If a question needs information synthesised from many different chunks or documents, retrieval may miss some of it.
Smarter chunking strategies help, but it stays a design challenge.
Retrieval isn't always perfect
The system retrieves the most semantically similar content, but most similar is not always most relevant. A poorly phrased question, or a document that uses different terminology than the query, can surface the wrong content.
System design, testing and query refinement help significantly.
Security and access control
Building over sensitive documents means making sure users can only retrieve content they're authorised to see. A finance document shouldn't surface in a response to a junior employee asking something unrelated.
Access-controlled RAG requires careful architecture.
These four points decide whether a RAG project succeeds or disappoints, and all four are settled by design rather than by the choice of model.
Retrieval is the one that most rewards a smarter design. An agentic RAG system reasons about its own retrieval: it decides when to search, grades the material it gets back, and rephrases the query when the results aren't good enough. We walk through building one, step by step, in Building an intelligent agentic RAG system with LangGraph.
What this means in practice: buying or building
For most businesses the practical question is how to get access to RAG. There are broadly two routes.
Using a product with RAG built in. A growing number of tools have RAG inside the product without you ever needing to know what's happening underneath: Notion AI lets you ask questions across your Notion workspace, Microsoft Copilot searches and retrieves across your Microsoft 365 documents, Google Gemini in Workspace draws on your Google Drive content, Intercom Fin answers support queries from your knowledge base, and ChatGPT with file upload gives you temporary RAG over the documents you attach. These are the fastest route to RAG value. The trade-off is limited control over how retrieval works, what gets indexed and how the system behaves.
Building a custom RAG system. For businesses with specific requirements, proprietary data sources, complex access controls, integration with internal systems, or the need for high accuracy in a regulated context, a custom implementation gives full control. The core building blocks are always the same five:
- Document sources: your files, databases and APIs
- Embedding model: converts text to vectors (OpenAI, Cohere, open source)
- Vector database: stores and searches the embeddings (Pinecone, Weaviate, pgvector)
- LLM: generates the final answer (GPT-4o, Claude, Gemini, Llama)
- Orchestration: connects it all together (LangChain, LlamaIndex, or custom code)
This is where outside help pays off. The architecture decisions made at this stage have a significant impact on accuracy, cost and maintainability, and they are considerably cheaper to make well than to unpick later.
Key questions to ask before you build
Whichever route you take, these are the questions worth answering before anything gets built. They shape the architecture more than any tooling choice does.
What documents or data sources does the system need access to?
How frequently does that content change, and how will updates be handled?
Who needs access, and are there documents some users shouldn't be able to retrieve?
What does "good enough" accuracy look like here? A customer support bot and a legal compliance tool have very different requirements.
Is an off-the-shelf product sufficient, or do we need custom architecture?
How will we evaluate whether it's actually working?
The last one is the one most often skipped, and it's the one that tells you whether any of the others were answered well.
Where to start
If you want to experience RAG before committing to building anything, you can do it this month, with tools you already have.
Week 1. Upload a set of internal documents to ChatGPT or Claude and ask questions about them. That's RAG in its simplest form, and it's enough to understand the value proposition immediately.
Weeks 2 to 4. Explore whether a tool you already use, Microsoft Copilot, Notion AI or Google Workspace, has RAG features over your existing document library. Turn them on and test with the real questions your team asks regularly, not with questions invented for the test.
Month 2 onwards. If the value is clear and the off-the-shelf options aren't meeting your needs, that's the point to start thinking about a custom implementation, with the six questions above answered first.
Wondering which of your documents an AI should be reading?
Let's turn AI, data and software into measurable business value for your team.

