What Is RAG? Retrieval-Augmented Generation Explained

RAG, or retrieval-augmented generation, is a way of making an AI model answer from your own documents instead of from memory. When someone asks a question, the system first retrieves the most relevant passages from your content, then gives those passages to the model and asks it to answer using only them, with citations. It is the standard method for building assistants on company knowledge that changes over time.

Key takeaways

• RAG has two halves: retrieval finds the right passages, generation writes the answer from them. Most quality problems are retrieval problems.
• The model is not retrained. Updating the assistant means updating the document index.
• Good RAG cites its sources and says "I don't know" when nothing relevant is found.
• Chunking, metadata, access control and freshness matter as much as the choice of model.
• You cannot tell whether a RAG system is good without a test set of real questions.

The problem RAG solves

A language model knows what was in its training data and nothing else. It has never seen your refund policy, your product manual or last week's price list. Ask it anyway and it will often produce a fluent answer that sounds right and is not yours.

You could paste the relevant document into the prompt each time. That works for one document. It does not work for several thousand pages, because there is a limit to how much text a model can take in at once, and because cost and response time grow with every page you send. RAG is the automated version of pasting in the right page: it finds the few passages that matter for this question and sends only those.

How RAG works

A RAG system has an offline part, which prepares your documents, and an online part, which answers questions.

Preparing the documents (indexing)

1. Collect. Gather the sources: PDFs, web pages, help articles, wiki pages, policy documents.
2. Extract and clean. Turn each file into plain text, keeping headings and tables where possible and removing navigation and boilerplate.
3. Chunk. Split the text into passages small enough to be specific and large enough to make sense on their own.
4. Embed. Convert each chunk into an embedding, a list of numbers that represents its meaning. Passages about similar things end up with similar numbers.
5. Store. Save the embeddings in a vector index together with the text and its metadata: source, title, section, date, and who may see it.

Answering a question

1. Embed the question in the same way as the chunks.
2. Retrieve the chunks closest in meaning. Many systems combine this with keyword search, which is better at exact terms such as part numbers and names.
3. Rerank. A second model re-scores the candidates and keeps the best few.
4. Generate. The model receives the question and the selected passages, with an instruction to answer only from them and to cite which passage supports each statement.
5. Return the answer with links to the sources.

The parts that decide quality

Chunking

Chunking is how you cut documents into passages, and it sets the ceiling for everything after it. Chunks that are too large bury the answer in unrelated text. Chunks that are too small lose the context that gives them meaning, such as a table row without its header. Splitting along the document's own structure, by heading and section, usually beats splitting at a fixed length. Carrying the document title and section heading into each chunk helps the retriever and the model understand what the passage is about.

Metadata and access control

Every chunk should record where it came from and who may read it. Access control has to be applied at retrieval time, as a filter, so that a passage the user is not allowed to see is never fetched. Telling the model "do not reveal salary data" is not access control.

Freshness

An index is a copy of your documents, and copies go stale. Plan how changes flow in: a scheduled sync, or an update whenever a document is edited. Deleted and superseded documents must be removed from the index, or the assistant will keep quoting the old policy.

Refusal

If retrieval returns nothing relevant, the correct answer is "I could not find that in the available documents". A system that always answers will sometimes invent. Decide what the assistant offers instead: a contact, a search link, a handoff to a person.

Evaluation

Write a set of real questions with the correct answer and the document that contains it. Then measure two things separately. Did retrieval return the right passage? Given the right passage, was the answer correct and supported by it? This tells you where to work. Rerun the set whenever you change chunking, the model or the prompt.

What RAG is good at, and what it is not

Works well	Works poorly
Questions answered by a specific passage: "what is the notice period for cancelling"	Questions over the whole collection: "how many of our contracts mention auto-renewal"
Knowledge that changes, such as policies and product details	Exact calculations over structured data, such as sales totals
Cases where a source link is required	Content that is contradictory or never written down
Large collections where no one remembers where things are	Scanned or image-heavy files with poor text extraction

For totals and counts, a query against the database is the right tool, and an assistant can use both. For changing how a model writes or formats, fine-tuning may fit better, a comparison covered in RAG vs fine-tuning.

A worked example

For example, an equipment supplier has installation guides, a warranty policy and a service bulletin archive. A technician asks, "Can the model X pump run on a single-phase supply?"

The system embeds the question and retrieves candidate passages. Keyword search contributes the chunks that mention the exact model name. The reranker keeps three: the electrical specification table from the installation guide, a paragraph on supply requirements, and a service bulletin about a wiring change. The model answers from those, states the supply requirement, mentions the bulletin, and links each source. If the guide for that model had not been indexed, the right response would be that no specification was found for that model, not a guess based on a similar pump.

Common misconceptions

• "RAG trains the model on our data." It does not. The model is unchanged. Your documents are looked up at question time.
• "RAG removes hallucination." It reduces it by grounding answers in text, but a model can still misread a passage or blend two sources. Citations and evaluation are how you catch that.
• "A bigger model fixes bad answers." If the right passage was not retrieved, no model can answer correctly from it.
• "Just upload everything." Outdated drafts and duplicates in the index produce outdated and conflicting answers.

Related guides

• Internal Knowledge Assistants: Give Your Team Instant Answers
• How to Build an AI Chatbot on Your Own Company Data
• RAG vs Fine-Tuning: Which Should You Choose?
• Why AI Chatbots Hallucinate and How to Reduce It
• Service: AI Chatbots and Knowledge Assistants

Frequently asked questions

Is RAG the same as training a chatbot on our data?

People often describe it that way, but technically no training happens. The documents are indexed and retrieved when a question arrives, which is why updates take effect as soon as the index is refreshed.

Do we need a vector database?

You need somewhere to store and search embeddings. That can be a dedicated vector database or a vector feature in a database you already run. For a small collection the choice matters less than chunking and evaluation.

How much content do we need to start?

There is no minimum. A focused, accurate set of documents on one topic makes a better first assistant than a large unreviewed archive.

Can RAG work with data in our business systems?

Yes, though records are usually better reached through tools that query the system live. Many assistants combine document retrieval for policies and guides with API lookups for orders, stock or account status.

Sources and further reading

Primary documentation and research behind this guide.

• Lewis et al.: Retrieval-Augmented Generation for knowledge-intensive NLP tasks
• Pinecone: What is a vector database
• pgvector: vector search for PostgreSQL

RAG is simple to describe and takes care to do well. The work is mostly in the documents, the retrieval and the testing. If you are considering an assistant on your own content, our AI chatbots and knowledge assistants page explains how Growmerz builds them, or you can book a free AI audit to talk through your use case.