Key points
- RAG finds the most relevant passages in your documents and gives them to the AI to answer from.
- It produces more accurate, up-to-date answers with citations — without retraining a model.
- The two things that make or break RAG: clean source documents and permission-aware retrieval.
How RAG works
RAG works in two phases: preparing your documents once, then retrieving from them on every question.
# 1. Prepare (once, then keep in sync)
documents → split into passages → create embeddings → search index
# 2. Answer (every question)
question: "How many PTO days do new staff get?"
→ search index # find the 5 most relevant passages
→ check permissions # only passages this user may see
→ model answers # using only those passages
→ "New staff receive 10 PTO days… [Handbook §4.2]"Embeddings are numeric fingerprints of meaning, so a search for "time off" also finds passages about "vacation" and "PTO." The model is instructed to answer only from what was retrieved — and to say so when the answer isn't there.
Why RAG beats "just asking the AI"
- Accurate & current: answers reflect your latest documents, not the model's training data.
- Citations: people can click through and verify the source.
- No retraining: update a document and the answers update with it.
- Privacy-friendly: your documents aren't used to train the model.
RAG vs. fine-tuning
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model can look up | How the model behaves or writes |
| Best for | Facts, policies, knowledge that changes | Consistent style, format, or specialized tasks |
| Updating | Instant — edit the document | Requires retraining |
| Citations | Yes | No |
| Typical choice | Most organizations, most of the time | Specialized cases, often combined with RAG |
Examples
- Staff handbook assistant that answers HR and process questions with links to the exact section.
- Policy Q&A for compliance, safety, or child-protection policies.
- Member or customer FAQ grounded in approved answers, with escalation to a person.
- Sermon or content archive search — "What have we taught about generosity?"
- Support assistant grounded in product manuals and past resolved tickets.
Permissions: the part people forget
A RAG system must respect who is allowed to see what. If the index contains HR files, board minutes, or counseling notes, a careless setup could surface them to anyone who asks the right question.
- Index only what the assistant actually needs.
- Carry document permissions into the search, so users only retrieve what they could already open.
- Exclude highly sensitive categories entirely unless there's a strong reason and strong controls.
- Test with users at different access levels before launch.
Built-in assistants like Microsoft 365 Copilot and Gemini in Google Workspace use your existing file permissions — which means over-shared folders become over-shared answers. Clean up sharing before you switch them on.
Getting started
You may not need a custom build. Many assistants now let you upload files to a project or connect a shared drive. Custom RAG makes sense when you need tighter permission control, many sources, high accuracy with citations, or a customer-facing experience.
Either way, the best investment is in the documents themselves: current, de-duplicated, clearly titled, and with obsolete versions removed. WHU AI's knowledge assistant builds include that cleanup, evaluation, and monitoring.