RAG in plain English: how AI can answer questions from your own documents

A customer rings with a question about one of your products. Will it work outdoors in winter? Can it be fitted to an older model? The answer is almost certainly written down somewhere: a product sheet, a manual, an email from the supplier two years ago. Finding it means asking the one colleague who remembers, or ten minutes of searching.

You may have heard that AI can now answer questions like this from your own documents. That's true, and the idea behind it has a slightly awkward name: retrieval-augmented generation, or RAG. It sounds technical, but the principle is simple, and once you understand it you'll be much better placed to judge the tools on offer and decide whether it's worth trying in your business.

This article explains how it works in plain English, where it genuinely helps, what can go wrong, and how to start.

The problem RAG solves

AI assistants such as ChatGPT, Copilot, Gemini and Claude learned from an enormous amount of public text. That makes them good at writing, summarising and general knowledge. It doesn't mean they know anything about your business. They have never seen your price list, your returns policy or the spec sheet for the product your customer is asking about.

Ask one a question about your own products and it has two options: say it doesn't know, or make a plausible guess. The guess is the problem. It can sound entirely confident while being wrong.

RAG is the most common way round this. Rather than hoping the AI already knows the answer, you give it the relevant information at the moment you ask.

The two steps: find, then answer

RAG works in two steps, and the name describes them.

1. Retrieval (finding). When someone asks a question, the system first searches your documents for the passages most likely to contain the answer. This is a search, much like the search in your file system, though modern versions are better at matching meaning rather than exact words. A question about "holiday entitlement" can find a passage that talks about "annual leave".
2. Generation (answering). The system then hands those passages to the AI along with the question, and asks it to answer using that information. The AI does what it's good at: reading, combining and explaining in clear language.

A helpful way to picture it is an open-book exam. The AI is handed the right pages and asked to answer from them, so it doesn't need to have memorised anything about your business.

Microsoft's documentation describes RAG as a pattern that grounds an AI's responses in your own content, and Google Cloud's explainer covers the same idea. "Grounding" is a word you'll see often in this area. It simply means tying the answer to real source material.

Why answers can show their sources

Because the system knows which passages it retrieved, it can tell you where the answer came from. Good tools show this as a link or a reference to the document, so you can click through and check.

This matters more than it might seem. It turns the AI from a confident voice you have to trust into a quick way of finding the right page, with a summary on top. If the answer looks surprising, you can open the source in seconds. For anything you're going to tell a customer, that check is a good habit.

Everyday examples

RAG is most useful where your business has a body of written knowledge that people regularly need to dip into. For example:

• The staff handbook: "How much notice do I need to give for a day's holiday?" answered from your actual policy, with a link to the section
• Product information: a sales or customer service team asking about specifications, compatibility or care instructions across dozens of product sheets
• Past proposals and quotes: "Have we done a project like this before, and how did we describe it?" when preparing a new proposal
• Procedures: a new starter asking how a particular task is done, answered from your process documents rather than by interrupting a colleague

In each case the information already exists. RAG makes it quicker to reach, and helps more people find it without relying on the person who knows.

What can go wrong

RAG is a genuinely useful idea, and it has limits worth knowing before you start.

Poor documents in, poor answers out. The AI can only answer from what the search finds. If there are four versions of the price list, or the handbook is out of date, the answer will reflect that. This is why the ordinary housekeeping in our article on getting your business information ready for AI pays off so directly here.

Permissions matter. A well-built system only searches documents the person asking is allowed to see. Microsoft, for example, states that Microsoft 365 Copilot's access to work data is scoped by user permissions. That's reassuring, but it also means any file that's shared too widely today will be easier to find tomorrow. A quick review of who can see what is a sensible step before switching anything on.

The search can miss. If the relevant passage isn't retrieved, the AI may answer from the wrong material or give a vague reply. Questions with clear answers in well-written documents work best. Questions that need judgement across lots of sources are harder.

It can still make mistakes. Grounding reduces guesswork a great deal, but it doesn't remove it entirely. Checking the source for anything important is still the right approach.

Built-in tools or a custom build

You may already have access to RAG without calling it that.

• Microsoft 365: with the paid Microsoft 365 Copilot licence, Copilot grounds its answers in your organisation's emails, files and meetings through Microsoft Graph, within each user's permissions. The free Copilot Chat can work with files you upload or have open, but doesn't search your organisation's content automatically.
• Google Workspace: on eligible plans, Gemini in Google Drive can answer questions about your files and shows inline citations you can click to view the source.
• Other assistants: if you use a standalone assistant, check its business plan to see whether it can work with your own files, and read the provider's business terms on how your information is handled.

We supply Microsoft licences and know these tools well, but the right choice depends on what you already use and how your information is organised.

Built-in tools are usually the best place to start. They work with files where they already live and respect the permissions you've set.

A custom build makes sense when the knowledge sits outside those tools (in a database, a website or a line-of-business system), when you want a focused tool for one job, such as a product question helper for your customer service team, or when customers rather than staff will use it. The major cloud providers offer the building blocks, and the work is mainly in preparing the content, controlling access and testing that the answers are good.

A sensible first step

Pick one well-defined set of documents that people ask about often. The staff handbook or your main product sheets are good candidates.

1. Tidy that one area. Remove duplicates and out-of-date versions, and check who can see it.
2. Write ten real questions your team or customers actually ask, along with the correct answers.
3. Try them in the AI tool you already have access to, and see how many it answers correctly with a sensible source.

That small test will tell you more than any demonstration, and it costs very little. If most answers are good, you have a clear case for wider use. If they aren't, you'll know whether the fix is in the documents or the tool.

If you'd like help choosing an approach or running a short trial, our AI Adoption service is a good starting point, and for a custom question-answering tool, take a look at Software & Integration.