You upload a PDF, ask a question, and an answer comes back in seconds with a neat little citation beside it. It feels like the AI sat down and read your document the way you would. It did not. Something quite different happened, and once you know what, every strange answer you have had from a document chatbot starts to make sense.

The name for it is RAG. It is one of the most useful ideas in everyday AI and one of the worst explained, because nearly every guide is written for software engineers. This one is not. Here is what RAG in AI actually is, where you already use it, and why a perfectly good document can still give you a poor answer.

What is RAG in AI, in plain English

RAG stands for retrieval augmented generation. Three long words, one simple sequence: before the AI writes a word of its answer, something fetches relevant material for it, and the AI writes from that material.

AWS describes it in its own guide to the term as optimising a model’s output so that it references an authoritative knowledge base outside its training data before it answers. Strip the jargon: look it up first, then answer.

Picture an open book exam where you are not allowed to touch the book. An assistant reads your question, runs off, photocopies three pages he thinks are relevant, and slides them across the desk. You write a strong answer from those pages. That is RAG. Notice the detail that decides everything: the assistant chose the pages, and if he brought the wrong three, you never find out.

Why AI needed this in the first place

A large language model learns from an enormous pile of text and then stops learning. It has never seen your tenancy agreement, your company handbook or your lecture notes, and it cannot be sent back to school every time one of those files changes.

Retrieval was the way around that, and the name comes from a 2020 research paper by Patrick Lewis and colleagues, presented at NeurIPS that year, which paired a language model with a searchable index of Wikipedia. On language generation tasks they reported it produced more specific and more factual language than the model working alone.

How RAG actually works, step by step

Here is the shape of it.

1. Your documents are chopped into small pieces, rarely more than a couple of paragraphs each. These are called chunks.
2. Each chunk is converted into a long list of numbers representing its meaning, and those numbers go into a searchable index.
3. You ask your question, and it gets converted into numbers the same way.
4. The index is searched for the handful of chunks closest in meaning. Good systems look for exact word matches at the same time, which matters more than you would think.
5. Those few chunks are pasted invisibly into the prompt, alongside your question.
6. The model reads that bundle and writes the answer.

Read steps four and six together, because this is the part most people never hear: the model does not search your documents. A separate search step does, and the model only ever sees what that step handed it. Your two hundred page report was never read. Six paragraphs of it were.

Where you are already using RAG

You almost certainly used retrieval this week without hearing the word once.

• Every time you upload a PDF and ask questions about it.
• Google’s research assistant, called Gemini Notebook in Google’s own help pages and still found at notebooklm.google. Google describes it as giving grounded information based on your sources with clear in line citations, and says it is designed to answer from the sources you uploaded. Our walkthrough covers the practical side.
• The support chatbot on a company website that somehow knows that company’s refund policy.
• AI search answers with links underneath. Those links are the retrieval step, made visible.

Building websites and small online tools is where this stops being abstract. Anyone who has set up a help chatbot learns the same lesson quickly: the bot is only ever as current as the folder behind it. Leave a superseded price list sitting in there and it will be quoted back at a customer with total confidence. The AI is not wrong. The shelf it is reading from is.

Why RAG answers can still be wrong

Retrieval reduces invented answers, which is why companies use it, but it does not remove them. The process can fail in two places: the search fetches the wrong material, or the model misreads the right material. Most people only think about the second. Anthropic published test results in which a standard retrieval setup left 5.7 percent of the relevant material out of its top twenty chunks. Their improved method brought that share down to 2.9 percent, and adding a reranking step to 1.9 percent. Those are one company’s own tests rather than a law of nature, but the direction matters: even careful retrieval misses things, and when it does, the model answers from the wrong pages with no sign anything went astray.

Chunking causes its own trouble. Anthropic gives a neat example: a chunk reading “the company’s revenue grew by 3% over the previous quarter” is hard to use on its own, because it never says which company or which quarter. The details that would anchor it were somewhere else in the document and did not travel with the chunk. This is how a document chatbot hands you a number that is real but attached to the wrong thing, which is a different failure from a plain hallucination.

Your wording decides what gets found

This is the practical part, and it sits in a troubleshooting section almost nobody opens. Google’s help pages, explaining why its notebook tool sometimes cannot answer, say that when a notebook holds many sources the tool retrieves the most relevant information based on your question first, then builds a response from what it found. Their advice is to rephrase the question to be clearer or more specific.

Sit with that for a second. Your phrasing is doing double duty as the search query. Being specific helps, and so does using the words the document itself uses. Anthropic’s engineering write up explains why: matching on meaning alone can miss an exact term, which is why serious systems run a word matching search alongside the meaning based one. Ask “what did it say about costs” of a document that only ever says “expenditure”, and you may be handed weaker chunks than the document deserves.

Important tip: if a document AI says it cannot find something, do not assume the information is missing. Ask again, more specifically, using the words the document itself uses. That one habit fixes more bad answers than any clever prompt template.

What your file loses on the way in

A second quiet failure catches people out with reports and research papers. OpenAI’s file uploads FAQ notes that its Enterprise product supports visual retrieval for PDFs, and that on all other plans document files support text based retrieval only, so ChatGPT extracts the digital text and discards the images.

So if the figure you are asking about lives inside a chart, a diagram or a photographed table, it may not be in the pile the AI is searching at all. The same goes for a scanned document with no text layer. The AI is not refusing to tell you. It never received that part of your file. Quick check: open the PDF and try to select a sentence with your mouse. If the text highlights, there is something to retrieve. If nothing highlights, you have a picture of a page, and it needs an optical character recognition step first.

Four habits that get better answers from any document AI

• Use the document’s own vocabulary, not your paraphrase of it.
• Ask one thing at a time. A three part question pulls the search in three directions and satisfies none of them well.
• Ask where the answer came from, then click the citation. Skipping it throws away the main safety feature you were given.
• Move outdated files out of the folder rather than renaming them. Old documents do not look old to a retrieval system, and anything left in there is fair game.

Common Questions

Is RAG the same as the AI searching the internet?

Same process, different shelf. Web search retrieves from the open internet. RAG usually retrieves from a specific set of documents, such as your uploads or a company’s files. Both fetch first and write second.

Does RAG stop AI from making things up?

It helps a great deal, because the model has real material in front of it and can cite where each claim came from. It does not eliminate the problem. If the search fetches the wrong passages, the model writes a confident answer from them.

Is RAG the same as training or fine tuning a model?

No. Training changes the model itself and is expensive. RAG leaves the model untouched and hands it material at the moment you ask, which is why a company can add a new policy document to its chatbot in an afternoon instead of waiting for the next retraining cycle.

Final takeaway

RAG in AI is a search step wearing a very long name. Fetch first, write second. Once you picture that assistant running off to photocopy a few pages, you stop treating a document chatbot as an oracle that has read everything, and start treating it as a fast reader handed a small stack.

That shift changes how you use these tools. You word questions like search queries, keep your folders clean, and click the citation instead of trusting it. None of that requires technical skill, and it will get you better answers from the AI you already use.