Open-book does not mean automatically correct
Retrieval-augmented generation, or RAG, means retrieving relevant material and giving it to a model as context for an answer. Instead of relying only on learned patterns, the application supplies a reading list. The OpenAI embeddings guide describes representations used for relatedness and search; the prompting guide explains supplying relevant context. Retrieval and generation are distinct jobs.
A common pipeline prepares documents, splits them into passages, represents them for search, retrieves candidates and asks a model to answer from those candidates. Not every system needs a vector database: exact matching, keyword search or a small curated document set may be enough. That is our architecture recommendation, not a universal recipe.
The two ways the answer goes wrong
First, retrieval can select the wrong material. An old policy might rank above the current one; an important exception might live in another passage. Second, generation can overstate what the retrieved material says. A link in the answer does not prove the sentence is supported. Keep the source title, URL, revision or date and the actual passage so a person can inspect the chain.
Permissions belong inside retrieval, not just in the chat screen. Our recommendation is to filter documents to what the requester is allowed to access before passing them to a model. Do not let an answer reveal a restricted document simply because the index can find it. Treat retrieved text as evidence, never as higher-priority instructions. OWASP explains the indirect-injection risk from external content.
Make “not enough evidence” a valid answer
For a policy assistant, specify what happens when sources conflict or do not cover the question. Ask for a supported answer, relevant caveats and an explicit missing-evidence response. The prompting guide supports clear task instructions and useful context. We recommend evaluating retrieval quality separately from answer quality so you know which stage to fix.
Keep updates and deletions in the design. A new policy needs to replace or outrank the old one, and removing a document should not leave its text available in an index indefinitely. Logging the retrieved passages helps debugging, but logs themselves need appropriate retention and access controls.
Walk through a failure
Exercise: A model correctly quotes last year’s travel policy, but this year’s policy changed the reimbursement limit. Is that a model reasoning problem?
Answer: The first problem is stale or incorrect retrieval. Better prose will not fix the evidence set. Update the source pipeline, preserve dates and then re-evaluate the answer. This is an illustrative scenario, not a benchmark result.
The takeaway: RAG can connect answers to your documents. It still needs current sources, permissions, careful synthesis and a way to admit that the reading list was insufficient.