Article reader Listen + reading controls
Article reader
Preparing the reader…
Reading settings
Retrieval is a knowledge-governance problem¶
International Business Machines (IBM) Research has published a clear explanation of retrieval-augmented generation, an approach that gives a large language model access to external sources at the time of a request. Instead of relying only on patterns encoded during training, the system retrieves relevant material and uses it to generate a more current, domain-specific, and potentially verifiable answer.
Retrieval-augmented generation (RAG) is a promising architecture. It is not a substitute for governing the knowledge being retrieved.
Grounding begins before the query¶
A RAG system typically searches an indexed collection, selects passages, places them in the model's context, and generates an answer. Each stage can fail.
The collection may contain obsolete or duplicated documents. The index may omit important metadata. Retrieval may select material that shares vocabulary but not meaning. The language model may ignore a relevant passage, blend conflicting sources, or make a claim the sources do not support.
The quality of the final answer is therefore bounded by the quality and governance of the source environment.
Authority has to be represented¶
Organizations rarely have one clean body of truth. A policy repository may contain a current directive, an outdated copy, a local procedure, training slides, and an employee's helpful notes. A conventional search result lets the user inspect those differences. A generated answer may smooth them into one voice.
The knowledge base needs metadata that helps the system and user reason about authority:
- document owner and approving authority;
- effective and expiration dates;
- version and superseded relationship;
- scope, audience, and jurisdiction;
- sensitivity and access rules; and
- relationship to source systems of record.
This work is not glamorous, but it is what turns a document pile into a governed knowledge source.
Retrieval should preserve disagreement¶
A good system should not force a single answer when authoritative sources conflict. It should expose the conflict, identify the sources, and route the question to a person when resolution requires judgment.
That is particularly important in defense, legal, safety, and policy work, where a local procedure may intentionally differ from enterprise guidance or where a newer document has not yet propagated through training.
Nonaka's theory of organizational knowledge creation also cautions against equating documents with knowledge. Explicit sources matter, but experienced practitioners hold tacit knowledge about exceptions, implementation, and why a rule exists. A RAG system can make written knowledge accessible; it should also make it easy for users to flag where practice and documentation diverge.
Evaluate retrieval separately from generation¶
When an answer is wrong, teams need to know whether the failure came from the source, retrieval, or generation. Evaluate the layers independently:
- Collection quality: Is the right source present and current?
- Retrieval quality: Does the query return the needed passage?
- Synthesis quality: Does the answer accurately represent the passage?
- Attribution quality: Can the user see which source supports each claim?
- Workflow quality: Does the answer help the user act appropriately?
Aggregate answer accuracy hides these distinctions and sends remediation to the wrong team.
Give the knowledge base an owner¶
Every production RAG system needs named stewardship. Data and platform teams can operate the pipeline. Domain owners must decide what is authoritative and how conflicts are resolved. Security teams must ensure retrieval respects the user's permissions. Product teams must design citation, uncertainty, and correction into the interface.
Corrections should update more than the prompt. If users repeatedly identify an obsolete source or missing procedure, the knowledge-management process itself needs repair.
RAG offers an “open-book” form of generative artificial intelligence (AI). The metaphor is useful if we remember that the book must be selected, maintained, and interpreted. Retrieval can ground an answer in organizational knowledge only when the organization has done the work to know which knowledge deserves ground beneath it.
Sources and research trail¶
- IBM Research, “What Is Retrieval-Augmented Generation?” (August 22, 2023).
- Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020).
- Nonaka, “A Dynamic Theory of Organizational Knowledge Creation” (1994).
- Walsh and Ungson, “Organizational Memory” (1991).
- National Institute of Standards and Technology, Zero Trust Architecture (2020).