What Governance Does a RAG or LLM Pipeline Need?

What governance does a RAG or LLM pipeline need? Governance for RAG and LLM pipelines usually covers chunking, embeddings, access filters and evaluation. All necessary. None of it addresses the thing that most often makes a retrieved answer wrong, which is the definition hiding inside a retrieved passage.

Documents carry definitions nobody registered, and retrieval pulls them in anyway.

What does a RAG pipeline actually retrieve?

Passages that score close to the question. Similarity is the whole selection rule. The index has no view on whether a passage is current, authoritative or in scope.

That is a reasonable design for open questions. It becomes a problem when the passage contains a definition.

Where do unregistered definitions come from?

Ordinary documents, written by people doing their jobs. Take the word customer inside one company.

Every one of those documents is correct for its own purpose. None of them says which purpose that is. The definition is embedded in prose, with no scope attached.

Why does retrieval pull them in anyway?

Because a passage defining customer scores extremely well against a question asking what a customer is. The superseded 2019 deck is a strong match on every term in the query.

Recency filters help a little and miss the partner handbook, which is current and out of scope. Access filters help a little and miss everything the user is entitled to read.

What breaks at generation time?

The model receives four passages holding four definitions. It writes one fluent answer. The blend is invisible in the output.

Sometimes it picks one. Sometimes it merges them into a definition that appears in no document and no system. That second case is the dangerous one, because the sentence reads like policy.

What controls actually help?

1. Register the definitions that matter. Hold them in a system, with scope attached, so prose is never the source.
2. Mark document scope at ingestion. Which business context a source belongs to, recorded as metadata.
3. Prefer registered definitions over retrieved ones. When a question asks what a term means, answer from the register and cite the document for colour.
4. Retire superseded sources from the index. The 2019 deck has an archive, and the archive can sit outside retrieval.
5. Require the answer to name its scope. The response says which definition it used and where that definition applies.

The third control does most of the work. It turns the pipeline into something that looks a definition up.

How do document definitions reconcile with system ones?

• Scope. The contracting entity definition governs legal and revenue. The login definition governs support. The reseller definition governs partner reporting.
• Relationship. One contracting entity holds many logins. Partner accounts sit beside both.
• Precedence. Where a document and the register disagree, the register wins and the document gets flagged for update.

That precedence rule is cheap to state and it ends a whole class of argument. Data governance for AI covers the underlying case. For the surrounding architecture, see five strategies for semantic data architecture, and for how the register is built, ontology management.

So what should you take away?

Your document corpus is full of definitions written for contexts it no longer records. Retrieval treats them all as equally true.

Register the definitions for your most contested term. Mark scope at ingestion. Make the register win.

TopQuadrant builds the systems that hold definitions, scope and ownership together. Check what your pipeline returns today for your own hardest term.

Related in this cluster: How Do You Govern Data for AI Agents? and Enterprise Metadata Management: Unlocking Business Value from Your Data.