AI & context
7 min

How to Measure Whether Your AI Answers from Governed Data

Key takeaways

  • A correct answer tells you nothing about where it came from. Measure provenance, not only correctness.
  • Four signals: resolution rate, local-copy count, version pinning, and unresolved terms.
  • Unresolved terms are the most useful signal and the most often discarded. They are a demand-driven backlog.
  • Measure resolution against the governed set, not the full universe of terms the system encounters.
In this article
Share

How do you measure whether your AI is answering from governed data?

You cannot measure it by sampling answers for correctness. A correct answer tells you nothing about where it came from, and an AI system that guesses well will pass that test until the day it does not.

What you can measure is provenance: for a given answer, which definitions and which facts did the system resolve, and were those the agreed ones. That is a different question from accuracy and it is the one that predicts whether accuracy holds.

This matters more than it sounds. In our own work with enterprise customers, roughly 80 to 85% of the accuracy gain comes from getting ground truth right, ahead of anything done to the model or the retrieval layer. If that is where the gain lives, it is also where the measurement should point.

What should you actually instrument?

Four signals, and only the first requires any new plumbing.

SignalWhat it tells youHow to get it
Resolution rateWhat share of governed terms in a query were resolved against the authorityLog every call the system makes to the context layer
Local-copy countHow many consuming systems keep their own definitionsCount the systems that never call
Version pinningWhether an answer can name which agreed version it usedReturn a version with every resolution
Unresolved termsTerms the system needed and could not resolveLog the misses, which is where the next governance work is

The fourth is the most useful and the most often discarded. A log of what the system asked for and could not find is a prioritised list of what to govern next, generated by actual demand instead of by a modelling exercise.

Why is sampling answers the wrong test?

Because it measures the output of a process whose inputs you are not watching.

A system can produce a correct answer from an ungoverned source, a stale definition or a lucky retrieval. It passes. Three months later the underlying document changes, the same question produces a different answer, and nothing in the sampling regime predicted it.

Worse, sampling creates a false sense of safety that scales badly. A 95% pass rate on a hundred sampled answers tells you very little about the ten thousand you did not sample. The failures cluster on exactly the contested terms a sample is unlikely to hit.

What sampling still earns its place for

  • Finding unknown failure modes. Reading real answers surfaces problems no metric was designed to catch.
  • Calibrating the provenance signals. If resolution rate is high and answers are still wrong, the governed definitions themselves are the problem.
  • Regression checks after a change. A fixed question set run before and after a model or prompt change.

Sampling and provenance measure different things, and a programme that runs only one of them has a blind spot it cannot see from the inside.

How do you establish a baseline?

Pick ten questions the business already argues about, and answer them three ways.

Ask the AI system. Ask the governed source, if one exists. Ask two people in different teams. Record all three. The gap between the second and third answers tells you whether the governance is real. The gap between the first and second tells you whether the system is using it.

Most organizations running this for the first time find that the three answers disagree, and that the system is confident in all cases. That is the baseline, and it is more useful than any accuracy percentage, because it names the specific terms where the work needs to happen.

What should the numbers be?

Resolution rate is the one worth a target, and the honest answer is that it depends on how much has been governed.

Measuring resolution against the full universe of terms a system encounters produces a demoralising number and a useless one, because most of those terms were never contested. Measure against the governed set. Of the terms that have an agreed definition, what share did the system resolve instead of inferring? That number should approach total, and anything well below it means consuming systems are holding local copies.

Track unresolved terms as a separate count that is allowed to grow, because growth there means the system is encountering more of the business, which is progress. Keeping context from going stale covers the freshness side, and data governance for AI the wider programme.

What does week one look like?

Smaller than a measurement programme. One system, one log line, ten questions.

A first week that produces something

  • Add one log line. Every time a consuming system resolves a term against the authority, record the term, the version and the timestamp.
  • Count the systems that never appear in that log. Each one is holding its own definitions.
  • Run the ten-question baseline. AI answer, governed answer, two humans. Record the disagreements verbatim.
  • List the terms the system needed and could not resolve. That is your governance backlog, ordered by real demand.

None of this needs a platform, a dashboard or a budget line. It needs somebody to add a log line and read the output for a week, and it will tell you more about whether your AI is grounded than any accuracy benchmark.

The common objection is that the log is incomplete because only some systems call the authority. That is the finding, not an obstacle to it.

Who actually needs this measurement?

Three audiences, and they want different cuts of it.

What each audience asks for

  • The team running the AI system wants unresolved terms, because that is their backlog.
  • The governance owner wants resolution rate and local-copy count, because those say whether the vocabulary is being used.
  • Risk and audit want version pinning, because their question is always about a past answer and what rule applied at the time.

The third is the one that forces the discipline. If a regulator asks why a system answered as it did in March, you need the definition that applied in March, attached to that answer. Nothing reconstructs it afterwards.

So what should I take away?

Measure provenance as well as correctness. Log every resolution, count the systems keeping local copies, pin a version to every answer, and treat unresolved terms as the backlog instead of a failure. The instrumentation sits on top of the work described in how to use a knowledge graph for AI context, and the commercial counterpart is knowledge graph ROI and what to measure.

TopQuadrant builds for this. Our knowledge graph platform resolves governed terms at query time and returns the version and the approval with them, so an answer can say which agreed definition it used. The 80 to 85% figure above is our own field measurement. It is no kind of industry statistic. Treat it as a reason to instrument your own and find out.

Give your AI the context it's been missing

See how the TQ Data Foundation turns your enterprise knowledge into trusted, Al-ready context.