How to Cut Agent Token Costs with Governed Context
Key takeaways
- Measure cost per correct answer, not cost per call.
- A governed answer is a lookup. An ungoverned one is re-derived from scratch, and you pay for the derivation every time.
- The difference is paid on every call, not once at build time.
- Read expensive traces before tuning anything.
There is a cost difference between a system that looks something up and a system that works it out, and with agents you pay that difference on every single call.
A governed answer is a lookup. The definition of an active customer is resolved from somewhere it has been agreed, and the model spends its tokens on the actual question. An ungoverned answer is a statistical judgment. The model is handed a pile of retrieved material, infers what the term probably means from context, and reasons its way to something defensible. Then it does all of that again on the next request, because nothing it concluded was written anywhere. What it produced was runtime context, and inferred context is exactly the kind that has to be rebuilt every time.
Those are two different operations with two different costs, and the second one recurs for as long as the system runs.
The distinction that matters is narrower than "governance saves money," which is a claim anyone can make. What turns an answer into a lookup is specific: an agreed definition, a named owner, and a record of when it last changed. Absent any one of those three, the model is still reasoning its way to a conclusion and you are still paying for it. That is the testable version of the claim, and it is the one worth defending.
How do you actually cut the cost?
Five steps, in order. Each one is cheap on its own; the sequence is what makes the saving stick.
1. Baseline token spend per answered question
Not per call, per answered question. The distinction matters because the expensive pattern is a question that takes four calls, two clarifications and a retry, and per-call metrics hide exactly that.
Instrument three things: total tokens consumed from question to accepted answer, how many round trips it took, and what proportion of the context window was occupied by retrieved material. That last number is usually the surprise.
2. Find where the model is re-deriving facts it could look up
Read actual traces. Not a sample of successful ones: a sample of expensive ones.
You are looking for a specific signature: the model reasoning its way to something that has a correct answer someone already knows. Working out from three retrieved documents what your fiscal calendar probably is. Inferring which of two customer records is the parent. Reconstructing an eligibility rule from examples. Each of those is a lookup being performed as a computation, and it is being performed again on the next request and the one after.
| What the model is doing | What it looks like in a trace | What it should be |
|---|---|---|
| Working out your fiscal calendar | Reasoning across three retrieved documents to infer quarter boundaries | A lookup against a settled definition |
| Deciding which customer record is the parent | Comparing names and addresses, then picking | A resolved entity, decided once |
| Reconstructing an eligibility rule | Generalizing from examples in the retrieved set | A rule resolved at query time |
| Re-reading policy to find a threshold | Long retrieved passages, short answer | A value, returned directly |
Rank what you find by frequency. A small derivation on a high-volume path costs more than a dramatic one that happens twice a week.
3. Move those facts into governed context
Take the top of that ranked list and give each item a decided answer. That means the definition, agreed by someone with the standing to agree it, stored where the system can resolve it at query time. That is context management applied to a cost problem.
This is not a retrieval optimization. The model does more than find the fact faster. It stops deriving the fact at all, because the fact arrives as an input.
Retrieval still runs, and should. The change is that when a question involves a governed term, the term resolves against the approved definition first, and retrieval works within it. Grounding an LLM in enterprise data is the construction method for that wiring.
4. Measure again, on the same questions
Re-run the baseline set. Expect the improvement to show up as shorter reasoning chains and fewer round trips. The prompt may be similar in size, but it now contains answers instead of raw material to reason over.
Watch accuracy on the same run. A token reduction that comes with a quality regression is a trade you did not agree to make. The governed path should improve both, and if it improves cost while hurting accuracy, the governed definition is probably wrong.
5. Reinvest the savings in harder reasoning
The useful outcome is not a smaller bill. It is that the same budget now buys reasoning on questions that are worth reasoning about.
Tokens spent re-deriving your fiscal calendar are pure waste. Tokens spent working through a genuinely multi-step question are the product. Moving spend from the first category to the second is the actual objective. Teams that treat cost reduction as the goal tend to hit their number by capping context and quietly losing answer quality.
Why should the budget owner care?
Most of this article is for the person doing the work. This section is for what they do next, which is hand the argument to someone who controls the spend and was not in any of the technical conversations. It is written to survive being pasted into an email with no surrounding explanation.
A governed answer is a lookup. An ungoverned answer is a statistical judgment. Those are two different operations with two different costs. The difference is paid on every single call, today and at every volume.
That is what makes this a financial question. An architecture that re-derives the same facts on every request has a recurring line item attached to it, and the line item grows with adoption. The better the system works, the more it gets used, and the more the waste compounds. Success makes it worse.
The decision in front of you is not whether to spend on AI. It is whether the same spend buys answers that are defensible, or answers that are plausible. A system that looks up what your company has agreed produces the first. A system that infers it each time produces the second, at a higher unit cost, with no record of what it assumed.
One honest caveat, because it affects the sizing. Not everything needs this treatment. Most of what an AI system handles can be inferred perfectly well and governing it would be wasted effort. The spend that matters is concentrated in the small set of terms and rules the business actually argues about, which is also where wrong answers are most expensive. That concentration is what makes this tractable.
What works in practice?
Measure cost per correct answer. A cheap wrong answer is not cheap. It costs the call, the retry, the human time to catch it, and (in the cases that matter) the decision that was made on it before anyone noticed. Per-call pricing is the vendor's unit of account, and yours is different.
Baseline before you change anything. This sounds obvious and is routinely skipped, and without it you cannot distinguish a real improvement from a quieter model release.
Govern narrowly and deliberately. The instinct after the first win is to govern everything. Resist it. The rule is explicit where it counts and inferred everywhere else, and the set that counts is small.
What goes wrong?
Do not treat a measured saving as a general one. A reduction measured on one workload tells you about that workload. Different question mixes, different corpora and different models produce different numbers, and quoting a figure without its conditions is how a defensible result becomes an indefensible claim.
This ground is contested, and the argument is worth knowing. Another vendor in this space has published a well-argued case that GenAI cost problems are not model problems, and the diagnosis overlaps with the one here. The specificity is where the two diverge. This article makes a narrower claim: what turns an answer into a lookup is an agreed definition with a named owner and a review trail. That is a narrower and more testable statement than a general appeal to governance, and it is the one worth defending.
Cheap and wrong is the failure mode to design against. Every lever in this article can be pulled too hard. Cap the context aggressively enough and costs fall beautifully while accuracy quietly degrades, and the metric that would have caught it is the one nobody instrumented.
A price cut is not an architecture. Model costs fall regularly, and each fall makes the waste less visible without making it smaller. The re-derivation is still happening; it is just cheaper per unit, and volume is rising faster than unit price is dropping.
TopQuadrant’s argument is that governance is where the saving comes from. Compression is a side effect. A definition agreed once and reused is the only context you never pay to re-derive.
