How to Cut Agent Token Costs with Governed Context

There is a cost difference between a system that looks something up and a system that works it out, and with agents you pay that difference on every single call.

A governed answer is a lookup. The definition of an active customer is resolved from somewhere it has been agreed, and the model spends its tokens on the actual question. An ungoverned answer is a statistical judgment: the model is handed a pile of retrieved material, infers what the term probably means from context, reasons its way to something defensible, and does all of that again on the next request, because nothing it concluded was written anywhere.

Those are two different operations with two different costs, and the second one recurs for as long as the system runs.

The steps

1. Baseline token spend per answered question

Not per call, per answered question. The distinction matters because the expensive pattern is a question that takes four calls, two clarifications and a retry, and per-call metrics hide exactly that.

Instrument three things: total tokens consumed from question to accepted answer, how many round trips it took, and what proportion of the context window was occupied by retrieved material rather than by the conversation. That last number is usually the surprise.

2. Find where the model is re-deriving facts it could look up

Read actual traces. Not a sample of successful ones: a sample of expensive ones.

You are looking for a specific signature: the model reasoning its way to something that has a correct answer someone already knows. Working out from three retrieved documents what your fiscal calendar probably is. Inferring which of two customer records is the parent. Reconstructing an eligibility rule from examples. Each of those is a lookup being performed as a computation, and it is being performed again on the next request and the one after.

Rank what you find by frequency rather than by how egregious it looks. A small derivation on a high-volume path costs more than a dramatic one that happens twice a week.

3. Move those facts into governed context

Take the top of that ranked list and give each item a decided answer: the definition, agreed by someone with the standing to agree it, stored where the system can resolve it at query time.

This is not a retrieval optimization. The point is not that the model finds the fact faster; it is that the model stops deriving the fact at all, because the fact arrives as an input rather than as a conclusion.

Retrieval still runs, and should. The change is that when a question involves a governed term, the term resolves against the approved definition first, and retrieval works within that rather than around it.

4. Measure again, on the same questions

Re-run the baseline set. Expect the improvement to show up as shorter reasoning chains and fewer round trips rather than as a smaller prompt: the prompt may be similar in size, but it now contains answers instead of raw material to reason over.

Watch accuracy on the same run. A token reduction that comes with a quality regression is not a saving, it is a trade you did not agree to make. The governed path should improve both, and if it improves cost while hurting accuracy, the governed definition is probably wrong rather than the approach.

5. Reinvest the savings in harder reasoning

The useful outcome is not a smaller bill. It is that the same budget now buys reasoning on questions that are worth reasoning about.

Tokens spent re-deriving your fiscal calendar are pure waste. Tokens spent working through a genuinely multi-step question are the product. Moving spend from the first category to the second is the actual objective, and teams that treat cost reduction as the goal tend to hit their number by capping context and quietly losing answer quality.

The argument for the person who controls the budget

Most of this article is for the person doing the work. This section is for what they do next, which is hand the argument to someone who controls the spend and was not in any of the technical conversations. It is written to survive being pasted into an email with no surrounding explanation.

A governed answer is a lookup. An ungoverned answer is a statistical judgment. Those are two different operations, they have two different costs, and the difference is paid on every single call, not once at build time, and not at some future scale threshold.

That is what makes this a financial question rather than an engineering preference. An architecture that re-derives the same facts on every request has a recurring line item attached to it, and the line item grows with adoption. The better the system works, the more it gets used, and the more the waste compounds. Success makes it worse.

The decision in front of you is not whether to spend on AI. It is whether the same spend buys answers that are defensible, or answers that are plausible. A system that looks up what your company has agreed produces the first. A system that infers it each time produces the second, at a higher unit cost, with no record of what it assumed.

One honest caveat, because it affects the sizing. Not everything needs this treatment. Most of what an AI system handles can be inferred perfectly well and governing it would be wasted effort. The spend that matters is concentrated in the small set of terms and rules the business actually argues about, which is also where wrong answers are most expensive. That concentration is what makes this tractable rather than boundless.

Best practices

Measure cost per correct answer, not per call. A cheap wrong answer is not cheap. It costs the call, the retry, the human time to catch it, and (in the cases that matter) the decision that was made on it before anyone noticed. Per-call pricing is the vendor's unit of account, not yours.

Baseline before you change anything. This sounds obvious and is routinely skipped, and without it you cannot distinguish a real improvement from a quieter model release.

Govern narrowly and deliberately. The instinct after the first win is to govern everything. Resist it. The rule is explicit where it counts and inferred everywhere else, and the set that counts is small.

Watchouts

Do not treat a measured saving as a general one. A reduction measured on one workload tells you about that workload. Different question mixes, different corpora and different models produce different numbers, and quoting a figure without its conditions is how a defensible result becomes an indefensible claim.

This ground is contested, and the argument is worth knowing. Another vendor in this space has published a well-argued case that GenAI cost problems are not model problems, and the diagnosis overlaps with the one here. The specificity is where the two diverge: the claim in this article is not merely that governance reduces cost, but that what makes an answer a lookup is an agreed definition with a named owner and a review trail. That is a narrower and more testable statement than a general appeal to governance, and it is the one worth defending.

Cheap and wrong is the failure mode to design against. Every lever in this article can be pulled too hard. Cap the context aggressively enough and costs fall beautifully while accuracy quietly degrades, and the metric that would have caught it is the one nobody instrumented.

A price cut is not an architecture. Model costs fall regularly, and each fall makes the waste less visible without making it smaller. The re-derivation is still happening; it is just cheaper per unit, and volume is rising faster than unit price is dropping.

For the discipline underneath this, see context management, and for why an inferred definition cannot do the job no matter how good the inference is, see inferred context. If you are building rather than optimizing, grounding an LLM in enterprise data is the construction method.