Large language models are remarkably good at reading something unfamiliar and explaining it. They're also capable of stating, fluently and confidently, things that aren't true. In a chat window that's an inconvenience. In an enterprise system that feeds a funding decision, a compliance report or a modernization plan, it's a design flaw.
The fix isn't a better prompt. It's an architecture that gives the model a narrower job.
Three different kinds of work
Most enterprise AI use cases mix three activities that should be kept apart:
- Extraction — establishing what is objectively there. Which packages does this project reference? Which clauses does this contract contain? Which fields are present on this invoice?
- Interpretation — reasoning over what was extracted. What is this service for? Is this clause unusual? Which migration path fits this application?
- Acceptance — deciding whether an interpretation is good enough to act on, and what happens if it isn't.
When one model call does all three, you can't tell which part failed. When they're separated, each can use the right tool and be checked in the right way.
Extraction should be deterministic wherever it can be
If a parser, an analyzer, a query or a schema validator can answer the question, use it. Deterministic extraction is repeatable, cheap at scale, and explainable: the answer comes with a file, a line number, a cell reference.
Some extraction genuinely needs a model — reading a scanned document, or pulling structure from free text. That's fine, but the output should then be treated as model-extracted, carry a confidence signal, and be validated against whatever deterministic checks exist (totals that must sum, IDs that must exist, dates that must be in range).
The rule I use: anything that can be counted, looked up or parsed is not the model's job. Every time a model is asked to count dependencies or list API calls, someone will later have to write a rule to catch it getting the count wrong.
Interpretation is where models earn their place
Models add real value when the question requires judgment over evidence: summarizing intent, comparing options, explaining risk in plain language, spotting a pattern a rule didn't anticipate.
Three practices make that interpretation usable:
- Ask specific questions. "Which modernization path fits, and what blocks it?" produces comparable answers. "Analyze this repository" produces essays.
- Bound the context. Give the model the relevant facts and excerpts, chosen by the system. Stuffing in everything raises cost, dilutes attention and makes it harder to know what the answer was based on.
- Require citations in a schema. Every claim in the output should reference the fact or excerpt it relies on. That's what makes validation possible.
Acceptance is a policy decision, not a model decision
Asking a model whether its own answer is correct is weak validation. Independent checks are stronger:
- Does every cited fact exist, and does it say what the answer claims?
- Do named entities — packages, clauses, accounts — appear in the extracted evidence?
- Do numbers match exactly?
- Is the output complete against the schema?
Then a versioned policy decides what happens: accept, accept with a caveat, retry with the validation error, or route to a person. Policy lives in code, gets reviewed, and changes deliberately. When the business decides to be stricter, it changes the policy — not the model or the prompt.
Uncertain output and human review
Some outputs will be uncertain no matter how good the system is. A few examples of what that looks like in practice:
- The model's classification is supported by the facts, but the facts are thin — a repository with almost no code, or a contract with a missing schedule.
- The interpretation conflicts with an earlier accepted result for the same input.
- The answer is well-supported but the consequence is high: recommending that an application be re-architected or retired.
Those should go to a reviewer who sees the question, the evidence, the model's answer and the validation result side by side, and whose decision is recorded with the same provenance as an automated one. Two details matter more than they seem. First, sample accepted results for review too, or the system never learns where policy is too loose. Second, track reviewer agreement — if reviewers approve everything, review has become ceremony.
Re-extract, re-interpret, or just re-evaluate?
Separating the three stages pays off most when something changes. There are three different triggers, and they cost very different amounts:
| What changed | What to rerun | Relative cost |
|---|---|---|
| Source changed (new commit, new document version) or an extractor was improved | Re-extract; invalidate interpretations that depended on changed facts | Moderate — mostly compute, model calls only for affected items |
| Question, prompt or model changed | Re-interpret over saved facts | Highest — model calls across the affected set |
| Only acceptance policy changed | Re-evaluate saved interpretations | Near zero — no extraction, no model calls |
Without this separation, every change means "run everything again," which is slow, expensive and makes results drift for reasons nobody can identify.
Model changes are releases
Models are updated and retired on the provider's schedule, not yours. A model change can shift outputs across the whole estate even when nothing in your data changed. Treat it like any other release:
- Pin the model identifier for each run and record it with every output.
- Keep a labelled evaluation set and rerun it before approving a new model.
- Compare not just accuracy but stability: does the new model change accepted decisions, and are those changes improvements?
- Roll out gradually, and keep the ability to re-evaluate with the old results for comparison.
Cost controls belong in the architecture
Cost problems in LLM systems are usually architecture problems. The same separation that improves auditability also controls spend:
- Deterministic first. Every question answered by a parser is a question not paid for in tokens.
- Bounded context. Cost scales with the evidence a question needs, not the size of the source.
- Cache by input hash. Same facts, same prompt version, same model: reuse the result.
- Budgets at the gateway. Per-question and per-run token limits, enforced centrally, with alerts before they're hit.
- Measure the tail. Average cost per item is misleading; a few very large inputs often dominate the bill.
Where this argument has limits
This isn't an argument against more autonomous, agent-style systems. Letting a model decide which files to open or which tool to call can improve answers on messy, unusual inputs. But it makes cost less predictable and widens what has to be audited. The same principles still apply: log every tool call, bound the budget, validate claims against evidence, and keep acceptance in policy.
Nor does validation make a conclusion wise. It proves the claims are supported. Whether the recommendation is right for the business is still a judgment — which is exactly why high-impact decisions keep a human in the loop.
The underlying idea is simple. Use models for what they're good at — interpretation — and surround them with systems that are good at the rest. That's what turns an impressive demo into something an enterprise can defend.
A full reference design built on this idea is in An evidence-first AI system for software modernization.