An evidence-first AI system for software modernization

The problem

Modernization programs need answers to the same questions for every application: what does this system do, what does it depend on, how risky is it to change, and which migration path fits? Answering by hand doesn't scale to hundreds of applications. Large language models are good at reading unfamiliar code and summarizing intent, so it's tempting to point a model at every repository and ask it.

That approach fails in predictable ways:

• The model states facts it did not observe — a dependency that isn't there, a framework version that's wrong.
• Two runs on the same repository give different answers, so nobody can tell whether a change in the result reflects a change in the code.
• Nobody can explain why an application was classified a certain way, which makes the output unusable for a release or funding decision.
• Cost grows with repository size instead of with the questions actually being asked.

The design goal is a system whose outputs can be defended: every conclusion traceable to evidence, every model contribution bounded and recorded, and every acceptance decision made by explicit policy.

Core principle

Deterministic tools establish facts. Models interpret facts. Policy decides what is accepted. Everything is recorded.

The model is a component with a narrow job, not the system of record.

Pipeline

1. 01.

   Source scan

   Clone at a pinned commit; record repository, commit SHA, scanner versions.

2. 02.

   Deterministic facts

   Parsers and analyzers extract project files, packages, APIs, config, call sites — with file and line provenance.

3. 03.

   Targeted interpretation

   The model answers specific questions using only the selected facts and code excerpts it is given.

4. 04.

   Independent validation

   Every claim in the model's answer is checked against the fact store; unsupported claims are flagged.

5. 05.

   Policy acceptance

   Versioned rules decide: accept, accept with caveat, or route to human review.

6. 06.

   Audit trail

   Inputs, prompts, model version, outputs, validation results and decisions stored immutably.

1. Source scan and deterministic extraction

Everything that can be known without a model is extracted without one. For a .NET estate that includes project files and target frameworks, package references, API usage from compiled or parsed code, configuration keys, connection strings (redacted), hosting markers and entry points. For other ecosystems the extractors change; the contract does not.

Each fact is stored with provenance: repository, commit SHA, file path, line range, extractor name and extractor version. A fact without provenance is not a fact.

2. Targeted model interpretation

The model receives a question, a bounded context and an output schema:

• Question: "Which modernization path best fits this application, and what are the blocking dependencies?" — not "Analyze this repository."
• Context: the relevant facts plus selected code excerpts, chosen by the system rather than by stuffing the context window with the whole repository.
• Schema: structured output where every claim must cite fact IDs or excerpt IDs.

This keeps token use proportional to the question, makes outputs comparable across runs, and gives validation something concrete to check.

3. Independent validation

Validation does not ask the same model whether it was right. It checks the answer mechanically:

• Every cited fact ID exists and says what the answer claims.
• Every named package, API or framework appears in the fact store.
• Numeric claims (counts, versions) match the extracted values exactly.
• The answer conforms to the schema, and required fields are present.

Where judgment is unavoidable — "is this summary of business purpose reasonable?" — a second, different evaluator can score it, but that score is recorded as an opinion, not promoted to a fact.

4. Policy-based acceptance

Acceptance rules are code, versioned and reviewed like any other code:

Because policy is separate from interpretation, the organization can tighten or relax acceptance without re-running a single model call.

5. Audit trail and re-evaluation

Each result stores: the inputs (fact IDs and excerpt hashes), the prompt template version, the model identifier and parameters, the raw output, validation results, the policy version and the decision. This enables two distinct operations:

• Re-extract when the source changes (new commit) or the extractor improves. Facts are rebuilt; interpretations that depended on changed facts are invalidated.
• Re-evaluate when only policy changes. Saved facts and interpretations are re-judged against new rules at almost no cost.

Re-interpretation — calling the model again — is needed only when the question, the prompt or the model changes. Separating these three is what keeps cost and drift under control.

Architecture

1. Intake

   • Repository registry
   • Commit pinning
   • Work queue
   • Idempotency keys

2. Extraction

   • Language-specific parsers
   • Package & API analyzers
   • Config & secret redaction
   • Provenance tagging

3. Interpretation

   • Question templates
   • Context selector
   • Model gateway
   • Structured output

4. Validation & policy

   • Claim checker
   • Schema validation
   • Versioned policy engine
   • Human review queue

5. State

   • Fact store
   • Interpretation store
   • Immutable audit log
   • Result cache by input hash

6. Security

   • Source access via scoped tokens
   • Data-residency controls
   • Redaction before model calls
   • No training on submitted code

7. Observability

   • Per-stage latency
   • Token & cost per question
   • Rejection rates
   • Drift between runs

Design decisions

A model gateway, not direct SDK calls. All model traffic goes through one service that enforces approved models, redaction, rate limits, budget caps and logging. Swapping or adding a model is a configuration change plus an evaluation run — not a code change across the system.

Cache by input hash. If the facts, excerpts, prompt version and model are identical, the previous output is reused. Re-running an unchanged repository costs nothing in model calls.

Human review as a first-class queue. Reviewers see the question, the evidence, the model's answer and the validation result side by side, and their decision is recorded with the same provenance as an automated one.

Alternatives considered

Risks and mitigations

Validation criteria

Before trusting the system on a full portfolio, I would expect evidence on a labelled sample:

• Unsupported-claim rate after validation (target: none accepted).
• Agreement with expert classification on a held-out set, broken down by modernization path.
• Run-to-run stability: the same inputs produce the same accepted decision.
• Cost per application and its variance — the long tail of huge repositories matters more than the average.
• Review load: the share of results routed to humans, and whether that share falls as policy and extractors mature.

What I would do differently, and open questions

• Build the evaluation set first. It's the only way to compare prompts, models and policies honestly, and teams consistently start it too late.
• Resist making the model do arithmetic or lookups. Counting packages or listing API calls is the extractor's job. Every time a model does it, a validation rule has to undo it.
• Open question: how much autonomy for agentic context selection? Letting a model choose which files to read improves answers on unusual repositories. The cost is harder-to-predict spend and a larger audit surface. I'd allow it behind a budget, log every tool call, and measure whether it beats a deterministic selector on the evaluation set.

The reasoning behind this design is laid out more fully in AI Can Interpret Evidence. It Shouldn't Invent It.