How I Would Analyze 5,000 Git Repositories Using AI

Suppose an organization asks a simple-sounding question: across our 5,000 repositories, what do we have, how risky is it, and what should we modernize first? At ten repositories, a few engineers can answer that by reading code. At 5,000, the question becomes a distributed-systems problem with an AI component — and the AI component is the smaller part.

Start with the output, not the model

Before designing anything, define what the analysis must produce. For a modernization question, a useful per-repository record might include:

• Ownership, activity and last meaningful change.
• Languages, frameworks and runtime versions.
• Dependencies, with known-vulnerable or end-of-support components flagged.
• Build and deployment signals: is there a pipeline, does it build, is it containerized?
• A short, cited summary of what the system appears to do.
• A modernization classification and its blocking factors.

Most of these fields can be filled without a model. That observation shapes the whole design.

Intake and repository metadata

The first stage builds a registry from the source-control platform's APIs: repository ID, default branch, size, last commit date, archived status, topics and owning team. This is cheap and fast and immediately useful. It typically shows that a substantial share of repositories are archived, forks, empty or abandoned. Those can be classified with metadata alone and skipped by deeper stages.

Every later stage keys its work on (repository ID, commit SHA). Pinning the commit is what makes results reproducible and lets the system skip repositories that haven't changed since the last run.

Static analysis does the heavy lifting

For repositories that need deeper analysis, workers do a shallow clone at the pinned commit and run deterministic extractors:

• Manifest parsers for each ecosystem (.csproj and packages.config, pom.xml, package.json, requirements.txt and so on).
• Language detection and size metrics.
• Framework and API-usage detection.
• Pipeline and container file detection.
• Secret scanning, so credentials are found and redacted before anything is sent to a model.

Each fact is written with provenance: repository, commit, file, line range, extractor version.

Orchestration: queues, concurrency, idempotency

The workload is embarrassingly parallel per repository, but it's lopsided and failure-prone. Clones time out, rate limits are hit, one repository holds a 4 GB binary history. The orchestration has to assume all of this.

1. 01.

   Registry

   Enumerate repositories and metadata; triage archived, empty and forked.

2. 02.

   Clone & extract

   Shallow clone at pinned SHA; deterministic extractors; secret redaction.

3. 03.

   Select questions

   Decide which repositories need model interpretation, and which questions.

4. 04.

   Interpret

   Bounded-context model calls through a gateway with budgets and caching.

5. 05.

   Validate & accept

   Check claims against facts; apply versioned policy; route exceptions to review.

6. 06.

   Publish

   Write accepted records to the portfolio dataset and dashboards.

Key properties:

• Durable queues between stages, with dead-letter queues for items that fail repeatedly.
• Idempotency keys of (repo, commit, stage, stage-version). Re-delivered messages do nothing if the result already exists.
• Bounded concurrency per dependency. Separate limits for source-control API calls, clone bandwidth, CPU-bound extraction and model calls. Each has a different bottleneck.
• Large-repository lane. Repositories over a size threshold go to a separate queue with longer timeouts, so they don't block the bulk.
• Checkpointing. A run can stop and resume without redoing finished work.

Selective LLM use

The question isn't "how do we send 5,000 repositories to a model?" It's "which questions, on which repositories, genuinely need one?"

• Repositories classified from metadata alone (archived, empty) need no model call.
• Fields answerable by extractors — versions, dependencies, presence of a pipeline — need no model call.
• Model calls are reserved for interpretation: summarizing purpose, judging modernization fit, explaining risk.

Each call gets a specific question, a context of selected facts and code excerpts, and a structured output schema with citations. Context selection matters for both quality and cost: a few relevant files and the dependency facts usually beat a truncated dump of the whole repository.

Validation and persistence

Model outputs are validated mechanically before anything is accepted: cited facts must exist, named packages and frameworks must appear in the extracted data, and numbers must match. A versioned policy then accepts, caveats, retries or routes to human review. The evidence-first architecture describes this layer in detail.

State lives in three places with different jobs: a fact store (queryable, append-friendly), an interpretation store keyed by input hash, and an immutable audit log recording prompts, model identifiers, outputs and decisions.

Security and privacy

Source code is sensitive, and some repositories will contain things they shouldn't.

• Read-only, scoped credentials for source access; short-lived tokens.
• Secret scanning and redaction before any external call.
• An allow-list or deny-list for repositories that may be sent to a model at all, owned by security, not by the analysis team.
• Model endpoints approved for the data classification, with contractual terms that submitted code is not used for training.
• Workers in an isolated network segment; clones deleted after extraction.

Observability

You can't operate what you can't see. Per stage: throughput, queue depth, failure rate by error type, and latency percentiles. For model calls specifically: tokens in and out, cost per question, cache hit rate, validation rejection rate and review-routing rate. A run dashboard should answer, at any moment, "how far along are we, what's stuck, and what is it costing?"

A worked capacity estimate (all assumptions)

To size workers and budget, I'd start with explicit assumptions and refine them after a pilot of a few hundred repositories.

From those assumptions:

• Extraction: 3,500 × 3 min ÷ 40 workers ≈ 263 minutes, about 4.4 hours of ideal wall-clock. Allowing 1.5× for the long tail and retries gives roughly 6.5 hours.
• Model tokens: 11,200 × 12,000 ≈ 134 million input tokens and 11,200 × 1,500 ≈ 17 million output tokens.
• Model throughput: if the approved endpoint allows, say, 2 million input tokens per minute for this workload, input alone needs at least 67 minutes. In practice, concurrency limits and retries will matter more than the theoretical ceiling.
• Cost: input tokens × input price + output tokens × output price, for the chosen model at the time of the run. I'd compute it from current pricing rather than quote a figure that will be out of date.

The uncertainty is large and it's worth saying so. The tail of very large repositories, the real share needing interpretation, and the context size per question can each move the estimate by a factor of two. That's why the pilot exists: run a stratified sample, measure, then re-estimate before committing the full budget.

What I'd watch for

• Treating the first full run as the deliverable. The real value is the second run, a month later, that only touches changed repositories and costs a fraction of the first.
• Model drift across a long run. Pin the model version for the duration of a run.
• Review capacity. If 10% of 2,800 repositories route to human review, that's 280 reviews. Plan for it, or tighten what reaches review.
• Over-reading the summary. A cited summary of what a repository appears to do is a starting point for the owning team, not a substitute for asking them.

At this scale, the AI is the easy part to demo and the hard part to trust. The engineering around it — triage, determinism, idempotency, validation, cost control — is what makes the answer worth acting on.