intelligence-at-the-edges
Intelligence at the Edges: An Experience Report on a Multi-Agent, Multi-Provider Code-Review System with Deterministic Orchestration and Independent Oracles
Mike Fullerton
Independent Research — mike@mikefullerton.com
Draft of 25 June 2026. This is a system-and-experience report; results are from instrumented use rather than a large-scale benchmark (§8, §10).
Abstract
Large language models (LLMs) can read a code diff and produce plausible review comments, but production AI reviewers converge on the same failure: ten to twenty comments per pull request at roughly eighty percent noise, with no model of architectural decay and no path from a finding to a resolved change. We report on the design and three-month development of myteam, a multi-agent code-review system built around a single organizing thesis — intelligence at the edges, deterministic plumbing in the middle — in which a headless Python conductor owns all control flow (routing, retry, persistence, stopping) and the LLM is invoked only at explicitly enumerated judgment nodes. We describe four design moves that follow from this thesis: (1) a composition model (coalition → team → specialist → specialty) in which capability is pure data and the orchestration engine names no team; (2) a two-tier selection that combines a deterministic, zero-cost signal prefilter with a bottom-up agentic interest poll, eliminating the central-triage LLM call; (3) an independent-evaluator discipline applied at four nested levels — a verifier subagent blind to the worker, a confidence consultant blind to the finder, an adjudication pass (finish-review) blind to the discovery pass, and an intent oracle (conformance-review) blind to both the author's claim and the code generator; and (4) an instrumented build loop that pairs a fresh depth-0 generator with that intent oracle and decides when to stop in Python from recorded facts, not in the agent. The independent-evaluator principle is not original to us — it is the most consistent finding in the recent literature on LLM-guided optimization loops, where an external gate, not the model's self-assessment, does most of the work — and we trace each of our design decisions to that finding. We report what worked, what broke (including a critical message-routing bug that a mock-LLM test was structurally unable to catch), and where the design is still unproven. We make no claim to a benchmark win; we claim a coherent architecture for treating code review as a closed loop rather than a bolt-on comment generator, and an honest account of building it.
1. Introduction
1.1 The diff-reviewer ceiling
A competent LLM, handed a unified diff and asked "review this," will return a list of comments that look like code review. A 2026 survey of the field we conducted at the outset of this work (Greptile, Qodo 2.0, Cursor BugBot, CodeRabbit, and Anthropic's own /code-review) found a remarkably uniform shape to the output: ten to twenty comments per pull request, of which roughly four in five are noise; a best reported F1 around sixty percent, all measured on bug-finding benchmarks; and no tool that measures architectural decay or design erosion at all. The economic problem is not recall — models find plenty — it is precision and follow-through. A reviewer that cries wolf four times in five trains its human to stop reading, and a finding that never becomes a fix is theater.
Two structural observations motivated everything that follows. First, codebase context is the moat: the differentiator between tools is not the model but how much of the surrounding system the reviewer can see and reason against. Second, and more subtly, the diff is the wrong unit of work. The competitors review the diff; the diff cannot tell you whether the change did what it was supposed to do, only whether the lines that changed are locally defensible. Divergence from intent — building the wrong thing correctly — is invisible to a diff reviewer and is discovered late, after other work is layered on top, when unwinding it costs an order of magnitude more.
1.2 The thesis
The system we describe takes a deliberately unfashionable position on where the intelligence should live. The popular architecture for agentic systems puts the LLM in charge: the model decides what to do next, whether to retry, when it is done, and the surrounding code is glue. We invert this. A headless Python process — the conductor — owns the entire control plane: it routes work, fans out concurrency, enforces timeouts and retries, persists every intermediate state to SQLite, and computes the stop decision. The LLM is called only at the leaves of this tree, at nodes where genuine judgment is required: is this code defective? does this finding hold up? did this change satisfy the contract? We summarize the stance in a phrase that became the project's north star during an April 2026 design session:
Intelligence at the edges, deterministic plumbing in the middle.
This is not asceticism about LLMs; it is a bet about which parts of an agentic system are reversible and which are not. Routing, retry, storage, and stopping are mechanical, high-frequency, and must be correct every time; they are exactly the wrong place to put a stochastic component. Judging whether a piece of code is wrong is irreducibly a matter of judgment; that is exactly where the model earns its cost. The architecture is the consequence of taking that distinction seriously.
1.3 Contributions
This paper is an experience report, and its contributions are architectural and methodological rather than a new benchmark result. Concretely:
-
A composition model that makes capability pure data (§3.2–3.3). Teams, specialists, and specialties are markdown-and-JSON; the orchestration engine contains no team name and is forbidden by a boundary test from selecting from the roster. Adding or deleting a capability touches no engine code.
-
A two-tier, bottom-up selection mechanism (§4.2) that replaces the conventional central-triage LLM call with a deterministic signal prefilter (free) followed by an agentic interest poll in which each specialist self-assesses in isolation, never seeing the full roster. Selection fails open.
-
The independent-evaluator principle instantiated at four nested levels (§4.3–4.5, §5.3–5.4): worker→verifier, finder→confidence-consultant, discovery→adjudication, and generator→intent-oracle. We argue this single principle, drawn from the optimization-loop literature, is the load-bearing element of the whole design.
-
Capabilities as data over one pipeline (§5). Three review capabilities —
code-review,finish-review,conformance-review— share a single orchestration class and differ only in an objective specification (a JSON file) and team markdown. We describe the parameterization boundary and a config-driven scope model that turns "review the right subset" into a deterministic pre-filter. -
An instrumented build loop (§6) that closes the generate→evaluate→refine loop with a fresh depth-0 generator and an independent intent oracle, decides when to stop in Python from a cross-run metrics store, and replaces a hand-chosen stopping constant with a measured diminishing-returns inflection as data accrues.
-
An honest account of failure (§8, Appendix E), including a critical message-routing bug whose root cause was that a synthetic context document reached no reviewer, and a methodological lesson — a mock-LLM end-to-end test is structurally blind to routing and prompt bugs.
1.4 Roadmap
§2 traces the development from an interview agent to a code reviewer and the pivot that produced the conductor. §3 presents the conductor architecture and the composition model. §4 walks the review pipeline end to end. §5 shows how one pipeline serves three capabilities and introduces the scope model, finish-review, and the conformance oracle. §6 describes the build loop. §7 distills the recurring design principles. §8 reports our evaluation and experience, honestly bounded. §9 situates the work in the literature. §10 discusses limitations. §11 concludes. Appendices give the objective spec, a contract, the data model, the development timeline, and a failure case study.
2. Background and Motivation
2.1 From interview agent to code reviewer
The project began on 1 April 2026 as something else entirely: an interview agent for product discovery. The first design decision of record was skill, not agent — a decision driven by the interactive nature of an interview. An agent dispatched as a subprocess runs to a turn limit and returns; a skill runs in the user's main conversation context with unbounded back-and-forth. The discovery use case needed the latter. This distinction — between work that belongs in the foreground conversation and work that belongs in a headless process — recurs throughout the system and was, in retrospect, the seed of the conductor.
The first generation (Gen-1) was a Claude Code plugin whose every behavior lived in markdown workflow files read at runtime, with specialists as named markdown agents each carrying its own toolset. There was no orchestration engine; all coordination happened inside the skill interpreter. Gen-1 worked for a single-threaded interview but had no answer for multi-team coordination, parallel work, or durable state.
2.2 The conductor pivot
By mid-April 2026 the limits of running the coordinator inside the user's conversation were acute: context bloat, awkward parallelism, and no home for cross-team coordination. The pivot of 11 April 2026 — the project's largest — moved coordination into a headless Python process owning an asyncio event loop, dispatching LLM work through a pluggable Dispatcher abstraction and writing all state through an Arbitrator API backed by SQLite. Two consequences of this design deserve emphasis because they shaped every later decision.
First, the dispatcher is provider-agnostic by construction. The conductor spawns claude -p --output-format stream-json as a subprocess; it does not link the vendor Agent SDK. The SDK was rejected for a concrete reason — its per-token billing model is incompatible with a flat-rate Claude Max subscription — but the architectural payoff was larger than the billing one: a subprocess-and-protocol boundary let us add CodexDispatcher (OpenAI) and QwenDispatcher (Ollama) behind the same interface, making the system multi-provider from the first review release.
Second, crash recovery is free. Because every intermediate state — every dispatch, every attempt, every finding — is durable in SQLite the instant it is produced, a killed run can be inspected, resumed, or audited after the fact. State is not a cache of an in-memory object graph; the database is the run.
2.3 The precision war
In early June 2026 we paused feature work to study the competitive landscape in depth, producing a five-document research series. The strategic conclusion reframed the project. The competitors are diff reviewers; their ceiling is set by the unit of work, not the model. The opportunity is not a marginally-better comment generator but a closed-loop engineering partner: a system that owns the cycle from finding to adjudication to a verified merge gate. That reframing is the line connecting every capability in §5 and §6. It also surfaced two specific, adoptable techniques from Anthropic's own reviewer — a separate confidence-scorer model, blind to the finder's confidence, scoring against an anchored rubric with a hard threshold; and an explicit false-positive catalog (ban comments on unmodified lines, anything CI already catches, pre-existing issues). Both landed in our design (§4.5).
3. System Design: The Conductor Architecture
3.1 Design tenets
The conductor architecture rests on a small set of tenets, each of which is a consequence of "intelligence at the edges":
- No LLM in the control plane. Routing, retry, timeout, persistence, and stopping are Python. The model never decides whether to retry or when it is done.
- The engine names no team. All orchestration code is generic; capability is data. The invariant is enforced by a test, not a convention (§3.3).
- Every judgment node is independently testable and replaceable. Because the LLM is reached only through the dispatcher protocol at enumerated nodes, any node can be mocked, swapped to another provider, or re-run in isolation.
- State is durable and truthful. No blobs, no stored counts, no derived booleans (§7). Run status is derived from terminal facts at read time, never stored as a flag.
3.2 The composition model
Capability is organized in four layers. Figure 1 shows the relationships.
Figure 1: The composition model (coalition → team → specialist → specialty).
Engine code (right) names none of the left-hand inhabitants.
┌───────────────────────────────────────────┐ ┌──────────────────────┐
│ COALITION coalitions/myteam/ │ │ GENERIC ENGINE │
│ coalition.json → composite_planner │ │ pipeline/engine/ │
│ SKILL.md (%%COMMANDS%% = union) │ bind │ source/ │
│ ▸ installs as ONE Claude Code skill │◀──────▶│ │
│ ▸ ONE ~/.local/bin/<coalition> symlink │ via │ myteam-cli.py │
└───────────────┬─────────────────────────────┘ planner│ conductor/ │
│ members by reverse-domain id block │ objective_planner │
┌──────────┼───────────┬───────────────┐ │ composite_planner │
▼ ▼ ▼ ▼ │ review/ (pipeline) │
TEAM TEAM TEAM TEAM │ toolkit/storage/ │
devteam projectteam researchteam storyteam │ │
team.json (identifier + planner block) │ (NO TEAM NAMES, │
│ specialists/ specialities/<area>/ │ imports no team) │
▼ └──────────────────────┘
SPECIALIST specialists/<name>/specialist.md
frontmatter: always_on, specialities[ area/name ],
scope_<name>: subset
│ references by qualified id
▼
SPECIALTY specialities/<area>/<name>.md
## Worker Focus (what the worker analyzes)
## Verify (what the verifier checks)
A coalition is a named bundle of member teams that installs as exactly one Claude Code skill with one CLI symlink. Its coalition.json declares a planner block pointing at the generic composite_planner and lists members by reverse-domain identifier (e.g. com.agenticdeveloperteam.development). The single user-facing coalition, myteam, composes four teams: devteam, projectteam, researchteam, storyteam.
A team is the unit of capability and is entirely data: a team.json (identifier, command vocabulary, scopes, default LLMs, and a planner block), a team.md, and three peer pools of markdown — specialists/, the area-grouped specialities/<area>/, and team-leads/. Teams are flat and mix-and-matchable; a team can belong to many coalitions. Teams ship no Python. The only coupling from data to engine is the planner.module + planner.factory strings in team.json, which the engine resolves via importlib.
A specialist (e.g. security, software-architecture, reliability) is a markdown file whose frontmatter declares whether it is always_on, which specialties it draws from the shared pool by qualified area/name id, and a scope_<name>: subset per scope (§5.2). The devteam carries 23 specialists across 21 specialty areas. A specialty (e.g. security/transport, architecture/boundaries) is a markdown file with two load-bearing sections: ## Worker Focus, the analytical brief given to the worker, and ## Verify, the acceptance criteria handed to the independent verifier. Crucially, a specialty is owned by no specialist — specialists reference specialties; they do not contain them. This is what makes the pools mix-and-matchable.
3.3 Generic engine vs. team data: the planner boundary
The separation between generic engine and team data is the system's central structural invariant, and it is enforced rather than merely intended. All Python lives under pipeline/engine/source/; none of it contains a team name or imports a team module. A boundary test — test_engine_does_not_select_specialists.py — asserts that the generic pipeline never enumerates or selects from the roster; the team-lead is the engine's enforced sole interface to a team's roster. Resolution of a coalition's members is registry-mediated: composite_planner looks each member identifier up in the installer-written teams.json registry rather than walking the filesystem, so an authored coalition.json cannot aim the loader at an arbitrary path. The payoff is the property we most wanted from the system: dropping a team in or deleting one touches no engine code. It also made a non-development team (researchteam) viable as pure data, proving the boundary holds.
Two planners realize the model. The per-team ObjectivePlanner loads a team's command vocabulary from team.json, routes an incoming prompt either by a literal fast-path (the first token matching a known command) or, failing that, an LLM-backed natural-language router, and dispatches to the shared SpecialistPipeline or a builtin. The coalition-level CompositePlanner fans an incoming prompt to all member teams in parallel via asyncio.gather; exactly one team must claim the command (e.g. devteam claims code-review; the others return None). Zero claims is a no-op; more than one is a hard error — fail-fast on an authoring collision rather than a silent precedence rule.
4. The Review Pipeline
A code-review invocation flows through eight stages. Figure 2 gives the end-to-end view; the subsections detail the stages that carry the design's weight.
Figure 2: End-to-end review pipeline. Deterministic Python stages in [brackets];
LLM judgment nodes in (parens). Every box writes durable facts to SQLite.
myteam code-review HEAD~3..HEAD --risk --effort high
│
[0 ROUTE] CompositePlanner → devteam.ObjectivePlanner → SpecialistPipeline
│ load LLM guides; MultiTeamExecutor: one review leaf per --llm
▼
[1 ADAPT] git-changeset: git diff → FileChange{ path, signals } per file
│ (.py→python, .swift→swift, .ts→typescript, …)
▼
[2 SELECT] Tier-0 prefilter [deterministic, free]:
│ always_on floor ∪ (signal ∩ changeset), ∩ scope roster
│ Tier-1 (interest) each discretionary survivor self-assesses
│ in isolation, cheap model, no repo access — FAILS OPEN
▼
[3 REVIEW] bounded_gather(concurrency=8):
│ per specialist → (worker: claude -p) over its files + Worker Focus
│ → structured JSON findings (schema from objective spec)
▼
[4 VERIFY] per specialist → (verifier: cheaper claude -p) over findings + ## Verify
│ verdict ∈ {pass, fail}; on fail+retries: re-dispatch worker with
│ steering; findings ACCUMULATE (never replaced)
▼
[5 UPSERT] arbitrator.upsert_finding: anchor_key = file+symbol (or file+line-bucket)
│ UNIQUE(session, objective, anchor_key) → one finding, N attributions
▼
[6 CONSULT] (confidence consultant: ONE dispatch over all findings)
│ re-scores validated_confidence + validated_severity; NEVER drops
▼
[7 REPORT] per-LLM report-{llm}.md; cross-LLM issues-summary.md;
│ finish-review-handoff.json (disposition: null); review.log
▼
[8 FINALIZE] status DERIVED from terminal facts ∈ {completed, partial, failed};
CLI exits nonzero on anything but completed; results committed to git
4.1 Input adaptation and signal classification
The git-changeset adapter runs git diff over the requested range and builds a Changeset of FileChange objects, each carrying its path and a set of signals derived from extension and content (.py → python, .swift → swift, and so on). Signals are the deterministic substrate the selector matches against; they are computed once, in Python, and never inferred by a model. Different capabilities supply different adapters behind the same interface (§5).
4.2 Two-tier selection: deterministic prefilter, then bottom-up interest
Selecting which specialists review which files is where most multi-agent reviewers either over-spend (everyone reviews everything) or introduce a fragile central-triage LLM that must hold the whole roster in its head. We do neither. Selection is two-tier and strictly bottom-up.
Tier 0 — prefilter (deterministic, free). always_on specialists (e.g. security) are an unconditional floor. A technology-bound specialist is included only when its declared review signals intersect the changeset's signals. Both are bounded by the active scope's roster, applied first (§5.2). No model is called; the cost is a set intersection.
Tier 1 — interest (agentic, cheap). Each discretionary Tier-0 survivor is dispatched independently, on the cheap verifier-tier model, with only a summary of the changed files and no repository access, and asked a single question: is this change in your remit? No specialist ever sees the full roster or another specialist's answer; there is no triage agent with global knowledge. The combine step takes the union of interested picks and the always-on floor.
Design finding. Selection fails open: a flaky or timed-out interest poll includes the specialist rather than dropping it. We would rather pay for one unnecessary review than silently lose coverage, because a missing reviewer produces a false sense of safety that is invisible in the output.
This bottom-up structure has a property a central triage lacks: no single actor's failure can corrupt the whole selection. Each poll is isolated, cheap, and independently retryable, and every decision (review.selection.prefilter, review.selection.interest, review.selection.combined) is recorded as an event for audit.
4.3 The worker→verifier pillar
Every specialist review is a pillar of two independent dispatches. A worker (claude -p) receives the ## Worker Focus of each of the specialist's active specialties plus the relevant slice of the diff, and emits structured JSON findings conforming to the objective's schema, which is fail-fast validated. A verifier — a separate dispatch on a cheaper model that never saw the worker's instructions — receives the worker's findings plus the ## Verify acceptance criteria and returns {verdict, reason, missing}. On a fail verdict with retries remaining, the worker is re-dispatched with the verifier's specific gaps appended as steering, and the new findings accumulate — they never replace the originals; the later upsert deduplicates. A verifier dispatch that itself errors keeps all worker findings and marks the run partial rather than discarding work. The verifier runs a cheaper model deliberately: verifying every finding must be affordable, so the expensive model generates and the cheap model checks.
4.4 Anchored finding identity and deduplication
When two specialists, or two LLM providers, independently flag the same defect, the system must recognize them as one finding rather than three. We deduplicate on structure, not string match. Each finding is assigned an anchor_key built from normalized anchor fields — file + code-symbol, falling back to file + a five-line bucket — and a UNIQUE index on (session, objective, anchor_key) collapses duplicates into a single finding row carrying one finding_attribution row per finder. Findings are ranked by severity alone; we deliberately do not blend severity with confidence into a single score, because the two answer different questions (how bad, versus how sure) and a product hides both.
4.5 The independent confidence consultant
After all specialists report, a single consultant dispatch reviews every finding at once against an anchored rubric and an explicit false-positive catalog (§2.3), and re-scores each finding's validated_confidence and validated_severity on a 0–100 scale, emitting a per-finding verdict of verified, revised, or not-applicable. It is structurally independent of the finder — it never sees who raised the issue or with what confidence — which is the entire point: self-assessed confidence is worth little, an adversarial second opinion is worth a great deal. The consultant never drops a finding; it annotates. Its validated_severity, when present, becomes the one severity basis used everywhere downstream — the merge gate, the handoff JSON, and the human-facing display all read the same number through effective_severity(). Unifying the severity basis was itself a fix (Appendix E): earlier, the gate and the display could disagree.
4.6 Reporting and run-status truthfulness
Each provider yields a report-{llm}.md; a cross-LLM issues-summary.md coalesces the deduplicated findings and attributes each to the providers that raised it, with their individual severities. A finish-review-handoff.json is emitted with every run, pre-serialized in the exact shape the adjudication capability ingests, with disposition: null awaiting the author. Finally, run status is derived, never stored: completed, partial, or failed is computed at read time from the terminal states of dispatches, the presence of findings, and verifier verdicts; the CLI exits nonzero on anything but a clean completed. There is no success flag to drift out of sync with reality.
5. Capabilities as Data: One Pipeline, Three Objectives
5.1 The objective spec as the parameterization boundary
Three capabilities — code-review (quality), finish-review (adjudicate a prior review), and conformance-review (intent) — run on the same SpecialistPipeline Python class. They differ entirely in an objective specification, a JSON file (objectives/<name>.json) plus team markdown. Table 1 enumerates the axes the spec parameterizes.
Table 1: Three capabilities, one pipeline. Everything below is data, not code.
Axis code-review finish-review conformance-review
─────────────── ──────────────────── ────────────────────────── ────────────────────────
input adapter git-changeset changeset-with-handoff changeset-with-contract
finding noun issue issue gap
extra fields — verdict, deferral fields criterion, status
anchor file + symbol file + symbol criterion
default scope risk risk conformance
merge gate none severity ≥ 70 status∈{unmet,divergent}
∧ severity ≥ 50
team-lead code-review.md dev-team-manager.md conformance-review.md
guides pack review finish-review conformance-review
No capability ships capability-specific orchestration Python. A new review capability is an objective JSON, an input adapter behind the existing interface, and team markdown. This is the composition model (§3.2) paying off at the capability level.
5.2 The scope model: review the right subset, deterministically
With 23 specialists drawing on a large specialty pool, running everything on every review is expensive and noisy. An early attempt derived a --focus axis from specialty-area directory names, but directory names are filesystem artifacts, not user vocabulary, and could not express cross-cutting concerns. The shipped model (15 June 2026) defines a scope as an explicit, deterministic pre-filter declared in config, not derived from folders, in two layers: team.json declares which specialists each scope convenes (the roster), and each specialist.md declares which of its specialties run under that scope (the subset). Eighteen scopes ship. risk — the default — is the union of the defect-family scopes (security, safety, data, api, reliability, architecture, code-quality, performance): what is wrong with the code that exists right now. completeness is opt-in and asks the orthogonal question — measured against best-practice principles, what should be here and isn't? — and is deliberately excluded from the default, because presence-checking is a different and noisier activity than defect-finding. all applies no filter. Scopes compose by union, and the scope filter is applied before the signal and interest tiers, which still narrow further by relevance. The key property is that scoping is a deterministic, team-blind pre-filter the engine applies from config — not a model deciding what to look at.
5.3 finish-review: the manager/IC adjudication loop
code-review produces a list of findings; nothing closes the loop from findings to a merge decision. finish-review (19 June 2026) is that closure. Rather than rediscovering from scratch, it re-adjudicates a prior review's findings against the current code via the changeset-with-handoff adapter, which folds the handoff (findings plus the author's dispositions) into the changeset as a synthetic context document. In this mode the system plays dev-team manager and never edits code: it renders one stateless, independent judgment per finding — verifying it, assigning its own validated_severity, ruling it confirmed, adjusted, or refuted, and judging each of the author's deferral requests — and then computes a machine-readable merge-readiness.json from a config-driven gate block so the surrounding skill branches on a verdict rather than on prose. The separation is deliberate and three-way: what was found (discovery), what the author says about it (disposition), and what the adjudicator rules (verdict) are three independent judgments, and conflating any two of them is how review theater happens.
5.4 conformance-review: the intent oracle
Neither quality review nor adjudication answers the question that matters most when work is built on top of work: did this change do what it was supposed to do? conformance-review (24–25 June 2026) answers it against an explicit acceptance contract — a list of criteria — and adjudicates each criterion with one of three verdicts:
- met — the change does this;
- unmet — not done;
- divergent — done, but wrong: the change built something other than what the criterion asked for.
The divergent verdict is the one a diff reviewer cannot produce, because locally-defensible code can still be the wrong code. The gate blocks on unmet or divergent at or above severity 50. The roster pairs a dedicated always_on conformance specialist — a coverage backstop guaranteeing every criterion is adjudicated — with in-lane domain specialists convened by a conformance scope. The contract is folded into the input as a context unit, a routing detail learned the hard way (Appendix E). The oracle is independent three ways: of the author's claim, of code-review (quality and conformance are orthogonal axes), and of whatever generator produced the code (§6).
6. Closing the Loop: The Instrumented Build Loop
The conformance oracle is what lets the system generate code and certify it, not merely review code a human wrote. myteam build --contract <path> is a loop: propose an increment, gate it against the contract, keep or discard, repeat until done or stopped. It is driven by the skill (the foreground conversation), but every consequential decision in it is deterministic Python.
6.1 Generator/oracle separation and depth-0 builders
Each iteration dispatches a fresh depth-0 builder subagent to make one small increment toward the open gaps, commits the work-in-progress, then runs conformance-review as an independent oracle to gate it. Generator and oracle are different subagents and, by default, different models — an explicit anti-gaming measure, so the thing being graded did not write the grader. Depth-0 (the builder spawns no children) is a deliberate bound chosen for two reasons grounded in the literature and in field reports: first, agents accrue context drift — past roughly thirty minutes or twenty files they begin to hallucinate signatures and contradict earlier decisions — and a fresh subagent per increment never accumulates that drift; second, unbounded subagent recursion has been observed to spawn children tens of levels deep and burn millions of tokens in minutes, so depth-0 bounds per-iteration spend by arithmetic rather than by hope.
6.2 Deterministic stopping
The decision to stop is computed in Python — loop_status.decide_stop() — from recorded facts, and the skill cannot override a stop: true signal. The logic is layered: automatic logical stops (the contract is satisfied; the loop is oscillating between the same gaps; a regression appeared; the contract drifted), then, while no historical data exists, an arbitrary placeholder backstop of max(3, 3·n_criteria) iterations, and finally — once at least three completed runs of a given contract size exist — a learned backstop: the diminishing-returns inflection for that size bucket, derived from the cross-run store. The hand-chosen constant is scaffolding that self-replaces with measured data.
6.3 Cross-run metrics and the self-replacing constant
A separate SQLite store (loop-metrics.db) records three tables — build_run, build_iteration, build_iteration_criterion — holding raw facts only: no stored cost, no counts, no ready booleans. Cost is derived from a pricing table at report time; the progress curve is derived by query. This is the schema discipline of §7 applied to the loop's own telemetry, and it is what lets the stopping heuristic improve as the system is used rather than requiring a re-tune.
7. Design Principles
Several principles recur across the subsystems above. We state them explicitly because they, more than any single component, are the reusable content of this report.
P1 — Independent evaluator as a primitive. An external check beats self-assessment, and the system applies it at four nested levels: verifier vs. worker (§4.3), consultant vs. finder (§4.5), adjudication vs. discovery (§5.3), oracle vs. generator (§6.1). Where the loop works, the gate — not the model's own confidence — is doing the work. This is the through-line of the whole design (§9).
P2 — Intelligence at the edges. No LLM is in the control plane. Routing, retry, persistence, and stopping are Python; the model is reached only at enumerated judgment nodes (§3.1). This buys independent testability, free crash recovery, and a provider-agnostic core.
P3 — Capability is data; the engine names no team. Enforced by a boundary test, not a convention (§3.3). Optimizing for change: a capability can be added or deleted without touching engine code.
P4 — Demote labor, not judgment. Model selection follows reversibility and blast radius, not phase. Irreversible, high-blast-radius judgment (schema, auth, public contracts) runs at full strength with no cost optimization; reversible, low-blast-radius labor (a log format, an internal function shape) can run cheaply. The category is assigned by a human up front, never by a model assessing its own depth. Concretely, expensive workers generate and cheap verifiers check (§4.3).
P5 — No blobs, no computed values, truthful status. Tables hold indexable, searchable raw facts; narrative lives in a side-table; counts, costs, and pass/fail are derived by query, never stored. Run status is computed from terminal facts at read time (§4.6). An early build-loop schema draft that stored total_cost_usd, blocker_count, and merge_ready was rejected for exactly this reason.
P6 — Fail-fast on authoring errors. A missing objective JSON, an unresolvable coalition member, or a consulting document missing its rubric raises immediately rather than degrading silently (§3.3). The system would rather stop loudly at author time than produce a quiet, wrong result at run time.
8. Evaluation and Experience
We are deliberate about what this section is and is not. It is an experience report from instrumented development and use over three months; it is not a large-scale benchmark of precision and recall against a labeled bug corpus. We have not yet run such a benchmark, and §10 treats that gap as the headline limitation. What we can report honestly is the instrumented behavior of the loop and a set of concrete bugs the architecture surfaced and fixed.
The loop's quantitative thread. The build loop is instrumented end to end, and its cross-run store makes three behaviors observable rather than anecdotal: oscillation (the loop returning to the same unresolved gaps), detected and used to stop; regression (a previously-met criterion becoming unmet), detected and used to reset --hard the offending increment; and diminishing returns (the iteration at which new criteria stop being resolved), which becomes the learned stopping backstop once enough runs of a given contract size exist. These are the analogs, in our setting, of the optimization-loop literature's central observation that the gate — not the generator — drives convergence (§9): in that literature only a minority of LLM proposals are even legal, and the legality rate climbs only because the gate feeds failure back. Our oracle plays the same role for our generator.
The qualitative thread: bugs the architecture caught. A /code-review max pass over the finish-review capability surfaced eleven findings, all fixed before merge. The most important, detailed in Appendix E, was a critical routing bug: the handoff document — the entire point of finish-review — reached no specialist worker, because a synthetic FileChange carrying no technology signals was filtered out of every discretionary specialist's file slice. The fix was an explicit context flag that routes a context unit into every selected specialist's slice. Two lessons compounded here. First, the bug was a selection bug, which validated the decision to make selection deterministic and event-logged — the absence was visible in the recorded selection events once we knew to look. Second, and more uncomfortable, our mock-LLM end-to-end test was structurally unable to catch it: a mock that ignores its prompt cannot reveal that the prompt was missing its most important input. We now assert routing deterministically rather than trusting an end-to-end pass with a stand-in model.
What we are claiming, and what we are not. We claim a coherent architecture, an enforced separation between engine and capability, and an honest instrumented account of a loop that converges under an explicit oracle. We do not claim a measured precision or recall figure, a benchmark win against any named competitor, or that the learned stopping backstop has accumulated enough runs to be more than promising. Those are §10.
9. Related Work
LLM-guided optimization loops. The single most direct influence is the line of work on closing a propose → measure → feedback → refine loop between an LLM and a hard, external evaluator. Merouani, Kara Bernou, and Baghdadi's Agentic Auto-Scheduling (arXiv:2511.00592, PACT 2025) places an LLM in such a loop with the Tiramisu polyhedral compiler, reporting a 2.66× geometric-mean speedup across roughly 150 benchmark instances — but the telling statistic for us is that only about a third of proposed schedules even ran successfully, and the illegal-proposal rate fell from roughly sixty percent at the first iteration as the model learned from the compiler's gate. The compiler, not the model's self-assessment, does most of the work. CudaForge (arXiv:2511.01884) reinforces the generator/oracle-heterogeneity point, reporting that pairing a coding model with a different judging model reached full correctness with a 1.83× speedup. These two findings — an independent gate is load-bearing, and a different model should hold the gate — are the direct ancestors of our independent-evaluator principle (P1) and our generator/oracle separation (§6.1).
Multi-agent debate and orchestration frameworks. We considered, and declined, both multi-agent-debate orchestration and the vendor workflow tooling. A deep-research pass over the debate literature found that across five debate frameworks, nine benchmarks, and four models, none consistently beat single-agent Chain-of-Thought or Self-Consistency at matched compute, and some flip correct answers under peer pressure (sycophancy). The robustness gains attributed to debate trace to model heterogeneity — which our multi-provider design already supplies — not to many copies of one model. Separately, the vendor Workflow tool is not an importable library and requires running inside the vendor's CLI session, which would break the provider-agnosticism that the conductor exists to protect. We therefore kept a deterministic Python-and-SQL orchestration core (§3) and got heterogeneity from multiple providers rather than from a debate protocol.
AI code-review tools. Our landscape study (§2.3) covered Greptile, Qodo 2.0, Cursor BugBot, CodeRabbit, and the vendor's own /code-review. The recurring lessons — codebase context is the moat, specialization beats one general reviewer, and precision is the war — shaped the composition model and the confidence consultant. We adopted, specifically, the vendor reviewer's separate, blind confidence-scorer with an anchored rubric and a hard threshold and its false-positive catalog (§4.5).
Agentic coding benchmarks as a counterweight. We hold the optimization-loop optimism against sobering performance-engineering results. On SWE-Perf, human experts achieved an order of magnitude larger speedups than an agent (≈10.85% vs. ≈1.76%); on SWE-fficiency, agents reached well under a fifth of expert speedup. We read these as a boundary condition, not a refutation: the build loop (§6) targets an explicit contract — a hard, well-defined oracle — rather than open-ended "make it better," which is precisely the regime where the optimization-loop literature shows an external gate helps most.
10. Discussion and Limitations
No precision/recall benchmark yet. The largest gap is the one named in §8: we have not measured the system against a labeled bug corpus, so every claim about review quality is qualitative. The field's best reported F1 is around sixty percent and only on bug-finding tasks; until we measure, we cannot place ourselves on that axis, and we will not pretend to. Building such a benchmark — and, harder, one that captures architectural decay, which no existing tool measures — is the top of the future-work list.
Mock-LLM blindness. The routing bug of Appendix E exposed a methodological hole: an end-to-end test with a mock model that ignores its prompt cannot catch prompt-construction or routing defects, which are among the most damaging. Deterministic routing assertions close part of this, but a class of prompt-quality regressions remains catchable only by real-model evaluation, which is expensive to run continuously.
Heuristics still maturing. The learned stopping backstop (§6.3) requires at least three completed runs of a given contract size before it replaces the hand-chosen constant; for novel contract sizes the system still runs on scaffolding. We believe the self-replacing design is right, but it is unproven at scale.
Provider variance and cost. Multi-provider fan-out is a strength for heterogeneity but a cost multiplier, and providers disagree in ways the cross-LLM summary surfaces but does not resolve. The cheap-verifier/cheap-consultant choices (P4) bound cost, but a full multi-provider, high-effort review is not free, and we do not yet have a principled cost/quality frontier.
Single-author, single-codebase experience. This report reflects development by a small team on a bounded set of repositories. The composition model's claim — that capability is cleanly addable as data — is supported by a second, non-development team existing as pure data, but it has not been stress-tested by many independent authors adding many teams.
11. Conclusion
We set out to build a code reviewer that does not cry wolf and does not stop at a comment. The architecture that resulted is organized around a single inversion of the prevailing agentic design — intelligence at the edges, deterministic plumbing in the middle — and around a single borrowed principle whose importance the recent literature makes hard to overstate: an independent evaluator beats self-assessment, and the gate, not the generator, is where convergence comes from. From those two commitments follow the composition model that makes capability pure data, the bottom-up selection that needs no central triage, the four nested independent-evaluator pillars, the three capabilities sharing one pipeline, and the build loop that generates code and certifies it against an explicit contract with a different model holding the gate. We have been candid about what we have not done — there is no benchmark number here, and some heuristics are still scaffolding. What we offer instead is a coherent, instrumented design for treating code review as a closed loop, and an honest record of building it, bugs and all.
Acknowledgments
The system described here was designed and built in close collaboration with Claude (Anthropic), via Claude Code, across the design sessions and research syntheses cited throughout. Several of the architectural commitments in this paper — the conductor pivot, the model-routing philosophy, and the no-workflows decision — were worked out in recorded human–model design conversations that are part of the project's own documentation.
References
The references below combine external works engaged during the project with the project's internal research syntheses and decision records, which are cited by repository path. Internal documents are dated design records, not peer-reviewed publications, and are marked [internal].
[1] M. Merouani, I. Kara Bernou, and R. Baghdadi. Agentic Auto-Scheduling: An Experimental Study of LLM-Guided Loop Optimization. arXiv:2511.00592, PACT 2025.
[2] CudaForge: LLM-Guided CUDA Kernel Optimization with Heterogeneous Generator and Judge. arXiv:2511.01884, 2025.
[3] SWE-Perf: Measuring Performance-Engineering Ability in Software Agents. Benchmark referenced in project research synthesis [9].
[4] SWE-fficiency: Agent Speedup Against Expert Baselines. Benchmark referenced in project research synthesis [9].
[5] Anthropic. Claude Code /code-review plugin. Product documentation, 2026. (Source of the blind confidence-scorer and false-positive-catalog techniques; see [8].)
[6] Greptile; Qodo 2.0; Cursor BugBot; CodeRabbit. AI code-review products surveyed. See landscape study [8].
[7] Architecture reference. docs/architecture.md. [internal]
[8] Code-review research series: official /code-review, tool landscape, gold-standard recommendations, vision and strategic moat, backend integration. docs/code-review-research/01–05. [internal]
[9] Agentic loops with Claude Code — the CompPilot pattern in practice. docs/research/agentic-loops-claude-code.md. [internal]
[10] Claude Workflows orchestration decision (no). docs/planning/2026-06-10-claude-workflows-orchestration-decision.md. [internal]
[11] Scopes redesign plan. docs/planning/2026-06-15-scopes-redesign-plan.md. [internal]
[12] finish-review follow-ups and hardening. docs/planning/2026-06-19-finish-review-followups-and-hardening.md. [internal]
[13] Conformance oracle and build loop design. docs/planning/2026-06-24-conformance-oracle-and-build-loop-design.md. [internal]
[14] Model selection, advisor, and delegation. docs/conversations-with-claude/2026-06-12-model-selection-advisor-opusplan-delegation.md. [internal]
[15] Pain-points catalog. docs/painpoints.md. [internal]
Appendix A — The Objective Specification (excerpt)
An objective JSON parameterizes the shared pipeline for one capability. Abridged from objectives/code-review.json:
{
"objective": "code-review",
"input_adapter": "git-changeset",
"finding": {
"noun": "issue",
"anchor": ["file", "symbol"],
"fields": {
"title": { "required": true },
"severity": { "type": "score", "range": [1,100], "required": true },
"confidence": { "type": "score", "range": [1,100], "role": "confidence",
"required": true },
"description": { "type": "text", "role": "body" },
"file": { "nullable": true },
"line": { "type": "integer", "nullable": true },
"symbol": { "nullable": true },
"speciality": { "type": "enum", "source": "specialist-specialities",
"required": true }
}
},
"default_scope": "risk",
"guides_pack": "review",
"verify": { "max_retries": 1 },
"team_lead": "code-review"
}
conformance-review.json is identical in shape but adds a required criterion field (which is also the anchor) and a status enum (unmet, divergent), and carries a gate block: block when status ∈ {unmet, divergent} and severity ≥ 50.
Appendix B — An Acceptance Contract (illustrative)
A contract is a list of adjudicable criteria. The conformance oracle returns met / unmet / divergent for each.
Contract: "Add rate limiting to the public ingest endpoint"
C1. Requests beyond N/min per API key receive HTTP 429. [adjudicable]
C2. The limit N is configurable, not hard-coded. [adjudicable]
C3. 429 responses include a Retry-After header. [adjudicable]
C4. Existing authenticated callers under the limit are
unaffected (no regression). [adjudicable]
C5. Rate-limit state survives a single-process restart. [adjudicable]
A change that adds a fixed-window limiter with a hard-coded N satisfies C1 but is divergent on C2 — locally working code that is not what was asked. A diff reviewer reports nothing; the oracle blocks the merge.
Appendix C — The Finding Data Model (generic, objective-neutral)
The findings schema is capability-independent; the same tables hold a code-review issue and a conformance-review gap.
Table 2: Finding storage. No blobs in primary rows; narrative in body side-table.
table columns (abridged) purpose
─────────────────── ──────────────────────────────────────────────── ─────────────────────
finding finding_id, session_id, objective, title, one row per distinct
anchor_key ── UNIQUE(session,objective,anchor) defect (dedup key)
finding_anchor finding_id, anchor_name, anchor_value queryable anchor parts
finding_attribution attribution_id, finding_id, llm, specialist_id, one row per finder
speciality, confidence_score
finding_score attribution_id, score_name, score_value severity, relevance, …
finding_attribute attribution_id, attribute_name, attribute_value line, citation, …
finding_validation finding_id, consultant, verdict, consultant's blind
validated_confidence, validated_severity re-score (§4.5)
body owner_type, owner_id, body_format, body_text all narrative content
Counts, costs, and pass/fail are not stored; they are derived by query (P5). dispatch rows carry raw token measurements (input_tokens, output_tokens, cache fields); cost is derived from a pricing table at report time, never persisted.
Appendix D — Development Timeline
Table 3: From interview agent to closed loop (2026).
date milestone
─────────── ────────────────────────────────────────────────────────────────────
04-01 Genesis as an interview agent. Decision: skill, not agent.
04-11 The conductor pivot. "Intelligence at the edges, deterministic
plumbing in the middle." Provider-agnostic Dispatcher; SQLite state.
04-19 Specialist-as-subprocess: worker + verifier pillar; dispatch/attempt
call tree recorded.
05-23 Gen-2 consolidation; Gen-1 archived to legacy (reference-only).
05-25 First code-review pipeline: changeset diffing, signal prefilter,
per-specialist dispatch, PR comments. Codex + Qwen dispatchers.
05-26 Parallel multi-LLM fan-out via MultiTeamExecutor; per-LLM reports.
05-28 Teams-as-data: team Python deleted; planner block + importlib;
objective-neutral finding* tables; a pure-data research team.
06-02 Code-review gold-standard research series; "closed-loop partner" vision.
06-09 Run quality: anchored cross-specialist dedup; confidence consultant;
concurrency 8; live narration.
06-10 Decision: no vendor Workflow tool / SDK; keep deterministic core.
06-13/14 Shared specialty pool; bottom-up interest replaces central triage;
coalition layer (coalition → team → specialist → specialty).
06-15 Config-driven scopes (18 scopes; risk default, completeness opt-in, all).
06-19 finish-review: handoff adjudication, dev-team-manager, merge-readiness gate.
06-24/25 Conformance oracle (met/unmet/divergent) + instrumented build loop;
depth-0 builders; Python-decided stops; cross-run metrics store.
Appendix E — Case Study: The Handoff That Reached No Reviewer
The single most instructive bug of the project is worth a full reconstruction, because it validates two design choices and indicts one testing choice.
Symptom. finish-review ran to a clean completed status and produced adjudications, but the adjudications were as if the prior review's findings did not exist. The handoff was being ignored.
Root cause. The handoff document was folded into the changeset as a synthetic FileChange. But a synthetic file has no source extension and therefore no technology signals. The two-tier selector (§4.2) routes a file to a discretionary specialist only when the file's signals intersect the specialist's; with no signals, the handoff intersected nothing and was filtered out of every discretionary specialist's slice. Only the always-on consultant, which sees all findings post hoc, ever saw it — and it is not a reviewer. The most important input reached no worker.
Fix. An explicit FileChange.context flag. A context unit is not subject to signal matching; it is routed into the slice of every selected specialist unconditionally. The handoff (and later the conformance contract, §5.4) is a context unit.
Why it validates the architecture. Selection is deterministic and every decision is logged as an event. Once we suspected selection, the absence of the handoff from every selection.combined event made the root cause visible in seconds — there was a durable, queryable record of the mistake rather than a guess about a model's behavior.
Why it indicts a test. The capability had an end-to-end test that passed, using a mock LLM. A mock that returns canned output regardless of its prompt cannot detect that the prompt was missing its most important section; the routing defect was invisible precisely because the mock ignored the very input that had gone missing. The lesson, now a standing rule: assert routing and prompt construction deterministically, and do not let a green end-to-end run with a stand-in model stand in for evidence that the real prompt is correct.