Ablation Protocol

Isolating what makes the pipeline work — reviewer-only, topology, and Red ablations.

Catch-rate deltas mean nothing until you know which part of the pipeline earned them. These three ablations isolate the three structural claims — independent review, multi-agent topology, and adversarial Red — using only niki eval --live with config variants. No code changes, no new harness: each ablation is a config file plus the standard manifest comparison.

All runs require API keys and a sandbox. Record every run’s eval-manifest.json alongside the claim; an ablation without manifests on both arms is an anecdote, not a result. Budget rule: identical economic budgets per cell (same models, same max_revision_rounds, same dataset slice) — the only thing that varies is the ablated dimension.

A1. Reviewer-only vs full chain (does the second agent earn its tokens?)

Hypothesis: most of the catch-rate gain comes from having any second reviewing pass, not from the full four-role chain.

• Arm A (full): default config (Planner → Coder → Tester → Reviewer).
• Arm B (reviewer-only): [pipeline] with exactly two stages — { role = "coder", ... } then { role = "reviewer", ... } — Tester skipped, no Red, no SecurityAuditor.
• Compare: niki_catch_rate delta vs total_cost_usd ratio from the two manifests. If Arm B keeps ≥90% of the catch rate at ≤50% of the cost, the default topology is overbuilt for that defect class.

A2. Single-agent vs multi-agent (does isolation earn its tokens?)

Hypothesis: SingleAgent (Planner + solo Coder) matches the full chain on low-complexity tasks and loses on cross-file ones.

• Arm A: [pipeline] topology = "multiagent" (forced).
• Arm B: [pipeline] topology = "singleagent" (forced).
• Stratify by difficulty (--difficulty easy|medium|hard) and compare per stratum. Expect: parity on easy, divergence on medium/hard. A divergence that does not appear is evidence the task set is too easy, not that topology is free.

A3. Red on vs off (does adversarial review catch what review misses?)

Hypothesis: Red challenges catch a distinct defect population (shared misreadings), visible as caught_by_red cases the baseline misses.

• Arm A: [red_blue] enabled = true.
• Arm B: [red_blue] enabled = false (this is also what the stock niki eval baseline approximates).
• Compare: count of caught_by_red cases in Arm A, and the false_approval_reduction_pct delta. Then grade both arms’ catches with niki eval grade — Red catches that maintainers rule not merge-worthy are noise, not signal.

Reporting

Each ablation publishes, per arm: the manifest fields (date, version, harness commit + dirty flag, dataset slice, models), both catch rates, both costs with cost-per-caught, and the graded agreement rate. Three runs minimum per arm (n ≥ 3, report mean + min–max); a single-run ablation is a demo.