chore(mutation): shrink to report-only — drop the ratchet, baseline and graduation (#1457, #1781)

[thymikee]

Implements the #1781 wave-2 decision for #1457: shrink the mutation harness to report-only.

Why

The harness's only two real catches — #1474 (scroll-edge-state, 976-line behavioural suite) and #1475 (kernel-errors 55.19% → 86.26%) — both came from a human reading the score report. The ratchet/graduation/gating half never operated:

• mutation-baselines/decision-kernels.json was committed exactly twice (8cce0ef6b 07-27, 60400d04b 08-02), both times reading stableRuns: 0, gating: false.
• Neither commit reflects test: add scroll-edge-state suite, close mutation gap (#1455) #1474/test(kernel): cover all six errors.ts exports, kernel-errors mutation 55.19% → 86.26% #1475 — the committed baseline still claimed scroll-edge-state 28.75% and kernel-errors 55.19% long after both were fixed.
• mutation-weekly.yml computed a new baseline with --update, copied it aside, then git checkout --'d it and uploaded a proposal. Three weekly runs (08-02, 08-09, 08-16) all succeeded; nobody ran pnpm mutation:baseline in 3+ weeks.
• The affected lane really mutates on ~4% of PRs (27 of the last 30 runs finished in 1–4 min with an empty matrix).

A gate nobody arms is harness weight. The report is the part that paid.

What changed

• Deleted scripts/mutation/ratchet.ts + ratchet.test.ts (448 LOC), mutation-baselines/ (71 LOC), and every baseline/graduation/gating path in run.ts (--update, readBaseline/writeBaseline, recordRun, the gating argument to affectedMatrixFor, the baseline fields in the lane envelope), plus the mutation:baseline package script and the mutation-baselines/** entries in .oxfmtrc.json and the affected-workflow path filter. 880 deletions, 402 additions across 21 files (net −478).
• run.ts never exits non-zero on a score. The only non-zero exits are harness failures: a missing Stryker report, an incomplete shard set, a bad argument. Envelope stage ratchet → score; result now states whether the lane produced a report at all.
• Kept --no-run / mutation:check: verified it is not baseline-only — it is a plain alias for --report <default path>, i.e. "score an existing report", which both workflows still use through --report-dir.
• report.ts now renders from ModuleScore[]: kernel · score · killed · survived · total · timeout, plus the surviving mutants per kernel (capped at 20) — the list a test-strengthening PR actually works from. score.ts gained a timeout count (timeouts are counted as killed by Stryker, so a score propped up by slow mutants was previously invisible).
• mutation-weekly.yml: the ratchet job is now report — no --update, no proposed-baseline copy/restore dance, no proposed-baseline artifact. The weekly sweep, the --expect-shards completeness check, the $GITHUB_STEP_SUMMARY table and the report artifact all stay.
• mutation-affected.yml: advisory as before, step summary kept, no PR-comment bot added. Its on.pull_request.paths is now exactly LANE_TOOLING (scripts/mutation/**, scripts/lib/**, stryker.config.json) plus the workflow file — the only diffs that can produce a non-empty matrix, asserted in both directions by workflow.test.ts against the exported constant. The kernel-source and **/*.test.ts triggers were only starting a 1–4 min no-op select job on ~96% of PRs.
• stryker.config.json and modules.ts (KERNEL_MODULES) are unchanged — kernel scoping is kept; only the ratchet goes.
• Docs: docs/agents/testing.md "Mutation ratchet over decision kernels" → "Mutation report over decision kernels", with the three-sentence statement (report-only over the seven kernels, weekly full sweep + per-PR affected sweep; never gates; a low score is an input for a human-authored PR). All pnpm mutation:baseline/graduation mentions removed repo-wide (grep over docs/, AGENTS.md, CONTEXT.md, README.md, scripts/gate/, scripts/check-affected/checks.ts).

One judgement call worth reviewing

Deleting gating forced a decision about what the PR lane selects. Selecting on derived kernel ownership (the pre-existing --affected behaviour, unlocked once gating is gone) would have raised the lane's cost roughly 15x rather than shrinking it: I measured the last 40 merged PRs through derivedAffectedModules and 24 of 40 select a non-empty matrix, 23 of those selecting all seven kernels — a full ten-shard sweep, up to 30 min per shard, on ~60% of PRs for a report nobody gates on. (Only 1 of 40 touches a kernel source.) So the PR lane keeps its current effective behaviour — it spends mutants only when the diff touches the lane's own tooling, where a harness change has to prove itself against real mutants — and the rationale is now stated as a cost rule instead of "until the baseline graduates". Widening it is a separate, cost-bearing decision.

The lane as it stands after this change

• Catches: weak or decorative tests in the seven decision kernels, surfaced as per-kernel mutation scores plus surviving-mutant lists.
• Evidence: test: add scroll-edge-state suite, close mutation gap (#1455) #1474 and test(kernel): cover all six errors.ts exports, kernel-errors mutation 55.19% → 86.26% #1475 — two substantive, merged test-strengthening PRs written from exactly this table.
• Cost: 2,026 LOC in scripts/mutation/ (down from 2,453, plus the 71-line baseline JSON), one weekly 10-shard ubuntu sweep, and an advisory affected sweep on harness diffs only — after the path narrowing the other ~96% of PRs start no job at all, rather than a 1–4 min no-op one.
• Kill-criterion: if no test-strengthening PR cites a weekly score by 2026-11-16, park the weekly sweep to workflow_dispatch.

#1412's open question

Answered: downscope. Changed-line mutation over arbitrary PR diffs stays a bounded wave-3 experiment, not a lane.

Test plan

• pnpm mutation:test — 37/37 pass (ratchet.test.ts deleted; report.test.ts covers the table columns, the survivor list, and the status classification — Ignored/CompileError/RuntimeError leave the denominator — replacing the coverage ratchet.test.ts carried).

• pnpm format:check, pnpm lint, pnpm typecheck, pnpm check:layering, pnpm check:gate-manifest (47 checks / 33 lanes), pnpm check:gate-manifest:test, pnpm check:fallow — all green. pnpm check:production-exports reports the same 17 pre-existing findings as main, none in scripts/mutation/.

• CLI verified without a real sweep (too slow locally): run.ts --help parses; --affected --base HEAD takes the no-op path ("no decision-kernel modules affected", exit 0, envelope pass/complete); --list-affected --base HEAD prints []; --update is now rejected as an unknown option and still writes a failure envelope.

• Report rendering and the "a 0% kernel still exits zero" invariant are covered by report.test.ts and envelope.test.ts respectively, both through the real CLI where the envelope is involved.

• Proven end-to-end on real mutants in CI: because this PR edits the harness, the affected lane ran its canary sweep on this very branch — Select affected kernels → [kernel-errors] → 185 real mutants → the new table:

  ## Mutation score — affected decision kernels
  Stryker `9.6.1` · config `sha256:d192a2ed5f01` · report only — this lane never fails a build.
  
  | Kernel | Score | Killed | Survived | Total | Timeout |
  | --- | --- | --- | --- | --- | --- |
  | `kernel-errors` — Error retriability + hints | 86.49% | 160 | 25 | 185 | 0 |
  

  That number is also the last piece of evidence for this change: the committed baseline this PR deletes still claimed 55.19% for the same kernel, three weeks after test(kernel): cover all six errors.ts exports, kernel-errors mutation 55.19% → 86.26% #1475 raised it. The ratchet was comparing against a fiction.

Closes #1457

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.30 MB	2.30 MB	0 B
JS gzip	756.4 kB	756.4 kB	0 B
npm tarball	878.0 kB	878.0 kB	-8 B
npm unpacked	3.07 MB	3.07 MB	-93 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	29.2 ms	29.9 ms	+0.8 ms
CLI --help	72.0 ms	77.8 ms	+5.8 ms

Top changed chunks: no changes in the largest emitted chunks.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-18 15:48 UTC

[thymikee]

Adversarial review of #1828 (report-only mutation harness). Verified in a detached worktree at 271d0ce12: ran pnpm mutation:test (36/36 green, matches the PR body), traced every exit path in run.ts, grepped the whole tree for leftover ratchet/baseline/gating references, and diffed mutation-affected.yml/mutation-weekly.yml against the claims in the PR body.

Findings, ranked

1. Moderate — a scoring-behavior test was deleted and never replaced.
scripts/mutation/ratchet.test.ts (deleted) carried the only assertions on summarizeReport's status classification, e.g.:

test('statuses outside the score (Ignored, CompileError) leave the denominator', () => {
  const [score] = summarizeReport(mutants(['Killed', 'Ignored', 'CompileError']), ['kernel-errors']);
  assert.equal(score?.total, 1);
  assert.equal(score?.score, 100);
});

The new scripts/mutation/report.test.ts exercises Killed/Timeout/Survived/NoCoverage through renderReport, but nothing anywhere in scripts/mutation/*.test.ts now asserts that Ignored/CompileError/RuntimeError are excluded from the denominator — that behavior lives only in a comment at scripts/mutation/score.ts:34-39. Failure scenario: a future edit to tally() in score.ts that starts counting Ignored/CompileError toward total (silently deflating every reported score) would pass pnpm mutation:test clean. For a PR whose whole thesis is "the report is the part that pays," losing coverage of the report's own arithmetic is the one regression that would go unnoticed by the harness watching itself. Suggest porting that one assertion into score.test.ts/report.test.ts before merge.

2. Minor — the job-summary score table is skipped on the "incomplete shard set" failure path, contradicting the workflow comments.
mutation-affected.yml:161 and mutation-weekly.yml:114-115 both state "[the score table] lands in $GITHUB_STEP_SUMMARY" for either the success or the harness-failure branch. But in scoreModules() (scripts/mutation/run.ts:387-396), assertShardsCoverModules() throws before sweep() reaches emit() (line 411 vs 416) whenever a shard's report is missing/incomplete. On that exact "harness failure, not a score" day, the kernels that did complete successfully still don't get their scores published to the summary — only the raw envelope JSON and an error string do. This predates the PR (the ordering isn't touched by this diff) so it's not a regression this PR introduces, but the PR's own comment updates assert a stronger guarantee ("Either invocation writes the score table") than the code delivers. Worth a doc/comment correction, not a blocker.

3. Nit — LOC claim is off by one. PR body says "816 lines deleted, 281 added (net −535)"; gh pr view reports 817 deletions / 282 additions (net −535, same). Immaterial.

Checks that came back clean

• No leftover ratchet/mutation-baseline/graduation/gating/stableRuns/--update references anywhere in the tree (scripts/, workflows, docs, AGENTS.md, CONTEXT.md, README, package.json, .oxfmtrc.json, scripts/gate/declarations.ts, scripts/check-affected/checks.ts) — every hit from a repo-wide grep belongs to unrelated systems (layering ratchets, replay --update).
• run.ts traced end to end: sweep() always return 0; the only non-zero exits are --fail-envelope (explicit harness-failure declaration) and thrown Errors (missing report, bad --shard, incomplete shard set, bad arg). No path returns non-zero on a score.
• .oxfmtrc.json edit is justified — it only drops the now-nonexistent mutation-baselines/** ignore entry; fallow-baselines/** (unrelated, still live) is untouched.
• mutation-affected.yml's select job works with no baseline path (--list-affected → affectedMatrix(base) → affectedMatrixFor(changedFiles(base)), no gating arg); kernel-source path triggers (packages/kernel/src/errors.ts, src/daemon/ref-frame.ts, etc.) are all preserved, only mutation-baselines/** was dropped as claimed.
• docs/agents/testing.md reads coherently as report-only; no stray graduation/ratchet language remains in that section.
• "Closes #1457" is present in the PR body.
• pnpm mutation:test passes 36/36 locally on this branch, matching the PR's test-plan claim.

VERDICT: needs-changes — restore test coverage for the status-classification behavior in score.ts (finding 1) before merging; finding 2 is a nice-to-fix comment correction.

[thymikee]

All three addressed in 456367a28.

1 (moderate, denominator coverage) — ported into scripts/mutation/report.test.ts: statuses outside the score (Ignored, CompileError, RuntimeError) leave the denominator asserts total === 1, score === 100 for ['Killed','Ignored','CompileError','RuntimeError']. You are right that this was the one regression the harness-watching-itself would have missed; tally() now has the assertion sitting on it rather than a comment.

2 (job summary skipped on the harness-failure path) — took the code fix rather than the comment fix, since it is one moved line and strictly better: assertShardsCoverModules moved out of scoreModules() and now runs in sweep() after emit(). An incomplete shard set still publishes the table for the kernels that did complete, then throws — same exit code, same stage: 'score' envelope, more information on the day you need it. envelope.test.ts's incomplete-shard test now asserts the surviving kernel's row reaches stdout, and the weekly workflow comment states the ordering instead of over-claiming it.

3 (LOC nit) — PR body now reads 818 deletions / 314 additions (net −504), matching gh pr view after this commit.

pnpm mutation:test 37/37, format/lint/typecheck/fallow green. The previous CI run was fully green (33/33), including the kernel-errors canary sweep that rendered the new table from 185 real mutants.

[thymikee]

P2: the report-only envelope still drops all partial results in the common missing-shard case. The weekly workflow always passes --expect-shards 10, but readShardedReports() throws on count mismatch before reports are summarized or emit() runs. The new regression omits --expect-shards, so it does not exercise this route. Merge available shard reports and emit the partial table before enforcing the expected-count failure, then add a regression with the exact workflow argument shape. The mutation downscope is otherwise coherent and its canary evidence is good.

[thymikee]

dd002e15e: narrowed on.pull_request.paths to exactly LANE_TOOLING + the workflow file (asserted both ways in workflow.test.ts against the now-exported constant), dropped the header's stale "scope is kernel sources and any test reaching one" paragraph, and kept ownership.ts — derivedAffectedModules still has a live production caller in affectedMatrixFor, which adds any kernel a harness diff also touches on top of the canary.

[thymikee]

Right, and my earlier reorder missed exactly that route — readShardedReports() threw on the count before anything was summarized, so the previous regression (no --expect-shards) only exercised the zero-mutants path. 79ab53fe2: the merge now returns { report, count } and both verdicts moved into one assertCompleteSweep() that runs after emit(), same exit code and score stage. New regression uses the weekly shape verbatim — --report-dir … --expect-shards 10 with one shard present — asserting | \kernel-errors` … 100% |reaches stdout, stderr says1 report(s), expected 10, and the run still exits non-zero. No new machinery: one return value threaded through, two throws merged. pnpm mutation:test` 38/38, format/lint/typecheck/fallow green.

[thymikee]

CI note: Layering Guard is red on main, not from this PR. main HEAD 9a0d6dead fails identically — [R9 type-cycle-size] largest type-level import cycle grew to 47 files (baseline 46) and [R10 daemon-modularity] 17 daemon-server file(s) (baseline 16), pointing at src/daemon/daemon-command-registry.ts. It first appears on d76e0f94e (#1779, snapshot → device runtime), which landed after #1825 made the R9 shrink mandatory and lowered the baseline to 46. This PR touches only scripts/mutation/**, the two mutation workflows and docs, so it cannot participate in a daemon type cycle; pnpm check:layering is green against this branch's base (46 vs baseline 47). Every other lane on this PR is green.

[thymikee]

Re-reviewed exact head 79ab53fe: the missing-shard reporting blocker is fixed. The production path now merges/emits available shard scores before enforcing expected count, and the real-CLI regression uses the weekly --expect-shards 10 shape, proving partial stdout plus nonzero failure/envelope; reverting the reorder fails it.

Not ready yet: Layering is red at stale 47/46 and 17/16 ceilings, so rebase onto main/#1838 and rerun. Also refresh body scope arithmetic: current diff is 21 files, +402/-880 (net -478), not +348/-861.

[thymikee]

Rebased onto main (4bba40424, includes #1838): pnpm check:layering green locally (largest type cycle 46, R10 16). Body arithmetic refreshed to the current diff: 21 files, +402/−880 (net −478). CI re-running on head e843a9aa1.

[thymikee]

CI green on the rebased head (33/33; the one Android smoke red was an emulator/adb boot flake on the hosted runner, rerun passed). Marking ready.

[thymikee]

Re-reviewed exact e843a9a: clean and merge-ready. The rebase is bounded to the 21-file mutation/workflow/docs scope (+402/-880), body arithmetic matches, and the missing-shard fix still emits partial scores before failing expected-count with an exact --expect-shards 10 real-CLI regression. Layering, the 185-mutant canary, Android rerun, and all other exact-head checks are green.