[willhea]

Based on develop. #720 has merged, so the diff shows only this PR's own commits. Code-only.

The runtime skip ceiling already in the repo watches only 25 of 104 test modules. This PR widens it to the whole suite — and in doing so found five live defects the static rules report green over. It does not follow, and this PR does not claim, that the runtime boundary could replace the static scan; see the note at the end.

1. The skip ceiling now watches the whole suite

tests/conftest.py carried a session-level ceiling: a skip in a watched module that is not declared fails the session. It watched an enumerated roster of 25 test modules out of 104, so a fixture-absence skip in any of the other 79 reported green. That roster is the shape #654 removed from the file scan one tier over, with the same defect: a module nobody listed is exempt the day it is created.

The default is now watched. The three tier rosters survive to route a skip to the allowlist documenting it; membership is no longer something anyone has to remember.

The suite named the widening cost, not me: seven declarations, six with the corpus present and one more CI-shaped. All seven are opt-in modes or environment requirements rather than corpus gaps — four regenerate maintenance commands behind a flag, one live-network gate (#278/#342), one local-only byte comparison, one two-root collision probe. None is a fixture that should simply be committed, so unlike ALLOWED_FAST_GATE_SKIPS an empty state is not the goal here.

Proven to fire, in a module no roster names. Pointing a fixture constant in test_classify_bill.py at an absent bill:

52 passed, 1 skipped        <- pytest's own verdict
rc=1, "default suite-wide skip ceiling"   <- the session's

The counterfactual isolates it: the identical mutation against the pre-widening conftest exits 0 with no ceiling output.

2. Five CWD-dependent defects, fixed

#404 moved test_data/ to tests/data/ and left a regex behind to stop the CWD-relative spelling coming back. That regex enumerates the tree names it knows — test_data and tests/data — so a CWD-relative path into any other directory is unpoliced. Running the suite from a directory that is not the repository root found five live ones:

Site	Shape
test_canonical_tree.py:26	Path("schema/canonical-diff.schema.json") (failed 2 tests)
test_diff_bill.py:1397	bare "diff_bill.py" to subprocess.run, no cwd=
test_corpus_manifest.py:681	child pytest collection against a relative target, exit 4
test_reconcile.py:209	repo-relative literals behind a skip guard — skipped green

The last one is the one that matters, and no boundary in the repository caught it alone:

• the static rules enumerate test_data/tests/data, never tests/corpus — and a bare repo-relative string is deliberately not treated as path construction
• the CWD-varying run missed it because it is a skip, not a failure
• the ceiling missed it because from the repository root the fixture resolves and nothing skips

The two composing caught it: the foreign CWD created the condition, the widened ceiling made the silent skip loud. That is the argument for the runtime boundary in one example — it observes the consequence, so it catches spellings no enumeration anticipated.

3. A cwd-independence CI job

Every other job runs from the repository root, so this class of defect is green in all of them.

Non-matrix on purpose: it varies the working directory, not the interpreter, and path resolution does not differ across supported versions.

Every non-browser tier, not the fast one originally proposed. That correction was measured: three of the five live dependencies sat behind @pytest.mark.slow, including the only one that skipped rather than failed — a fast-tier job would have shipped green while missing the case it exists for.

Proven to fire, and the proof is the whole argument:

	result
repository root	10 passed, rc=0 — the bug hides, as it really did
this job	3821 passed, zero failures, rc=1

It reddens with no test failing at all.

4. The browser tier is exempt, and that is deliberate

Widening the ceiling swept in the browser tier, whose skip channel is the module-scoped chromium fixture. #599 established that skipping there is correct for the default tier — a contributor's machine may lack Playwright — while CI's browser step passes --run-browser to make the same condition a failure.

Measured before the exemption: with the browsers path pointed at nothing, 41 undeclared skips and exit 1. The widening turned documented local behaviour into a wall of red for exactly the contributor #599 accommodates. That regression was introduced and fixed inside this branch, found by running the one tier the non-browser suite never covers.

The flag is the stronger control and needs no list. An allowlist could not express it anyway: the skip reason interpolates the launch exception, so no entry could ever match. The exemption is two named modules, not the marker or a prefix, so it cannot widen into a way to silence any skip by phrasing it like a browser one — both directions are pinned in a test.

⚠️ Behaviour change for in-flight branches

This makes previously-green sessions fail. Any branch with an undeclared skip in one of the 79 previously-unwatched modules now fails the session with a named case and reason. That is the intended effect, but it will surprise someone mid-review, so it is worth saying out loud rather than being discovered.

The remedy is in the failure message: fix the regression, or declare the case in the allowlist its ceiling names.

Verification

• Full non-browser suite: 3827 passed, rc=0, on the current base.
• All six ci.yml job invocations run verbatim in a CI-shaped checkout (no bills/): rc=0 each. The ceiling is session-scoped, so each job's own narrow subset had to be checked rather than inferred from a full run.
• Browser tier: 41 passed rc=0 with Chromium, 41 skipped rc=0 without it.
• The new CI job's exact command: 3822 passed, rc=0, 2m40s — it does not land red.
• Suite is now CWD-independent: byte-identical totals from the repository root and from outside it.

Review follow-up: collection-time skips, and #700's lockfile rule

Two review findings, fixed on top of a merge of develop (which now contains #700).

Collection-time skips reached exit 0. The ceiling recorded skips only in pytest_runtest_logreport. A module that skips while importing (pytest.skip(..., allow_module_level=True), or a module-level importorskip) never creates its tests, so they never reach that hook. With one passing module next to it, the session exited 0 while that module's assertions never ran. pytest_collectreport now writes into the same record, so the existing policy covers it unchanged: exact nodeid + reason matching, the allowlists, the two-module browser exemption, and "escalate only a clean run". Under xdist (the default -n auto), the controller re-emits a worker's skipped collection report through this hook.

test_ceiling_fails_session_on_a_collection_time_skip_end_to_end (serial and xdist) runs a child session with an unlisted module that skips during import plus a passing control module. It asserts exit 1 and a banner that names tests/test_unlisted.py with reason: missing fixture. Proven decisive: with the hook body disabled, both cases fail with expected exit 1, got 0 (1 passed, 1 skipped), and the existing end-to-end ceiling tests stay green. The new test covers a different hook, so no older test became redundant.

cwd-independence now passes #700's guard. The job now runs uv sync --locked and uv run --locked --project "$GITHUB_WORKSPACE" pytest "$GITHUB_WORKSPACE/tests" -m "not browser" -q. It still runs from outside the checkout and still covers every non-browser tier. After the merge, test_ci_workflow.py went from 1 failed (exactly these two commands) to 33 passed.

Local verification of this revision (GitHub checks are reported separately on this PR):

Gate	Result
ruff check . / ruff format --check .	clean
pytest -m "not slow and not browser"	2089 passed, 2 skipped, 15 xfailed, rc=0
pytest -m browser --run-browser	41 passed, rc=0
pytest -m browser, Chromium unavailable	41 skipped, rc=0 (exemption intact)
pytest -m slow --deselect tests/test_govinfo_corpus_parity.py	1809 passed, 30 skipped, rc=0
cwd-independence command verbatim, CI-shaped clone (no bills/, CI=true), cwd outside it	3894 passed, 37 skipped, 15 xfailed, rc=0

The last run collects every module, so it is also the check that the new hook flags no collection-time skip anywhere in the suite today.

Known limitation, not addressed here: an allowlist key for a whole module (no ::) would match exactly. However, test_every_allowlist_key_names_a_test_that_still_exists would report that key as stale, because it looks for a def. No such declaration exists or is needed today. If one is ever needed, that guard has to accept module-level keys.

Review notes

test_ci_workflow.py's aggregator and leaf-job rosters are widened to eight by hand. Those are ceilings of the shape the audit argues for — literals inside an assertion — so adding a job is deliberately an edit to a test.

One thing to know for later: #720's G2 exemption ceiling would retire if the static scan were ever narrowed to scripts/ — no fetcher or test file would be scanned, so the exemption sets it bounds would cease to exist. Whether that narrowing should happen is unresolved, so the ceiling is not written as a temporary measure: it is the right bound for the scan as it stands.

This PR is not a step toward narrowing the static scan. Whether the scan could shrink is unresolved and out of scope: file-level coverage does not establish that a file's path-bearing lines ran, and a skip ceiling sees only skips that actually occur, so neither supports replacing the scan. The ceiling widening and this CI job stand on their own — the ceiling closes a hole that exists regardless, and the job tests a property the regex only approximates.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NwVE7osYo39KaUe7CEitFJ