The validator has no way to say "this is my bug, not your document's". Every failure it can express is a finding against the schema under test, so an unexpected exception has only two possible fates, and both are wrong:
Why the second is not acceptable either
validate_directory (pipeline.py:649-652) loops every schema and every instance and accumulates into one report. Collect-everything-then-report is the model. Nothing in that loop is guarded, so one unexpected exception on one file discards the verdicts for every other file in the directory, including the ones already computed.
Report.fatal_error exists but is not the home for this. It is documented as "Set only when the run could not start at all" (mcp_server.py:88) and is used for exactly that: not a directory, and a meta-schema that will not load. Reusing it for a mid-run fault would make a partial run indistinguishable from one that never started.
Worth noting the MCP front end is already immune: mcp_server.py:168 and its siblings wrap each call and convert any exception into a fatal_error result, so the server never dies. Only the CLI directory path is exposed. That asymmetry is itself a reason to fix this centrally rather than per call site.
What is needed
A way to report that the validator itself failed on a given target, such that:
- it is attributable: the message says this is a validator fault, not a finding about the document
- it does not abort: the remaining files still run and their verdicts still appear
- it still fails the run, loudly, with a non-zero exit
- it is distinguishable in the machine-readable output, not only in the human-readable one
This is a public interface decision rather than an implementation detail. Check ids are a public interface in this repository (see docs/architecture.md, "Validation subsystem design"), so whatever this reports under becomes one, and the MCP result models change shape with it.
Relationship to #127
#127 narrows 12 broad except Exception clauses. Every narrowing converts a swallowed exception into a propagating one, so #127 creates the need for this rather than merely being blocked by it. Three sites were narrowed in the first pass; roundtrip.py and pipeline.py landed, and predicates.py did not, because it is the one site with a test asserting the catch.
That test, test_a_processor_failure_is_not_downgraded_to_a_coverage_warning, monkeypatches jsonld.expand to raise RuntimeError. Once this exists, the right change is to narrow predicates.py to jsonld.JsonLdError and retarget the test to raise that, since the type is incidental to what it actually pins, which is that a processor failure does not read as a permitted omission. Doing that before there is somewhere for the unexpected case to go would only move the problem.
So: this issue, then the remainder of #127.
The validator has no way to say "this is my bug, not your document's". Every failure it can express is a finding against the schema under test, so an unexpected exception has only two possible fates, and both are wrong:
Why the second is not acceptable either
validate_directory(pipeline.py:649-652) loops every schema and every instance and accumulates into one report. Collect-everything-then-report is the model. Nothing in that loop is guarded, so one unexpected exception on one file discards the verdicts for every other file in the directory, including the ones already computed.Report.fatal_errorexists but is not the home for this. It is documented as "Set only when the run could not start at all" (mcp_server.py:88) and is used for exactly that:not a directory, and a meta-schema that will not load. Reusing it for a mid-run fault would make a partial run indistinguishable from one that never started.Worth noting the MCP front end is already immune:
mcp_server.py:168and its siblings wrap each call and convert any exception into afatal_errorresult, so the server never dies. Only the CLI directory path is exposed. That asymmetry is itself a reason to fix this centrally rather than per call site.What is needed
A way to report that the validator itself failed on a given target, such that:
This is a public interface decision rather than an implementation detail. Check ids are a public interface in this repository (see docs/architecture.md, "Validation subsystem design"), so whatever this reports under becomes one, and the MCP result models change shape with it.
Relationship to #127
#127 narrows 12 broad
except Exceptionclauses. Every narrowing converts a swallowed exception into a propagating one, so #127 creates the need for this rather than merely being blocked by it. Three sites were narrowed in the first pass;roundtrip.pyandpipeline.pylanded, andpredicates.pydid not, because it is the one site with a test asserting the catch.That test,
test_a_processor_failure_is_not_downgraded_to_a_coverage_warning, monkeypatchesjsonld.expandto raiseRuntimeError. Once this exists, the right change is to narrowpredicates.pytojsonld.JsonLdErrorand retarget the test to raise that, since the type is incidental to what it actually pins, which is that a processor failure does not read as a permitted omission. Doing that before there is somewhere for the unexpected case to go would only move the problem.So: this issue, then the remainder of #127.