Repository navigation
fix(oracles): label the per-law table as first-rejecting-law credit - #22
Merged
Merged
Conversation
The gap report's "Kills by law" table credited each history to the first law that rejected it, since a check returns its first violation, but it read as every law's rejections. A law that a first law shadows looked like a law that rejects nothing (field report V80, yocto-queue). The table is now "First law to reject, by law", and its line says that a later law on the same history is not checked. The classify oracle's expected section text changed first. RED on the parent: "the kills-by-law section does not direct the author to a next step" (10 failures). GREEN: the fast gap oracles pass, 31,249 tests. The maintainer's option (b), a rerun that ignores other laws' verdicts, is deferred: the CLI issues no per-law directive, and a rerun can't judge a shadowed law on the same history without changing the check API. Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: 29d9c38 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Pre-merge checks:
  |
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Title check |  |
This assessment has not completed. The check will be retried on the next pre-merge check run. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled.
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap report's per-law table now says what it counts. It credits each history to the first law that rejected it, so a law with no count is not shown to reject nothing.
Why it matters
A check returns its first violation, so a later law on the same history is not checked. The table was titled "Kills by law", and it read as every law's rejections. On yocto-queue, laws P1 and P2 got no count, and the table made them look like restated laws. Hand mutants showed that both laws are sensitive. This was field report V80.
What changes
Evidence
For the maintainer
The maintainer chose option (b) for F18: rerun the mutants with other laws' verdicts ignored, before a "law kills nothing" directive. This PR defers that option, for two reasons. The CLI issues no per-law "kills nothing" directive, so the rerun has nothing to gate. A rerun also cannot judge a shadowed law on the same history without a change to the check API, which the item ruled out. Please approve the deferral, or pick another design.
This pull request and its description were written by Isaac.