Repository navigation
fix(oracles): working replay -t, one-line gap lines, typed operands, and a two-step not-observed directive - #20
Merged
Merged
Conversation
…lines, typed operands, F22's not-observed directive - The baseline's replay -t pattern is the test's suite and own names, regex-escaped, joined for Vitest 4 and 5, anchored, and single-quoted; a file with no failing test replays whole. Classify law 9 checks it over generated names; the end-to-end oracle runs the printed command. - A gap line's code and intent collapse whitespace and cut at 60 characters; results.json keeps the full text (classify law 8, with authored mutants now generated). - not-infected operands print as typed code spans, never two values alike (classify law 7). - not-observed also directs a history where the mutated code computes something different (F22). Co-authored-by: Isaac <no-reply@databricks.com>
…erand texts, intents as code - Control characters in a test's name or failure print as \\uXXXX escapes in the baseline (where the replay pattern still selects the test) and in messages; classify law 9 owes one-line baseline lines. - The gap line's and the capture's cuts count code points, so an astral-plane character is never split. - Classify law 7 owes each operand text equal to the model's own rendering, not only distinct texts. - An authored intent prints as a code span, so its words are not read as the report's. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: 138c63e The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
commit: |
KyleAMathews
added a commit
that referenced
this pull request
Nov 11, 2026
…le format, and no listener warning with many contexts (#26) * feat(oracles): --config selects the Vitest config; .mts wrappers; listener accounting per warm context - --config <file>, relative to --package, selects the package's Vitest config; the baseline names the config loaded, or says it found none; a --config that is not a file of the package refuses the run (F12). - The runner's wrapper configs are .mts, so Vite loads them and the package's config as ES modules in a package of any module type (F13). - The process's listener limit grows by one per open warm context and shrinks when it closes, so eleven or more contexts print no MaxListenersExceededWarning (F15). - The end-to-end oracle generates config place, module type, a top-level-await config, and eleven contexts; the static oracle generates a missing or directory --config. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracles): a failing owner's replay names --config; bound e2e shrinking - A failing owner's replay command names the --config file the run was given, so it runs the failing test under that config even when Vitest would discover another one (F12 with #20's replay law). - The end-to-end oracle writes a discovered config that includes no test beside every config that only --config names, and a constructed history replays a failing owner there. - The end-to-end oracle's counterexamples shrink through at most six histories, so a campaign reports its first law verdict within its 900 s limit instead of a timeout. - The four config states were listed twice in the required states. Co-authored-by: Isaac <no-reply@databricks.com> * fix(oracles): wrapper format follows the package config; config line once loaded - The runner's wrapper config has the module format that Vite gives the package's own config: .mts for an ES module config, .cts for a CommonJS one, so a top-level await and require both load as under Vitest. - The run names its config once its contexts loaded it, so a refusal by the analysis or the baseline names it too. - A context that fails to open gives back its listener allowance, and one whose standalone or init step rejects is closed. - The end-to-end oracle generates config extensions .ts, .mts, .cts, .js, .mjs, and .cjs in either module type, CommonJS configs that call require and require.resolve, a law 9 edit that switches --config, and owes the config line on every run whose config loaded. Co-authored-by: Isaac <no-reply@databricks.com> --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes four fixes to the
oracle-gapsoutput, and an extended oracle drives each one. One fixes the F22 directive. The others are #12 leftovers: the baseline replay command, one-line gap lines, and typed operands.Changes
not-observedsurvivor had an infection, but no recorded observation changed. The records cannot separate an unobserved change from a forced branch that matched the original (klona's M6). The directive now has two steps: "Observe the value this code changes in<test>, and assert on it. Where the mutated code computes the same result as the original, add a history where it computes something different."-t "<a > b > test>"in JSON quotes. Vitest 4 joins names with a space, so-t "t0 > t0"selected nothing. Regex syntax in a name broke the pattern, and$or a backtick broke the shell. A load failure printed-t undefined. Now the pattern holds the suite and test names, regex-escaped and joined with(?: | > )for Vitest 4 and 5. It has anchors and POSIX single quotes. A load failure replays the whole file with no-t.results.jsonkeeps the full text.classify.mjsprints each operand as a code span. Before,1and"1",0and-0,1nand1, and long strings with one prefix printed alike. Also,"or?"read as an exit.The oracles
The classify oracle (
oracle-gaps.report-classify) gets three laws.not-infectedmessage names, in order, the operand pairs that the model saw. No text stands for two values that the model tells apart. The model now counts a non-primitive operand as a change, as the capture does.pass, orfailwith a replay command. A small POSIX word reader splits the command and refuses expansions. The command adds-t <pattern>only when a test failed. Under both Vitest joins, the pattern must select exactly the failing test.The grammar adds these inputs:
-0, numeric strings,"NaN","1n",undefined,null,NaN, bigints, booleans, objects,or?, and two long strings with one prefix.find.(a|b),$HOME,it's,q"t,[t0]+,^x$,a.b, and a backslash. A name can end with another name.The end-to-end fixture puts the failing test in the suite
broken (b.*) $x it's, next to a top-levelfixed seed 999. The driver runs the printed replay in a realshand observes which tests ran.Evidence
@vitest/runner4.1 and the Vitest rerun command, and POSIX.1-2017 sections 2.2 and 2.6.refused brokenOwner.names no readable operand pairs, owed 1(not-observed) has message "…", which is not its directivewhich does not show its change on one line, shortened to 60 charactersthe replay pattern "t0 > t0" … runs tests [] when Vitest joins names with " "broken (b.*) $x it's > fixed seed 999ran[].-0as0, no digest.-t ''on a load failure.N,undefinedasvoid.shown. No law judges that encoding.>, and they are distinct in a file.unit()) records names for the baseline.Review round 1
Each fix first extended the classify oracle, which failed on the round-0 code.
\uXXXX, which still selects the same test in a pattern.operandOf), not only differ from the others.GREEN after the round: 31249 tests passed, and the end-to-end seed passed in 209 s.
Decisions for the maintainer
or, which the no-exit law forbids. So the directive has two steps. It says "mutated code" because the class also covers comparisons and authored mutants.L?exemption. The exit check now readsL?as a placeholder for a gap with no line. It is safe because it matches onlyL?between spaces. Any other?still counts as an exit.Changeset
.changeset/gap-report-output-fixes.md(patch,@tanstack/oracles).This pull request and its description were written by Isaac.