Repository navigation
Make weak tests fail when their feature breaks - #8
Merged
Merged
Conversation
A review of the existing tests found eight that would still pass with their feature broken. They now check outcomes after each step, with expected values written out. Only tests changed. - layers.test.ts: the jump test writes its offsets out. The settleAll test checks the layers instead of an id that never exists. The in-place edit test checks after each cycle. The fork and drop-version tests check the saved Markdown after undo, and the fork test also after redo. - lib.rs: the save test reads the file back, and expects the literal "changed_on_disk" that src/lib/files.ts matches. - ai.rs: the request body test compares the whole body, with the model id written out. - claude_code.rs: the arguments test, now builds_the_command_line, compares the whole list, in order. - Each new check failed against a planted bug that the old test missed, then passed with the code restored. - Reviewed with the review-changes skill: a task review and the tests check. Their findings are fixed or left as follow-ups.
gnapse
force-pushed
the
tests/tighten-weak-tests
branch
from
October 26, 2026 14:40
640275a to
c7a37f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A review of the existing tests found eight that would still pass with their feature broken. This PR makes each one fail when its feature breaks. Only tests changed. No real bug turned up.
What changed
src/editor/layers.test.tsversionedItems, whichjumpToItemitself calls.findAlt(state, "x"), an id that never exists.src-tauri/src/lib.rs: the save test reads the file back. It expects the literal"changed_on_disk"thatsrc/lib/files.tsmatches, not the code's own constant.src-tauri/src/ai.rs: the request body test compares the whole body: model, system, prompt,max_tokens, effort, and schema.src-tauri/src/claude_code.rs: the arguments test, renamed tobuilds_the_command_line, compares the whole list.Evidence
Each bug was planted in the code. The old test (from
main) and the new one ran against it, then the code was restored.versionedItemsputs alternatives at their end (from: a.to)[9, 0], expected[6, 0]settleAllleaves one-variant alternativeslayersOfstill has alternativescycleAltdoes nothing ((alt.active + n) % n)the strugglse here, expectedthe tension hereactiveto 0Above.\n\nA.\nlayers.test.ts:169)write_textreturns Ok without writing when the file is unchanged"mine", expected"edited"CHANGED_ON_DISKrenamed to"changed-on-disk"Err("changed-on-disk")max_tokensfixed at 4096 / prompt sent as system / schema dropped--system-prompt/--safe-mode/--strict-mcp-configremovedTwo fresh reviewers ran with the
review-changesskill: a task review and the.agents/checks/tests.mdcheck. Both passed the change, and between them they reproduced rows 1–6, 7b, and 8b.pnpm verifypasses.Choices to look at
ai.rswrites the model idclaude-opus-5-5out. ChangingMODELnow means changing this test too. That is on purpose: the API relies on the value.claude_code.rschecks the exact flag list, in order. It catches a flag that was added (say, one that turns tools back on), not only one that was removed. The cost: reordering the flags breaks the test.Follow-ups (not in this PR)
forkVersion,settleAll,removeAltVariant, ordeleteGhosts, thoughCLAUDE.mdasks for one for each command that changes layers. A reviewer plantedaddToHistory.of(false)inforkVersionand insettleAll, and all tests still passed.cycleAltorcycleAtCaretwith -1), which ⌥K uses. AcycleAltthat always steps forward passes every test. A backward test needs three or more variants.