test: add scenarios.json testing pattern, close claude-dev's installCli gap - #97
Merged
Conversation
Prompted by the vite-plus VP_HOME regression: install.sh's installCli path already degrades gracefully (warns instead of aborting) if the CLI install fails, which is correct — but installCli defaults to false and no CI matrix entry ever set it to true, so that whole code path has never once run in CI. The exact same silent-degrade shape that let vite-plus's own regression through, just on an untested path this time instead of a buggy one. Verified directly: `devcontainer features test --features claude-dev --skip-autogenerated .` runs the real claude.ai installer with installCli:true and installs correctly — /usr/local/bin/claude present and runnable. Not a bug, just previously unverified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… pattern Wires claude-dev's new with_cli scenario into both pr-validation.yml and test.yml as a separate job (matching devcontainers/features' own convention: --skip-scenarios on the regular per-baseImage job, --skip-autogenerated on the scenario job — a scenario defines its own image, so running it once per baseImage matrix entry would be both wrong and wasteful). Documents the pattern in AGENTS.md under "Testing non-default options" so the next default-off, silently-degrading option gets a scenario instead of staying untested indefinitely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
✅ PR Validation Passed
📋 Pipeline Status
🤖 Generated by @helpers4 CI • 2026-09-09 |
…ditions
A new test/<name>/scenarios.json or scenario script doesn't change what
already-published consumers get — it only verifies a non-default option
combination, the same "doesn't touch what ships" reasoning already applied
to README-only changes. The check's own grep was broader than its
documented policy ("test/<name>/test.sh" specifically, per this file's
"Modifying an existing feature" section above), so it flagged claude-dev's
new with_cli scenario as needing a bump. Narrowed it to match: src/<name>/
(minus README) or test/<name>/test.sh specifically trigger the check, not
every file under test/<name>/.
Also drops the version bump and changelog entry this added to claude-dev
for the scenario-only PR — no longer needed now that the check matches its
own policy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CodeQL flagged this workflow for not limiting the default GITHUB_TOKEN permissions. pr-validation.yml already scopes to contents:read + pull-requests:write; test.yml only checks out code and runs the devcontainer CLI locally, so contents:read alone covers it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 task
baxyz
added a commit
that referenced
this pull request
Sep 9, 2026
Found during a final consistency audit after #97: this paragraph still described the pre-fix, broader behavior (any file under test/<name>/ triggers a bump) instead of the corrected one (test/<name>/test.sh specifically — a scenario file doesn't count). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4 tasks
baxyz
added a commit
that referenced
this pull request
Sep 9, 2026
…changes Corrects an over-broad rule from #97: test/<name>/test.sh itself was kept as a bump trigger there (reasoning: "the one file CI runs against every consumer's build"), only exempting scenario files. That's still wrong — test.sh is exactly as invisible to a consumer as a scenario file. A version bump publishes a new GHCR tag and triggers a website docs rebuild; doing that for a change nobody running the feature can see is pure noise, not a safety measure. Narrows the check to src/<name>/ (minus README) only. Nothing under test/<name>/ ever requires a bump anymore. Updated AGENTS.md's "Modifying an existing feature" and "Testing non-default options" sections to state this explicitly, and reverted the version bumps this branch had already added to copilot-dev and essential-dev for test-only changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
baxyz
added a commit
that referenced
this pull request
Sep 9, 2026
…changes Corrects an over-broad rule from #97: test/<name>/test.sh itself was kept as a bump trigger there (reasoning: "the one file CI runs against every consumer's build"), only exempting scenario files. That's still wrong — test.sh is exactly as invisible to a consumer as a scenario file. A version bump publishes a new GHCR tag and triggers a website docs rebuild; doing that for a change nobody running the feature can see is pure noise, not a safety measure. Narrows the check to src/<name>/ (minus README) only. Nothing under test/<name>/ ever requires a bump anymore. Updated AGENTS.md's "Modifying an existing feature" and "Testing non-default options" sections to state this explicitly, and reverted the version bumps this branch had already added to copilot-dev and essential-dev for test-only changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nub,github-dev,cline-dev,claude-dev).nubandgithub-devalready assert the symlink explicitly;cline-devuses a different, unaffected install pattern (npm install -g).claude-devwas the one real gap: itsinstallClioption already degrades gracefully and correctly on failure, but defaults tofalseand had zero CI coverage — that code path has never once run in this repo's CI.test/claude-dev/scenarios.json+with_cli.sh, the devcontainers-cli's own mechanism for testing non-default option combinations (this repo had never used it before).test-scenariosjob into bothpr-validation.ymlandtest.yml, following the same conventiondevcontainers/featuresitself uses (--skip-scenarioson the per-baseImage job,--skip-autogeneratedon the scenario job).AGENTS.mdso the next default-off, silently-degrading option gets a scenario instead of staying untested.Test plan
devcontainer features test --features claude-dev --skip-autogenerated .runs the real installer withinstallCli:trueand installs correctlytest-scenariosjob)