Skip to content

feat(experiments): support suite test selectors#1514

Merged
christso merged 1 commit into
mainfrom
feat/av-oyx-experiment-suite-selectors
Jun 25, 2026
Merged

feat(experiments): support suite test selectors#1514
christso merged 1 commit into
mainfrom
feat/av-oyx-experiment-suite-selectors

Conversation

@christso

Copy link
Copy Markdown
Collaborator

Summary

  • add canonical experiment suites[] entries with suite-local select.test_ids[] selectors
  • resolve selected suite tests into experiment run metadata and CLI execution
  • update schemas, docs, examples, and eval-builder guidance to use suites/test_ids only
  • hard-remove prerelease experiment evals surface without compatibility messaging

Verification

  • bun test ./test/evaluation/experiment.test.ts ./test/evaluation/validation/eval-schema-sync.test.ts (packages/core)
  • bunx biome check packages/core/src/evaluation/experiment.ts packages/core/test/evaluation/experiment.test.ts apps/web/src/content/docs/docs/evaluation/experiments.mdx
  • earlier on this branch: bun test apps/cli/test/eval.integration.test.ts -t "runs a native experiment file with suite test selection and run knobs"
  • earlier on this branch: bun run validate:examples
  • earlier on this branch: bun --filter agentv typecheck

@christso christso merged commit 3916c98 into main Jun 25, 2026
6 of 8 checks passed
@christso christso deleted the feat/av-oyx-experiment-suite-selectors branch June 25, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant