[Agents Extension] Add test scenarios using interactive test tool - #9342

Draft

Travis Angevine (trangevi) wants to merge 54 commits into

mainfrom

trangevi/test-scenarios

Commits on Jun 2, 2026

Add test scenarios for the cli-interactive-tester tool
Travis Angevine (trangevi)
committed
16db9dd
Some scenario edits
Travis Angevine (trangevi)
committed
4fbaa67

Commits on Jun 3, 2026

Picking up recent tester tool updates
Travis Angevine (trangevi)
committed
61fc535
Some scenario updates
Travis Angevine (trangevi)
committed
596826b

Commits on Jun 4, 2026

Some more fixes to the scenarios
Travis Angevine (trangevi)
committed
71767db
Some more improvements
Travis Angevine (trangevi)
committed
df67a66

Commits on Jun 5, 2026

Remove optimization and evals because I don't understand them yet, will add back later
Travis Angevine (trangevi)
committed
3eac215
Add prompt to readme
Travis Angevine (trangevi)
committed
c18315c
Add parameterization support
Travis Angevine (trangevi)
committed
1d1e5f2
Add tags
Travis Angevine (trangevi)
committed
d3cd9b5
Agents.md update, to direct people to the testing
Travis Angevine (trangevi)
committed
ea2bac4
cspell
Travis Angevine (trangevi)
committed
8a63c1d
PR comments
Travis Angevine (trangevi)
committed
4793e9f

Commits on Jun 8, 2026

Add agent-scenario-tests skill for local PR scenario regression checks

Glenn Harper (glharper)
and
Copilot
committed
b3ce2e3

Commits on Jun 9, 2026

Commits on Jun 10, 2026

Add test scenarios for delete, endpoint show, and code download (#8591)

authored
904defe

Commits on Jun 11, 2026

fix: replace hardcoded fixture fallback with fail-fast :? syntax (#8606)

authored
69538a6

Commits on Jul 13, 2026

Add setup-wsl.sh for native Linux azd builds in WSL

Travis Angevine (trangevi)
and
Copilot
committed
a42a0d6
Update default test model to gpt-5.4-mini

Travis Angevine (trangevi)
and
Copilot
committed
7088035
Add no-retry rule to driving conventions

Travis Angevine (trangevi)
and
Copilot
committed
a674c2d
fix: setup-wsl.sh build from module root and use full commit hash

Travis Angevine (trangevi)
and
Copilot
committed
125c78c
Merge branch 'main' into trangevi/test-scenarios
Travis Angevine (trangevi)
committed
4e64700
Auto-rebuild WSL binaries before every scenario run

Travis Angevine (trangevi)
and
Copilot
committed
2fa6941

Commits on Jul 14, 2026

Strict goal-matching rule for driving conventions

Travis Angevine (trangevi)
and
Copilot
committed
145106f
Merge remote-tracking branch 'origin/main' into trangevi/test-scenarios
Travis Angevine (trangevi)
committed
d8bfc5d
Add timestamp postfix to shared_agent_name for run isolation

Travis Angevine (trangevi)
and
Copilot
committed
66d0eac
Fix scenario issues from run 20260714-100000

Travis Angevine (trangevi)
and
Copilot
committed
7d3e9f1

Commits on Jul 17, 2026

fix: update test scenarios to match current extension API

Travis Angevine (trangevi)
and
Copilot
committed
04d6d1f
fix: address run-20260716 scenario failures

Travis Angevine (trangevi)
and
Copilot
committed
b1143d6
fix: address run-20260717 scenario failures

Travis Angevine (trangevi)
and
Copilot
committed
3e673bf
fix: tighten observation goals for sample-list and optimize

Travis Angevine (trangevi)
and
Copilot
committed
4c0abbf
fix: optimize gating check should only accept SubscriptionNotRegistered

Travis Angevine (trangevi)
and
Copilot
committed
22e67d3

Commits on Jul 22, 2026

docs: address PR review comments for README, profile, and skill refs

Travis Angevine (trangevi)
and
Copilot
committed
39ecddd

Commits on Jul 23, 2026

fix: rewrite setup-wsl.sh to use azd x tooling for reliable dev-build registration

Travis Angevine (trangevi)
and
Copilot
committed
d790f42
fix: harden endpoint-update scenario hooks and goals

Travis Angevine (trangevi)
and
Copilot
committed
8f24eac
fix: replace agent.manifest.yaml references with azure.yaml

Travis Angevine (trangevi)
and
Copilot
committed
56f4913
fix: add MMDDH to cspell word list for timestamp format patterns

Travis Angevine (trangevi)
and
Copilot
committed
ac1289d

Commits on Jul 28, 2026

Restructure scenario files into tier subdirectories

Travis Angevine (trangevi)
and
Copilot
committed
4ff3698

Commits on Jul 30, 2026

Add Tier 1b verify-deploy scenarios, requires field, and pin agent names

Travis Angevine (trangevi)
and
Copilot
committed
34082ed

Commits on Aug 3, 2026

Harden workflow steps and prohibit agent workarounds

Travis Angevine (trangevi)
and
Copilot
committed
359df4e

Commits on Aug 4, 2026

Commits on Aug 6, 2026

Fix Tier 1b scenario prerequisites
Travis Angevine (trangevi)
committed
c9c32c4

Commits on Aug 7, 2026

Fix Tier 1b scenario execution
Travis Angevine (trangevi)
committed
ae79168
Fix Tier 2 monitor and endpoint scenarios
Travis Angevine (trangevi)
committed
9d96c34
Clean up interactive scenarios
Travis Angevine (trangevi)
committed
d840d1d
Separate scenario intent from tester mechanics
Travis Angevine (trangevi)
committed
e824671
Fix interactive scenario expectations
Travis Angevine (trangevi)
committed
2849e96
Complete interactive prompt contracts

Travis Angevine (trangevi)
and
Copilot
committed
a1e474d

Commits on Aug 10, 2026

Clarify multi-select scenario handling

Travis Angevine (trangevi)
and
Copilot
committed
2bd1ebd
Harden scenario invoke contracts

Travis Angevine (trangevi)
and
Copilot
committed
42f180e