Commit ca87ff9
authored
Update copilot test expectations with proper behavior definitions and standardized report templates (#1702)
## Problem
The current copilot test prompts lack clear expected behavior
definitions for each GitHub Guard mode, leading to inconsistent testing
and unclear pass/fail criteria. Test output formatting was also
inconsistent across different modes.
## Solution
This PR updates `scripts/run_copilot_test.sh` with:
### 1. Clear Expected Behavior Definitions for All 6 Modes:
- **all**: all repos and all objects within the repos are accessible
(min-integrity == none)
- **public**: only public repos and all objects within the public repos
are accessible (min-integrity == none)
- **owner**: only repos owned by the owner and all objects within the
owner's repos are accessible (min-integrity == none)
- **repo**: only the single repo and all objects within it are
accessible (min-integrity == none)
- **prefix**: only repos that match the prefix and all objects within
those repos are accessible (min-integrity == none)
- **multi**: only repos that match the prefixes and other matching
criteria and only merged objects within those repos (min-integrity ==
merged)
### 2. Standardized Markdown Report Templates
Each mode now includes a consistent report template with:
- Test configuration summary
- Global/User API results table
- Repo-scoped API results table
- Pass/fail summary with counts
- Final assessment section
### 3. Enhanced Validation Criteria
- Clear expected vs actual behavior sections
- Proper pass/fail conditions for each test scenario
- Better distinction between filtering expectations vs repository
allowlist behavior
## Benefits
- **Consistency**: All test modes now have uniform reporting format
- **Clarity**: Clear expectations for what each mode should and
shouldn't allow
- **Automation**: Standardized templates enable better automated result
parsing
- **Debugging**: Enhanced validation criteria help identify specific
failure points
## Testing
The updated test expectations have been verified against the behavior
matrix:
- Repository access patterns (all/public/owner/repo/prefix/multi
scoping)
- Integrity level requirements (none vs merged)
- Global API behavior consistency
- Proper blocking of out-of-scope data
## Files Changed
- `guards/github-guard/scripts/run_copilot_test.sh`: Updated all 6 test
mode definitions with proper expectations and report templates
This addresses the need for clear, consistent test expectations across
all GitHub Guard modes.1 file changed
Lines changed: 441 additions & 73 deletions
0 commit comments