Skip to content

Commit ca87ff9

Browse files
authored
Update copilot test expectations with proper behavior definitions and standardized report templates (#1702)
## Problem The current copilot test prompts lack clear expected behavior definitions for each GitHub Guard mode, leading to inconsistent testing and unclear pass/fail criteria. Test output formatting was also inconsistent across different modes. ## Solution This PR updates `scripts/run_copilot_test.sh` with: ### 1. Clear Expected Behavior Definitions for All 6 Modes: - **all**: all repos and all objects within the repos are accessible (min-integrity == none) - **public**: only public repos and all objects within the public repos are accessible (min-integrity == none) - **owner**: only repos owned by the owner and all objects within the owner's repos are accessible (min-integrity == none) - **repo**: only the single repo and all objects within it are accessible (min-integrity == none) - **prefix**: only repos that match the prefix and all objects within those repos are accessible (min-integrity == none) - **multi**: only repos that match the prefixes and other matching criteria and only merged objects within those repos (min-integrity == merged) ### 2. Standardized Markdown Report Templates Each mode now includes a consistent report template with: - Test configuration summary - Global/User API results table - Repo-scoped API results table - Pass/fail summary with counts - Final assessment section ### 3. Enhanced Validation Criteria - Clear expected vs actual behavior sections - Proper pass/fail conditions for each test scenario - Better distinction between filtering expectations vs repository allowlist behavior ## Benefits - **Consistency**: All test modes now have uniform reporting format - **Clarity**: Clear expectations for what each mode should and shouldn't allow - **Automation**: Standardized templates enable better automated result parsing - **Debugging**: Enhanced validation criteria help identify specific failure points ## Testing The updated test expectations have been verified against the behavior matrix: - Repository access patterns (all/public/owner/repo/prefix/multi scoping) - Integrity level requirements (none vs merged) - Global API behavior consistency - Proper blocking of out-of-scope data ## Files Changed - `guards/github-guard/scripts/run_copilot_test.sh`: Updated all 6 test mode definitions with proper expectations and report templates This addresses the need for clear, consistent test expectations across all GitHub Guard modes.
2 parents cc22a4e + c8b76ee commit ca87ff9

1 file changed

Lines changed: 441 additions & 73 deletions

File tree

0 commit comments

Comments
 (0)