Skip to content

Skip Vally eval job for fork-originated pull requests - #3301

Draft
Shraddha Jain (shrja-ms) with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-eval-job-failure-again
Draft

Skip Vally eval job for fork-originated pull requests#3301
Shraddha Jain (shrja-ms) with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-eval-job-failure-again

Conversation

Copilot AI commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

The "Vally evaluations / eval-job" workflow was failing for pull requests opened from forked repositories, reporting 45/45 eval failures.

Root cause

GitHub Actions never exposes repository secrets (COPILOT_GITHUB_TOKEN) to workflow runs triggered by pull_request events from a fork. Without the token, vally's copilot-sdk executor can't create a session:

Error: Execution failed: Error: Session was not created with authentication info or custom provider

This was confirmed by comparing against other recent runs of the same workflow: same-repo PR runs either passed or failed on legitimate grader mismatches, while every fork-originated run failed uniformly on this session error regardless of the code changes in the PR.

Fix

  • .github/workflows/vally-eval.yml: added a job-level if condition that skips eval-job when the run is a pull_request from a fork (head.repo.full_name != github.repository), while still running normally for same-repo PRs and manual workflow_dispatch runs.

This avoids surfacing a false failure for a structural GitHub Actions limitation, without exposing secrets to untrusted fork-submitted code.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

Co-authored-by: shrja-ms <77041475+shrja-ms@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix failing GitHub Actions job Vally evaluations / eval-job Skip Vally eval job for fork-originated pull requests Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants