Skip to content

Harden Vally pull_request auth path by adding token fallback and source diagnostics - #3294

Draft
Shraddha Jain (shrja-ms) with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-eval-job-failure
Draft

Harden Vally pull_request auth path by adding token fallback and source diagnostics#3294
Shraddha Jain (shrja-ms) with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-eval-job-failure

Conversation

Copilot AI commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Vally evaluations / eval-job (pull_request) was failing because all eval sessions were launched without usable Copilot auth when COPILOT_GITHUB_TOKEN was unavailable (common in forked PR contexts). This caused every stimulus to error before execution.

  • Auth token resolution for PR evals

    • Updated .github/workflows/vally-eval.yml to set COPILOT_GITHUB_TOKEN from secrets.COPILOT_GITHUB_TOKEN with fallback to github.token, so vally can still initialize sessions when repo secrets are not exposed.
  • Failure diagnosability

    • Added a dedicated workflow step before Run vally that emits whether the secret-backed token is present vs fallback path, without exposing token material.
  • Security-conscious implementation detail

    • Presence check is done via an env var (COPILOT_TOKEN_SECRET) in PowerShell, avoiding direct secret interpolation in script text.
env:
  COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_GITHUB_TOKEN || github.token }}

- name: Resolve Copilot auth token source
  shell: pwsh
  env:
    COPILOT_TOKEN_SECRET: ${{ secrets.COPILOT_GITHUB_TOKEN }}
  run: |
    if ($env:COPILOT_TOKEN_SECRET) {
      Write-Host 'Using COPILOT_GITHUB_TOKEN secret for vally auth.'
    }
    else {
      Write-Host '::notice::COPILOT_GITHUB_TOKEN secret is unavailable; falling back to github.token for vally auth.'
    }

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI and others added 2 commits August 17, 2026 09:44
Co-authored-by: shrja-ms <77041475+shrja-ms@users.noreply.github.com>
Co-authored-by: shrja-ms <77041475+shrja-ms@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix failing GitHub Actions job for Vally evaluations Harden Vally pull_request auth path by adding token fallback and source diagnostics Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants