Skip to content

Test the runner images in a dedicated runner-images PR check - #4313

Draft
henrymercer wants to merge 6 commits into
mainfrom
henrymercer/runner-image-coverage
Draft

henrymercer wants to merge 6 commits into
mainfrom
henrymercer/runner-image-coverage

Conversation

@henrymercer

Copy link
Copy Markdown
Contributor

Following on from #4309, this PR ensures that the PR checks cover the images listed in the runner images README.

  • The linux-arm64 check is renamed to runner-images, and now runs on each image listed in the README, apart from the exceptions below. On macOS, it also analyzes Swift. It tests the linked CodeQL version, since other checks cover the supported versions. The Linux Arm64 job previously used nightly-latest because releases before CodeQL 2.27.0 had no Linux Arm64 bundle. The macOS jobs use larger runners to keep them off the critical path.
  • The codeql workflow now has a single matrixed analyze job, replacing analyze-javascript and analyze-other. It runs on ubuntu-26.04 with the default CodeQL bundle, since other checks cover the runner images and CodeQL versions, so the check-codeql-versions job is no longer needed. Both languages now use build-mode: none, and the Actions configuration has moved to .github/codeql/codeql-config-actions.yml.
  • The Deflake workflow now reruns failed runner-images jobs, as it does for the codeql workflow.

This leaves ubuntu-slim and windows-11-arm untested, since ubuntu-slim is intended for lightweight jobs and CodeQL doesn't support Windows on Arm64. The xcode-27 preview image is only covered by the unsupported-macos check.

The Ubuntu 22.04 images have brownouts from 23 March 2027 (actions/runner-images#14254), so we'll need to remove them before then.

I'm investigating some automation to maintain the set of tested runner images as a potential followup PR.

Risk assessment

For internal use only. Please select the risk level of this change:

  • Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

Environments:

  • Testing/None - This change does not impact any CodeQL workflows in production.

How did/will you validate this change?

  • End-to-end tests - I am depending on PR checks (i.e. tests in pr-checks).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

  • Development/testing only - This change cannot cause any failures in production.

How will you know if something goes wrong after this change is released?

  • Other - CI failures.

Are there any special considerations for merging or releasing this change?

  • Special considerations - This PR removes thirteen required checks and adds fourteen, so the required checks need to be updated with pr-checks/sync-checks.ts before it can merge. The release branches will pick up the workflow change in the next release, as they did for Remove macos-13 from codeql workflow #3274. After merging, the previous analyze-javascript and analyze-other configurations will probably need deleting from the code scanning tool status page, since those jobs have been replaced.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Consider adding a changelog entry for this change.
  • Confirm the readme and docs have been updated if necessary.

henrymercer and others added 6 commits November 9, 2026 14:40
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…nked`

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI auto review requested due to automatic review settings November 9, 2026 18:30
@github-actions github-actions Bot added the size/M Should be of average difficulty to review label Nov 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the expanded runner-image coverage and simplified CodeQL workflow.

No open issues 🎉

What changed in this PR

This PR moves runner-image coverage into a dedicated integration check and simplifies the repository’s own CodeQL analysis workflow, following the runner cleanup in #4309.

Changes:

  • Replaces linux-arm64 with a 12-image runner-images check using the linked CodeQL bundle, including Swift analysis on macOS.
  • Consolidates repository analysis into an Actions/JavaScript language matrix using the default bundle and build-mode: none.
  • Enables automatic retries for failed runner-image checks through Deflake.
File Description
pr-checks/​checks/​runner-images.yml Adds cross-platform integration coverage.
pr-checks/​checks/​linux-arm64.yml Removes the superseded Arm64-only check.
.github/​workflows/​deflake.yml Adds the runner-images workflow to automatic retries.
.github/​workflows/​codeql.yml Consolidates analysis jobs and removes bundle-version comparison.
.github/​workflows/​__runner-images.yml Generates the expanded runner-image workflow.
.github/​codeql/​codeql-config-actions.yml Extracts the existing Actions analysis configuration.

Review effort: Balanced (auto)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Should be of average difficulty to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants