Skip to content

Rerun only the failed tests on CI retries - #3178

Open
antonwolfy wants to merge 2 commits into
masterfrom
retry-last-failed-tests
Open

antonwolfy wants to merge 2 commits into
masterfrom
retry-last-failed-tests

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

The test steps wrapped in nick-fields/retry reran the whole suite on every attempt. A real failure cost a second full run (up to ~26 min for the all-integer-dtypes run) before the job went red, and since #3152 a tensor test killed by pytest-timeout reruns the whole tensor suite.

Each pytest command inside a retry step now passes --lf:

  • The first attempt has no recorded failures, so it runs everything (--lfnf defaults to all).
  • A retry runs only the tests that failed in the previous attempt.
  • An attempt killed by the step timeout writes no cache, so the next attempt runs everything again.

Covered: conda-package.yml (Linux and Windows, dpnp and tensor tests), conda-build-test.yml (conda-forge), check-onemath.yaml and the rerun steps of cron-run-tests.yaml. generate_coverage.yaml keeps full reruns, since a partial rerun would replace the coverage data of the full one.

Checked locally with pytest 9.2.0, pytest-xdist 3.8.0 and pytest-timeout, on a package run through --pyargs as in CI, with one test that fails once and one that hangs once:

Attempt Result
1, no cache (-n 2 --lf --timeout=3 --timeout-method=thread) all 5 tests ran: 3 passed, 2 failed (one in the worker killed by the timeout)
2, same command only the 2 failed tests ran, both passed
3, same command no failures left in the cache, all 5 ran
after a run killed with SIGTERM no cache written, all 5 ran

This touches the same two tensor test lines of check-onemath.yaml as #3177, so whichever merges second needs a rebase.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

The `nick-fields/retry` steps reran the whole suite on every attempt, so a real failure cost a second full run (up to ~26 min) before the job went red, and a test killed by `pytest-timeout` reran all tensor tests.

Each pytest command inside a retry step now passes `--lf`. The first attempt has no recorded failures and runs everything; a retry runs only the tests that failed. An attempt killed by the step timeout records nothing, so the next one runs all tests again. Coverage keeps full reruns, since a partial rerun would drop coverage data.
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev1=np2py314h8d9cdd5_6 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

coveralls commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.397%. remained the same — retry-last-failed-tests into master

@antonwolfy antonwolfy self-assigned this Nov 8, 2026
@antonwolfy
antonwolfy requested a balanced review from Copilot November 8, 2026 19:50
@antonwolfy antonwolfy added this to the 0.22.0 release milestone Nov 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Serial pytest attempts can stop on collection errors, allowing a successful --lf retry to pass CI without running the remaining suite; the affected full attempts need --continue-on-collection-errors.

1 open issue
What changed in this PR

This PR reduces CI retry time by using pytest’s last-failed cache instead of rerunning entire suites, building on the existing tensor-test timeout handling.

Changes:

  • Adds --lf to retry-wrapped pytest commands across package, conda-forge, oneMath, and scheduled workflows.
  • Documents the intended retry behavior while leaving coverage runs unchanged to preserve full-suite coverage data.
File Description
.github/​workflows/​cron-run-tests.yaml Limits Linux and Windows reruns to previously failed tests.
.github/​workflows/​conda-package.yml Adds last-failed selection to dpnp and tensor retries on Linux and Windows.
.github/​workflows/​conda-build-test.yml Adds last-failed selection to conda-forge test retries.
.github/​workflows/​check-onemath.yaml Adds last-failed selection to dpnp and tensor retries for both oneMath configurations.

Review effort: Balanced

Comment thread .github/workflows/conda-package.yml Resolved
A collection error interrupts a serial pytest run before any test runs and records only the broken module, so a `--lf` retry ran just that module and could pass without the rest of the suite. With `--continue-on-collection-errors` the first attempt runs the other tests, and the retry covers their failures too. The `-n auto` runs don't need it, since xdist already runs the remaining tests after a collection error.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The targeted CI retry changes have no active review issues.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3178/index.html

@ndgrigorian

Copy link
Copy Markdown
Collaborator

@antonwolfy
One concern here is that some tests may only fail as part of a full suite run. It's probably fine either way (if this is consistent, we will notice) but it's something worth considering as unexpected by-product (creation of seemingly "flaky" tests)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants