Repository navigation
Rerun only the failed tests on CI retries - #3178
antonwolfy wants to merge 2 commits into
Conversation
The `nick-fields/retry` steps reran the whole suite on every attempt, so a real failure cost a second full run (up to ~26 min) before the job went red, and a test killed by `pytest-timeout` reran all tensor tests. Each pytest command inside a retry step now passes `--lf`. The first attempt has no recorded failures and runs everything; a retry runs only the tests that failed. An attempt killed by the step timeout records nothing, so the next one runs all tests again. Coverage keeps full reruns, since a partial rerun would drop coverage data.
|
Array API standard conformance tests for dpnp=0.22.0dev1=np2py314h8d9cdd5_6 ran successfully. |
There was a problem hiding this comment.
🟡 Changes recommended
Serial pytest attempts can stop on collection errors, allowing a successful --lf retry to pass CI without running the remaining suite; the affected full attempts need --continue-on-collection-errors.
1 open issue
What changed in this PR
This PR reduces CI retry time by using pytest’s last-failed cache instead of rerunning entire suites, building on the existing tensor-test timeout handling.
Changes:
- Adds
--lfto retry-wrapped pytest commands across package, conda-forge, oneMath, and scheduled workflows. - Documents the intended retry behavior while leaving coverage runs unchanged to preserve full-suite coverage data.
| File | Description |
|---|---|
.github/workflows/cron-run-tests.yaml |
Limits Linux and Windows reruns to previously failed tests. |
.github/workflows/conda-package.yml |
Adds last-failed selection to dpnp and tensor retries on Linux and Windows. |
.github/workflows/conda-build-test.yml |
Adds last-failed selection to conda-forge test retries. |
.github/workflows/check-onemath.yaml |
Adds last-failed selection to dpnp and tensor retries for both oneMath configurations. |
Review effort: Balanced
A collection error interrupts a serial pytest run before any test runs and records only the broken module, so a `--lf` retry ran just that module and could pass without the rest of the suite. With `--continue-on-collection-errors` the first attempt runs the other tests, and the retry covers their failures too. The `-n auto` runs don't need it, since xdist already runs the remaining tests after a collection error.
|
View rendered docs @ https://intelpython.github.io/dpnp/pull/3178/index.html |
|
@antonwolfy |

The test steps wrapped in
nick-fields/retryreran the whole suite on every attempt. A real failure cost a second full run (up to ~26 min for the all-integer-dtypes run) before the job went red, and since #3152 a tensor test killed bypytest-timeoutreruns the whole tensor suite.Each pytest command inside a retry step now passes
--lf:--lfnfdefaults toall).Covered:
conda-package.yml(Linux and Windows, dpnp and tensor tests),conda-build-test.yml(conda-forge),check-onemath.yamland the rerun steps ofcron-run-tests.yaml.generate_coverage.yamlkeeps full reruns, since a partial rerun would replace the coverage data of the full one.Checked locally with pytest 9.2.0, pytest-xdist 3.8.0 and pytest-timeout, on a package run through
--pyargsas in CI, with one test that fails once and one that hangs once:-n 2 --lf --timeout=3 --timeout-method=thread)SIGTERMThis touches the same two tensor test lines of
check-onemath.yamlas #3177, so whichever merges second needs a rebase.