Split the test_invalid_json CI job into its three cases - #3541
Merged
Merged
Conversation
The pattern was the critical path of the Test workflow at 51 minutes, where the next slowest was 10. Of that, 48 minutes was tenacity sleeping off Vuforia 429s in ten-second units and six seconds was test work. Split the single parametrised test into the three cases it conflated: a body which is not JSON, valid JSON which is not an object, and a JSON object which is not UTF-8. Each is now its own ci_pattern, so the critical path becomes the five-value case rather than all seven. Splitting adds patterns past the 100 available credentials files, where the secrets index wraps and two jobs share a database. Order the pattern list so that every entry from index 100 on is one which never parametrises over the Real Vuforia backend, which also fixes the existing overlap between test_target_validators.py and test_query.py. Refs #3540 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
adamtheturtle
temporarily deployed
to
development
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
adamtheturtle
temporarily deployed
to
vuforia
September 4, 2026 18:51 — with
GitHub Actions
Inactive
pylint's wrong-spelling-in-docstring check does not accept "parametrised". Reword rather than add it to the private dictionary, since the rest of the codebase does not use the word in prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #3540.
test_invalid_json.py::TestInvalidJSON::test_invalid_jsonis the critical path of theTestworkflow at 51 minutes, where the next slowest pattern is 10 minutes and 61 of the other patterns finish under 2. Since the workflow is serialised onconcurrency: vuforia_credentials, that one job sets how long the shared lock is held.Measuring the gaps between test results in that job's log:
Six seconds of work, forty-eight minutes of sleeping —
wait_fixed(wait=10)inRETRY_ON_TRANSIENT_VWS_FAILURE, ten seconds per 429.What this changes
The single test parametrised over seven bodies becomes the three cases it was conflating, each its own
ci_pattern:test_not_json—b"a"test_not_an_object—b"[]",b'"hello"',b"5",b"null",b"true"test_not_utf_8— a JSON object encoded as latin-1The shared body moves to a
_assert_body_rejectedhelper, so the assertions are unchanged and stated once. The critical path becomes the five-value case, so that job should fall to roughly 36 minutes and a run from ~52 to ~37.The secrets wrap
Adding patterns pushes past the 100 available credentials files, where
SECRET_INDEX=$((JOB_INDEX % 100))makes two jobs share a database. Rather than add to that, the pattern list is now ordered so every entry from index 100 on is one which never parametrises over the Real Vuforia backend:I confirmed each of those collects zero
Real Vuforiaparametrisations. This also fixes an overlap that exists onmaintoday, wheretest_target_validators.pyat index 100 shares a database withtest_query.py::TestContentTypeat index 0.Caveat on the expected gain
The saving assumes Vuforia rate limits per database. Each job gets its own credentials file and therefore its own database, so if that holds, sharding really does divide the waiting. If the limit is per account, the 429s move rather than disappear. I have not established which it is — #3540 records this, along with the larger point that pacing requests or using exponential backoff would attack the cause rather than divide it across jobs.
Verification
pytest-check-partitionpasses, so the patterns still partition the suite exactly.tests/mock_vws/test_invalid_json.py --skip-real: 160 passed, 80 skipped.pre-commitandpre-pushhooks pass.No behaviour change, so no newsfragment.
🤖 Generated with Claude Code