Skip to content

Split the test_invalid_json CI job into its three cases - #3541

Merged
adamtheturtle merged 2 commits into
mainfrom
split-invalid-json-ci-job
Sep 5, 2026
Merged

adamtheturtle merged 2 commits into
mainfrom
split-invalid-json-ci-job

Conversation

@adamtheturtle

Copy link
Copy Markdown
Member

Refs #3540.

test_invalid_json.py::TestInvalidJSON::test_invalid_json is the critical path of the Test workflow at 51 minutes, where the next slowest pattern is 10 minutes and 61 of the other patterns finish under 2. Since the workflow is serialised on concurrency: vuforia_credentials, that one job sets how long the shared lock is held.

Measuring the gaps between test results in that job's log:

tests:  71
span:   48.3 min
gaps > 5s:  55 gaps, totalling 48.2 min
gaps <= 5s: 0.1 min

Six seconds of work, forty-eight minutes of sleeping — wait_fixed(wait=10) in RETRY_ON_TRANSIENT_VWS_FAILURE, ten seconds per 429.

What this changes

The single test parametrised over seven bodies becomes the three cases it was conflating, each its own ci_pattern:

  • test_not_jsonb"a"
  • test_not_an_objectb"[]", b'"hello"', b"5", b"null", b"true"
  • test_not_utf_8 — a JSON object encoded as latin-1

The shared body moves to a _assert_body_rejected helper, so the assertions are unchanged and stated once. The critical path becomes the five-value case, so that job should fall to roughly 36 minutes and a run from ~52 to ~37.

The secrets wrap

Adding patterns pushes past the 100 available credentials files, where SECRET_INDEX=$((JOB_INDEX % 100)) makes two jobs share a database. Rather than add to that, the pattern list is now ordered so every entry from index 100 on is one which never parametrises over the Real Vuforia backend:

100 -> tests/mock_vws/test_requests_mock_usage.py
101 -> tests/mock_vws/test_respx_mock_usage.py
102 -> tests/mock_vws/test_flask_app_usage.py
103 -> tests/mock_vws/test_healthcheck.py
104 -> tests/mock_vws/test_docker.py
105 -> README.rst
106 -> docs/

I confirmed each of those collects zero Real Vuforia parametrisations. This also fixes an overlap that exists on main today, where test_target_validators.py at index 100 shares a database with test_query.py::TestContentType at index 0.

Caveat on the expected gain

The saving assumes Vuforia rate limits per database. Each job gets its own credentials file and therefore its own database, so if that holds, sharding really does divide the waiting. If the limit is per account, the 429s move rather than disappear. I have not established which it is — #3540 records this, along with the larger point that pacing requests or using exponential backoff would attack the cause rather than divide it across jobs.

Verification

  • pytest-check-partition passes, so the patterns still partition the suite exactly.
  • tests/mock_vws/test_invalid_json.py --skip-real: 160 passed, 80 skipped.
  • All pre-commit and pre-push hooks pass.

No behaviour change, so no newsfragment.

🤖 Generated with Claude Code

The pattern was the critical path of the Test workflow at 51 minutes,
where the next slowest was 10. Of that, 48 minutes was tenacity sleeping
off Vuforia 429s in ten-second units and six seconds was test work.

Split the single parametrised test into the three cases it conflated: a
body which is not JSON, valid JSON which is not an object, and a JSON
object which is not UTF-8. Each is now its own ci_pattern, so the
critical path becomes the five-value case rather than all seven.

Splitting adds patterns past the 100 available credentials files, where
the secrets index wraps and two jobs share a database. Order the pattern
list so that every entry from index 100 on is one which never
parametrises over the Real Vuforia backend, which also fixes the
existing overlap between test_target_validators.py and test_query.py.

Refs #3540

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pylint's wrong-spelling-in-docstring check does not accept
"parametrised". Reword rather than add it to the private dictionary,
since the rest of the codebase does not use the word in prose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant