PYTHON-6167 Split the test-non-standard task family into per-suite task families by blink1073 · Pull Request #3131 · mongodb/mongo-python-driver

blink1073 · 2026-10-16T10:43:41Z

Summary

Splits the shared test-non-standard Evergreen task family into one family per suite, so a task's name and tags identify the suite it runs. The encryption and load balancer matrices now run as separate sync and async tasks.

Before: 1325 tasks across 76 build variants
After: 833 tasks across 63 build variants

Motivation

The test-non-standard family was shared by the encryption, load balancer, doctest, and PyOpenSSL suites, and only the build variant decided which suite a task actually ran. That made failures hard to attribute.

Additionally, we were not testing sync and async separately for encryption or load balancer, and we were only testing load balancer with its own marker. The load balancer tests are meant to run the whole suite.

Changes

Replaces test-non-standard with one family per suite; a task's name identifies its suite, and distro tags decide which variant runs it.
Selects the sync or async half of a suite from a SYNC environment variable.
Builds OpenSSL 3 on RHEL 8 so free-threaded PyOpenSSL cells can run there, using the updated drivers-evergreen-tools submodule.

Testing

python3 -m pytest .evergreen/scripts/test_generate_config.py passes (21 tests), covering the new suite-identity, sync/async-split, and matrix-size invariants.
New tests pass in Evergreen: XXXX Patch URL

…sk families

Copilot

🟡 Changes recommended

Unresolved critical and moderate issues affect shell safety and test correctness.

2 open findings

What changed in this PR

Splits the shared Evergreen non-standard test family into suite-specific families with sync/async selection and updated platform matrices.

Changes:

Generates per-suite task families and selectors.
Adds sync/async filtering and OpenSSL 3 setup.
Regenerates Evergreen variants, tasks, and functions.

File	Summary
`.evergreen/scripts/utils.py`	Adds `--sync`; moderate issue (1 vote) with positional argument handling.
`.evergreen/scripts/test_generate_config.py`	Adds matrix and suite-shape assertions.
`.evergreen/scripts/setup_tests.py`	Adds sync filtering and OpenSSL setup; critical shell-injection issue (2 votes).
`.evergreen/scripts/generate_config.py`	Generates suite-specific tasks; moderate issues (2, 1, and 1 votes) affecting load-balancer splitting, disabled test commands, and compressor propagation.
`.evergreen/generated_configs/variants.yml`	Updates build-variant selectors.
`.evergreen/generated_configs/tasks.yml`	Regenerates suite-specific task matrices.
`.evergreen/generated_configs/functions.yml`	Forwards new task environment variables.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Unresolved critical and moderate findings affect test coverage and task configuration.

2 open findings

2 resolved since last review

🧠 Review effort: Lite

…o-less names across all platforms

Copilot

🔵 Needs a closer look

Three moderate review findings remain unresolved in task generation and configuration.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Allowlist TEST_CRYPT_SHARED for crypt_shared test tasks

.evergreen/scripts/generate_config.py:1615

The crypt_shared tasks now put TEST_CRYPT_SHARED in the run tests command vars, but the environment allowlist still omits it. setup_tests.py reads this value to enable the crypt_shared setup, so the new crypt_shared tasks will run without that configuration and test the regular libmongocrypt path instead. Add TEST_CRYPT_SHARED to this allowlist.

2 resolved since last review

_{Review effort: Lite}

Copilot

🟡 Changes recommended

Address the critical environment-value shell-injection issue and preserve TEST_CRYPT_SHARED for encryption tests.

2 open issues

Shell-escape values written to env.sh · New
Persist TEST_CRYPT_SHARED for encryption tests · New

_{Review effort: Lite}

+    for line in env.splitlines():
+        name, _, value = line.partition("=")
+        if name in ("OPENSSL_DIR", "LD_LIBRARY_PATH"):
+            os.environ[name] = value
+            write_env(name, value)


+        expansions.update(extra_expansions)
+    if crypt_shared:
+        tags.append("crypt-shared")
+        expansions["TEST_CRYPT_SHARED"] = "true"


Copilot

🟡 Changes recommended

Unresolved critical and moderate findings affect shell safety, option parsing, test coverage, and compression execution.

4 open issues

_{Review effort: Lite}

        CRYPT_SHARED_DIR = Path(config["CRYPT_SHARED_LIB_PATH"]).parent.as_posix()
        LOGGER.info("Using crypt_shared_dir %s", CRYPT_SHARED_DIR)
+        # The trailing expansions must stay live; escape the literal prefix.
+        escaped_dir = CRYPT_SHARED_DIR.replace("$", "\\$").replace("`", "\\`")


        value = str(value).replace('"', "")
+        if escape:
+            # Double quotes expand `$` and backticks: keep the value literal.
+            value = value.replace("$", "\\$").replace("`", "\\`")


blink1073 added 2 commits October 16, 2026 05:42

PYTHON-6167 Split the test-non-standard task family into per-suite ta…

53cfac2

…sk families

Address review: consolidate per-suite task families and share task names

e2edd5a

blink1073 requested a lite review from Copilot October 16, 2026 16:51

Copilot started reviewing on behalf of blink1073 October 16, 2026 16:52

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Comment thread .evergreen/scripts/setup_tests.py

Comment thread .evergreen/scripts/generate_config.py

blink1073 added 2 commits October 16, 2026 13:03

Address review: consolidate per-suite task families and share task names

e975c96

trim tests

ebd77de

blink1073 requested a lite review from Copilot October 16, 2026 18:41

Copilot started reviewing on behalf of blink1073 October 16, 2026 18:42

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Comment thread .evergreen/scripts/generate_config.py Outdated

Comment thread .evergreen/scripts/utils.py

Address review: per-version per-suite task families with shared distr…

65326c9

…o-less names across all platforms

blink1073 requested a lite review from Copilot October 16, 2026 22:21

Copilot started reviewing on behalf of blink1073 October 16, 2026 22:22

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Address review: per-version per-suite task families with shared names

18ec45e

blink1073 requested a lite review from Copilot October 16, 2026 22:41

Copilot started reviewing on behalf of blink1073 October 16, 2026 22:41

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Address review: shell-escape env values and persist TEST_CRYPT_SHARED

6008d53

blink1073 requested a lite review from Copilot October 17, 2026 01:18

Copilot started reviewing on behalf of blink1073 October 17, 2026 01:19

View session

Copilot AI reviewed Oct 17, 2026

View reviewed changes

Navigation Menu

Repository navigation

PYTHON-6167 Split the test-non-standard task family into per-suite task families - #3131

PYTHON-6167 Split the test-non-standard task family into per-suite task families#3131
blink1073 wants to merge 7 commits into
mongodb:mainfrom
blink1073:PYTHON-6167

blink1073 commented Oct 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

blink1073 commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

blink1073 commented Oct 16, 2026 •

edited

Loading