PYTHON-5909 / PYTHON-5979 Add GA support for Queryable Encryption string queries + Add QE prefix+suffix GA and rename API to string - #2981
Conversation
9845688 to
694f2ca
Compare
There was a problem hiding this comment.
Pull request overview
Adds GA support for Queryable Encryption string queries (prefix/suffix/substring) targeting MongoDB 9.0+, including API updates and refreshed prose/integration coverage, and introduces CI/process changes to better manage uv.lock maintenance.
Changes:
- Introduces
Algorithm.STRING,StringOpts, and GAQueryTypevalues (PREFIX,SUFFIX,SUBSTRING) while deprecatingTextOpts/Algorithm.TEXTPREVIEWand keeping preview query types for pre-9.0 servers. - Updates encryption prose/integration tests (sync + async) to cover GA vs preview behavior, plus adds unit tests for the deprecation shims.
- Adds/adjusts dependency-management workflows and docs around
uv.lock(scheduled lockfile update workflow,uv lock --checkin CI, Dependabot tuning).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/test_encryption.py | Sync encryption tests updated for string GA/preview query behavior + deprecation shim tests. |
| test/asynchronous/test_encryption.py | Async encryption tests updated for string GA/preview query behavior + deprecation shim tests. |
| pymongo/encryption_options.py | Adds StringOpts; deprecates TextOpts as a compatibility shim. |
| pymongo/synchronous/encryption.py | Sync encryption API updated to accept string_opts and deprecate text_opts; adds GA string/query enums. |
| pymongo/asynchronous/encryption.py | Async encryption API updated to accept string_opts and deprecate text_opts; adds GA string/query enums. |
| doc/changelog.rst | Documents GA support and the new/Deprecated APIs. |
| pyproject.toml | Adjusts uv dependency constraints to avoid problematic back-solving for boto3 across forks. |
| CONTRIBUTING.md | Updates dependency/lockfile workflow guidance for contributors. |
| .pre-commit-config.yaml | Excludes uv.lock from the large-file pre-commit check. |
| .gitignore | Stops ignoring uv.lock so it can be committed/checked. |
| .github/workflows/uv-lock-update.yml | New scheduled workflow to regularly update uv.lock. |
| .github/workflows/test-python.yml | Adds uv lock --check and removes the custom exclude-newer action usage. |
| .github/dependabot.yml | Disables routine uv version-update PRs (handled by the scheduled uv-lock-update workflow). |
| .github/actions/set-uv-exclude-newer/action.yml | Removes the custom action previously used to set UV_EXCLUDE_NEWER. |
| .evergreen/run-mongodb-aws-ecs-test.sh | Stops deleting uv.lock, aligning with committed lockfile usage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (5)
pymongo/asynchronous/encryption.py:609
- This preview-query note omits the specification's required warning that the feature's security is not guaranteed and that GA may not be backward compatible with preview payloads. Include those caveats so users do not treat
SUFFIXPREVIEWas production-safe merely because it remains available.
.. note:: The preview query types are for experimental workloads only and
are only supported by MongoDB versions before 9.0. Use
:attr:`QueryType.SUFFIX` instead.
pymongo/asynchronous/encryption.py:619
- This preview-query note omits the specification's required warning that the feature's security is not guaranteed and that GA may not be backward compatible with preview payloads. Include those caveats so users do not treat
SUBSTRINGPREVIEWas production-safe merely because it remains available.
.. note:: The preview query types are for experimental workloads only and
are only supported by MongoDB versions before 9.0. Use
:attr:`QueryType.SUBSTRING` instead.
test/asynchronous/test_encryption.py:3363
- The prose-test baseline is libmongocrypt 1.18.1, and the
substringPreviewcases are explicitly required to run at that version. This class-level 1.19.0 gate skips the entire suite before_params()can apply its 1.18.1 requirement, leaving that supported combination untested. Lower the class gate to 1.18.1 and keep the per-case gates for newer query types.
@async_client_context.require_libmongocrypt_min(1, 19, 0)
pymongo/asynchronous/encryption.py:537
- The client-side encryption specification requires drivers to document that
Stringpayloads must be processed by a client configured withAutoEncryptionOpts, withbypass_auto_encryption=False(whilebypass_query_analysismay be true). The new public algorithm currently omits this operational requirement, so users can follow the API docs and send an unusable payload. Add the required usage note to this docstring.
STRING = "String"
"""String.
.. versionadded:: 4.18
pymongo/asynchronous/encryption.py:599
- This preview-query note omits the specification's required warning that the feature's security is not guaranteed and that GA may not be backward compatible with preview payloads. Include those caveats so users do not treat
PREFIXPREVIEWas production-safe merely because it remains available.
This issue also appears in the following locations of the same file:
- line 607
- line 617
.. note:: The preview query types are for experimental workloads only and
are only supported by MongoDB versions before 9.0. Use
:attr:`QueryType.PREFIX` instead.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
dabaa85 to
1da3a3e
Compare
|
Okay, we need two things to close this out:
|
aclark4life
changed the title
…ry type - Re-export the deprecated TextOpts from pymongo.encryption so 'from pymongo.encryption import TextOpts' keeps working, with a regression test. - Replace the hardcoded libmongocrypt version tuples in the prose tests with a single _STRING_QUERY_MIN_LIBMONGOCRYPT table keyed by query type, and gate each case on the query types it exercises. - Correct the changelog: prefix/suffix need libmongocrypt 1.19.0+, substring needs 1.20.0+.
… on 8.0 Servers before 9.0 exercise the preview query types, which need the deprecated 'textPreview' algorithm: 'String' was only added in libmongocrypt 1.19.0. Pick the algorithm from the installed libmongocrypt version, lower the class gate to 1.18.1, and record the 1.19.0 hole where prefixPreview/suffixPreview were removed before being restored in 1.19.1. On EVG, pin MONGODB_VERSION=8.0 tasks to pymongocrypt<1.19 and use the libmongocrypt bundled in that wheel, so the preview path is tested against bindings users can actually install.
MONGOCRYPT-838 switched release-branch builds to a restricted bucket,
So master builds did move to
2. GitHub release assets — e.g. Signed, and what the 1.18.0 changelog explicitly tells drivers to migrate to. Every target is nocrypto except
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (4)
.evergreen/scripts/setup_tests.py:362
- Evergreen exposes the selected server as
VERSION(this script already reads it at line 328);MONGODB_VERSIONis only assigned inside the separaterun_server.pyprocess. Consequently this is false on the 8.x tasks, so they continue installing master instead of the released pre-1.19 binding and do not exercise the preview combination described here. Fall back toVERSIONwhen selecting the dependency.
use_released_pymongocrypt = os.environ.get("MONGODB_VERSION", "").startswith("8.")
pymongo/asynchronous/encryption.py:1035
- This inserts the new parameter into the positional slot formerly occupied by
text_opts. Existing positional callers will therefore bind their old argument tostring_optsand never receive the promised deprecation warning. Preserve the old slot by keepingtext_optsbefore the newly appended parameter.
string_opts: Optional[StringOpts] = None,
text_opts: Optional[StringOpts] = None,
test/asynchronous/test_encryption.py:3435
- On MongoDB 9.0 with libmongocrypt 1.18.x (allowed by the class decorator), this selects
TextPreviewand setup immediately encrypts fixtures for GA collections before any per-test gate runs. The cases error during setup rather than skip. Skip the GA class below the 1.19.0 floor before selecting the algorithm.
self.algorithm = (
Algorithm.STRING if _libmongocrypt_at_least(1, 19, 0) else Algorithm.TEXTPREVIEW
)
test/asynchronous/test_encryption.py:3483
- Setup always performs GA substring encryption, although the advertised 1.19.x configuration supports only prefix and suffix. It will fail here before
_require_query_type("substring")can skip those cases, preventing the valid prefix/suffix tests from running. Only build this fixture for pre-9.0 preview tests or libmongocrypt 1.20+.
string_opts=StringOpts(
case_sensitive=True,
diacritic_sensitive=True,
substring=dict(strMaxLength=10, strMaxQueryLength=6, strMinQueryLength=2),
…gate Three fixes to get the string query prose tests actually running: - setup_tests.py gated the released-pymongocrypt pin on MONGODB_VERSION, which Evergreen never sets (it is only assigned inside the separate run_server.py process). Use VERSION, which is passed to "run tests". - MONGOCRYPT-838 moved the per-variant libmongocrypt release builds to a restricted bucket, leaving master/latest frozen at 1.18.0 and skipping the whole prose suite. Fetch the signed GitHub release assets instead, which are keyed by libc flavor rather than distro. - The prose setup encrypted the substring fixture unconditionally, so on libmongocrypt 1.19.x it errored before the substring cases could skip, taking the prefix and suffix cases with it. Only build the fixture where the query type exists.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (3)
pymongo/asynchronous/encryption.py:1035
text_optspreviously occupied this positional slot. Insertingstring_optsbefore it makes existing positional calls bind their legacy value tostring_opts, silently bypassing the deprecation warning. Keeptext_optsin its original position and append the new parameter so positional compatibility is preserved.
string_opts: Optional[StringOpts] = None,
text_opts: Optional[StringOpts] = None,
.evergreen/scripts/setup_tests.py:392
- On every 8.x task this branch now bypasses
setup_libmongocrypt()and never setsPYMONGOCRYPT_LIB, even when the documentedLIBMONGOCRYPT_URLoverride is supplied. That makes custom libmongocrypt validation jobs silently test the wheel's bundled library instead of the requested build. Only select the released wheel when no URL override is present.
use_released_pymongocrypt = os.environ.get("VERSION", "").startswith("8.")
pymongo/encryption_options.py:414
- Replacing the explicit public constructor with
*args: Any, **kwargs: Anyremoves the argument names from introspection and causes type checkers to accept invalidTextOptsarguments throughout the deprecation period. MirrorStringOpts' constructor signature while emitting the warning to retain the existing API and static validation.
def __init__(self, *args: Any, **kwargs: Any) -> None:
The preview query types need a server that is at least 8.2 and older than 9.0, but ALL_VERSIONS jumps straight from 8.0 to 9.0, so the prose cases skipped on the server version gate no matter which libmongocrypt was installed. Add a dedicated task for the encryption variants.
|
@aclark4life, note, the new pymongocrypt only accepts |
…lled signature pymongocrypt renamed text_opts to string_opts in 1.19, so the hardcoded text_opts kwarg raised TypeError on the master builds the GA query types require, failing all 11 prose cases on every encryption variant. The 8.x preview tasks still pin pymongocrypt < 1.19, which only accepts text_opts, so resolve the name from the installed signature. A version check would not work: master reports 1.19.0.dev0, which sorts below 1.19.0. Also fix the Windows dll chmod, which shelled out to a POSIX chmod that cannot resolve the drive-lettered absolute path get_libmongocrypt_base() returns, aborting setup before any test ran on the win64 variants.
| # unreleased master build. | ||
| # Evergreen exposes the server version as VERSION, not MONGODB_VERSION | ||
| # (which is only set inside the separate run_server.py process). | ||
| use_released_pymongocrypt = os.environ.get("VERSION", "").startswith("8.") |
There was a problem hiding this comment.
This name will be confusing once we release 1.20, perhaps use_pymongocrypt_text_preview?
| # name from the installed signature rather than from a version comparison: | ||
| # the rename landed on master before any release carried it, so a version | ||
| # check would misclassify the master builds the GA query types require. | ||
| _STRING_OPTS_KWARG = ( |
There was a problem hiding this comment.
Using inspect is expensive, we should defer this calculation and not have it in the import path
| if string_opts is not None: | ||
| raise ConfigurationError("Cannot set both string_opts and text_opts") | ||
| warnings.warn( | ||
| "The text_opts parameter is deprecated. Use string_opts instead.", |
There was a problem hiding this comment.
I don't think this warning makes sense. I think we should error if string_opts is supported by pymongocrypt but they pass text_opts, and vice versa.
| class TextOpts(StringOpts): | ||
| """**DEPRECATED** Options to configure encrypted queries using the text algorithm. | ||
|
|
||
| .. note:: ``TextOpts`` is deprecated. Use :class:`StringOpts` instead. |
There was a problem hiding this comment.
I think we should be more explicit and say which versions apply here. Like TextOpts was support in pymongocrypt x.i -> x.j, but removed support in x.k.
- Rename use_released_pymongocrypt to use_pymongocrypt_text_preview, which says what the pin is for rather than which pymongocrypt is newest. - Resolve the pymongocrypt kwarg name lazily in a cached helper and import inspect inside it, keeping both off the import path. - pymongocrypt accepts only one of text_opts/string_opts per release, so raise ConfigurationError when the name the installed binding does not support is passed, instead of silently aliasing text_opts with a DeprecationWarning. The prose tests now pass StringOpts under the resolved name. - Name the pymongocrypt versions in the TextOpts docs and the changelog.
|
All of the scheduled tests are using latest server, can you please schedule some 8.2 variants and make sure the expected tests are run? |
Like d6865ff or something else? |
|
I mean in Evergreen, run the 8.2 encryption tasks specifically |
|
@aclark4life the linked EVG patch build is failing at startup |
The single quotes were passed through to uv verbatim: write_env strips
double quotes but not single ones, and run-tests.sh expands ${UV_ARGS}
unquoted, so bash word-splits without quote removal. uv then failed with
'Failed to parse: `'\''pymongocrypt<1.19'\''`'.
The quotes were unnecessary -- a '<' arriving via variable expansion is
not a redirection operator, since bash recognizes redirections at parse
time, before expansion.
fixed |
EVG:
https://spruce.corp.mongodb.com/version/6a88c18021f67700074dfeadhttps://spruce.corp.mongodb.com/version/6a8f57263d147f0007ebab7a/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESCPYTHON-5909
PYTHON-5979
Changes in this PR
Adds general availability support for Queryable Encryption prefix, suffix, and substring string queries against MongoDB 9.0+, and renames the preview API to its GA form. Support landed in libmongocrypt per query type: prefix and suffix require 1.19.0+, substring requires 1.20.0+.
Public API
pymongo.encryption_options.StringOptsreplacesTextOpts.TextOptsis kept as a deprecated subclass ofStringOptsand emits aDeprecationWarningon construction. It also stays re-exported frompymongo.encryption, sofrom pymongo.encryption import TextOptscontinues to work for the deprecation period.Algorithm.STRINGreplacesAlgorithm.TEXTPREVIEW, which is now deprecated.QueryType.PREFIX,QueryType.SUFFIX, andQueryType.SUBSTRING, backing the$encStrStartsWith,$encStrEndsWith, and$encStrContainsoperators. The existingQueryType.PREFIXPREVIEW/SUFFIXPREVIEW/SUBSTRINGPREVIEWmembers remain for experimental use against servers older than 9.0.ClientEncryption.encrypt()andAsyncClientEncryption.encrypt()gain astring_optsparameter, deprecatingtext_opts.text_optsis not a silent alias: pymongocrypt renamed this parameter in 1.19 and accepts only one of the two names per release, so passing the name the installed pymongocrypt does not support raisesConfigurationErrorrather than failing deeper in the binding. Passing both names also raisesConfigurationError.Internal
_string_opts_kwarg()resolves the name pymongocrypt gives the string index options (text_optsthrough 1.18,string_optsfrom 1.19) by inspecting the installedExplicitEncrypter.encryptsignature rather than comparing versions: the rename landed on master before any release carried it, so a version check would misclassify the master builds the GA query types require. It isfunctools.lru_cached and doesimport inspectinside the function body, keepinginspectoff the module import path._resolve_string_opts()validatesstring_opts/text_optsagainst that resolved name so the sync and asyncencrypt()paths share one code path._STRING_QUERY_MIN_LIBMONGOCRYPTin the test suite declares the minimum libmongocrypt version per query type in one place, so the test gates and the changelog cannot drift apart.CI
mciuploads/.../master/latestfrozen at 1.18.0 and skipped the entire prose suite on every variant.setup_tests.pynow fetcheslibmongocrypt-{target}-{version}.tar.gzfrom the release page, pinned byLIBMONGOCRYPT_VERSION = "1.20.2"(1.20.0 is the floor for substring GA). Release assets are keyed by libc flavor rather than distro, so the old Debian/Ubuntu/RHEL target table collapses to arch plus glibc/musl, which also picks up ppc64le and s390x. Aget_libmongocrypt_base()helper absorbs the layout difference: release archives putlib/at the archive root, while the master builds nested everything undernocrypto/. An explicitLIBMONGOCRYPT_URLstill wins.pymongocrypt<1.19pin now readsVERSION, notMONGODB_VERSION. Evergreen never setsMONGODB_VERSION— it is only assigned in-process byrun_server.py— so the 8.x branch of that gate was dead code and those tasks kept installing the master binding.test-string-query-previewtask on server 8.2. The preview query types need a server at least 8.2 and older than 9.0, butALL_VERSIONSjumps straight from 8.0 to 9.0, so the preview cases skipped on the server version gate no matter which libmongocrypt was installed.create_string_query_preview_tasks()emits one replica-set task pinned to 8.2, selected by the six encryption variants.ALL_VERSIONSis untouched, so the rest of the matrix does not grow.Test Plan
TestStringExplicitEncryptionProseimplements the spec's "String Explicit Encryption" tests, cases 01-11: find by prefix/suffix/substring, the corresponding no-match cases,contentionFactorbeing required, and the new case-insensitive and diacritic-insensitive prefix/suffix/substring cases.require_version_min(8, 2, -1)andrequire_libmongocrypt_min(1, 18, 1), the floor for the preview query types, so both API generations can run. Setup then encrypts withAlgorithm.STRINGwhere it is available and the deprecatedAlgorithm.TEXTPREVIEWbelow 1.19.0. Two helpers gate the individual cases, and both look their requirements up in_STRING_QUERY_MIN_LIBMONGOCRYPTrather than hardcoding them at the call site:_params(kind)returns the(query_type, collection)pair for the cases that run against the GA type on 9.0+ and the preview type on earlier servers, and_require_ga(*query_types)skips the GA-only cases, naming each query type they exercise. A case is skipped below server 9.0, or when libmongocrypt is too old for any query type it uses.TestStringOptsDeprecationcovers the compatibility surface directly:TextOptswarns and remains re-exported frompymongo.encryption; the resolve helper accepts whichever ofstring_opts/text_optsthe installed pymongocrypt supports, raisesConfigurationErrorfor the other one, and rejects both at once; and the resolved kwarg name is asserted against the installed binding's signature, so a mismatch fails without a server rather than only in the prose suite. These need no server or libmongocrypt, so the compatibility shims are covered on every CI task. Verified locally: 12 tests pass across both suites.Checklist
Checklist for Author
Checklist for Reviewer