fix(storage): authorize MGF1 digest to prevent INCOMPATIBLE_DEVICE on…3.x

[utkrishtsahu]

Changes

What & why: On newer Android Keystore2 hardware, SecureCredentialsManager can fail immediately after a successful login with CredentialsManagerException.INCOMPATIBLE_DEVICE ("This device is not compatible with the ... class"), blocking token persistence even though authentication succeeded. Reported via react-native-auth0 (which depends on this 3.x line) on Pixel 10 / Galaxy S26 (Android 16+).

Root cause: CryptoUtil encrypts stored credentials with an RSA-OAEP key whose cipher uses the MGF1/SHA-1 digest, but the key was generated without explicitly authorizing an MGF1 digest. Older KeyMint defaulted the MGF1 digest to SHA-1, so it matched. Newer Keystore2 firmware strictly enforces the authorized MGF1 set and its default is vendor-dependent (some default to SHA-256), so the private-key decrypt is rejected with INCOMPATIBLE_MGF_DIGEST ("Incompatible padding mode") and surfaces as INCOMPATIBLE_DEVICE.

Changes (both in CryptoUtil.java, no public API change):

• getRSAKeyEntry() — when generating a new RSA key, explicitly authorize the MGF1 digests (SHA-1 and SHA-256) via KeyGenParameterSpec.Builder#setMgf1Digests, guarded to Build.VERSION_CODES.VANILLA_ICE_CREAM (API 35+, where the API exists). The key no longer depends on the vendor's MGF1 default.
• RSADecrypt() — InvalidKeyException / InvalidAlgorithmParameterException (the MGF1-authorization rejection on an existing, pre-fix key) are now treated as recoverable: delete the stale RSA + AES keys and throw CryptoException so the caller regenerates a correctly-authorized key, instead of the terminal IncompatibleDeviceException. NoSuchAlgorithmException / NoSuchPaddingException remain terminal (genuine device incompatibility). This self-heals users already stuck (one re-login).

No endpoints, classes, or public methods added/removed/deprecated. CryptoUtil is internal; behavior change only.

Testing

CryptoUtil cannot exercise the real Android Keystore under unit tests (Robolectric mocks it), so the hardware behavior was validated on physical devices and the exception-handling logic via unit tests.

• Updated the existing RSADecrypt InvalidKeyException test to assert the new recovery behavior (throws CryptoException, deletes RSA + AES keys) and added an equivalent test for InvalidAlgorithmParameterException.
• Full :auth0:testDebugUnitTest and :auth0:lintDebug pass (JDK 11).
• On-device verification (Pixel 10 / Android 17 and Galaxy S26 Ultra / Android 16): confirmed an MGF1 digest mismatch reproduces the exact INCOMPATIBLE_MGF_DIGEST failure, and that generating the key with setMgf1Digests(SHA-1, SHA-256) + the SDK's MGF1/SHA-1 cipher round-trips successfully.
• Not reproduced end-to-end on hardware: a device whose KeyMint defaults MGF1 to SHA-256 (the customer's failing-on-default case) — our test devices default to SHA-1, so the failure was reproduced by forcing the mismatch. The fix removes reliance on the vendor default in all cases.

Checklist

• I have read the Auth0 general contribution guidelines

• I have read the Auth0 Code of Conduct

• All existing and new tests complete without errors

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c12bb85e-ca2e-423d-9ba9-3c1540ac5baa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}