Guest provider: oversized RSA-PSS salt length traps on wasm32

[lannbot]

Reject impossible RSA-PSS salt lengths without trapping on wasm32

Review baseline: 75aee5b (v0.6.1).

Finding

PSS imports at rust/core/src/sig.rs:317-339 accept every u32 salt length. Verification at lines 472-474 passes it through pss_scheme (lines 529-535), which casts the value to usize without checking the modulus-dependent upper bound. The guest calls this core method directly at rust/guest-provider/src/provider.rs:1685-1687.

With SHA-256 and salt-length = u32::MAX, pinned rsa 0.9.10 overflows h_len + s_len + 2 on wasm32 in src/algorithms/pss.rs:187. Release arithmetic wraps, so the initial impossible-length check is bypassed. Verification of a valid published signature then panics on rest[0] at dependency line 227, aborting the wasm execution instead of returning authentication-failed.

This is an input-triggered availability/correctness defect. wit/rsa.wit:107-110 specifies that signatures made under a different salt length fail authentication. The current behavior is also target-dependent: 64-bit verification rejects the oversized salt without this overflow.

Reproduction

Use the public SPKI, message and signature from conformance/vectors/rsa_pss_2048_sha256_mgf1_32_test.json, tcId 1. Bind SHA-256 with salt length u32::MAX, then verify the published signature.

A focused release wasm32-wasip2 harness using the exact locked dependency and the same RsaPublicKey::verify(Pss::new_with_salt::<Sha256>(...), ...) API reproduces a bounds panic at dependency line 227 and a wasm unreachable trap. The same fixture verifies successfully with salt length 32. Guest reachability and unchecked import were source-traced; this reproduction isolates the dependency call rather than executing a composed guest invocation.

Requested resolution

Check impossible PSS salt lengths with overflow-safe arithmetic before invoking the backend, returning authentication-failed during verification. Add a wasm32 conformance regression for u32::MAX and adjacent arithmetic-boundary values. A native-only unit test cannot detect this target-width-specific failure.