feat: 100% Spark-compatible JSON support via codegen dispatcher

[andygrove]

Which issue does this PR close?

Closes #.

Rationale for this change

The native Rust JSON expressions in Comet have known compatibility gaps and feature restrictions. from_json only supports PERMISSIVE mode with simple schemas, to_json does not handle map or array at the top level, and get_json_object differs from Spark on single-quoted JSON and unescaped control characters.

This PR makes the Spark-compatible path the default for the JSON family, and lets the native path opt in for speed where it is correct. Any case the native path does not cover falls through to the compatible path rather than back to Spark. The compatible path routes through the Arrow-direct codegen dispatcher, which runs Spark's own doGenCode inside the Comet pipeline for byte-exact results, at the cost of a JNI roundtrip per batch.

Configs

• spark.comet.exec.json.engine in {java, rust}, default java
  • java (default): routes the JSON expressions through the codegen dispatcher so Spark's own implementation runs inside the Comet pipeline. This rides spark.comet.exec.scalaUDF.codegen.enabled (enabled by default). If that dispatcher is disabled, the operator falls back to Spark.
  • rust: native DataFusion implementation. Faster, but has known compatibility gaps. An expression or input case with no native implementation falls back to the java engine, not to Spark.

What changes are included in this PR?

• Make java the default JSON engine and drop the "experimental" wording from the config doc and the compatibility guide.
• Use the codegen-dispatch serde base for the JSON family. CometGetJsonObject, CometStructsToJson, and CometJsonToStructs extend CometCodegenDispatch and override only to prefer the native rust path when that engine is selected and a native implementation exists for the case. Everything else falls through to the dispatcher. Under rust this covers to_json with options or array/map types and from_json with an unsupported schema.
• Document the model in docs/source/user-guide/latest/compatibility/json.md and wire the page into the compatibility navigation.

json_array_length and json_object_keys are intentionally out of scope. Both are RuntimeReplaceable in Spark 4.x and Catalyst's ReplaceExpressions rewrites them to StaticInvoke before Comet sees the plan, so the classOf[LengthOfJsonArray] / classOf[JsonObjectKeys] registrations never match. Adding support requires recognizing the rewritten StaticInvoke form in Comet's serde dispatch and is left to a follow-up.

How are these changes tested?

• CometJsonExpressionSuite pins engine=rust and covers the native path. The to_json cases the native path cannot handle now run through the codegen dispatcher, and the suite asserts they execute in Comet rather than falling back to Spark.
• CometJsonJvmSuite covers get_json_object, from_json, and a to_json(from_json(...)) round-trip on the java engine.
• CometSqlFileTestSuite covers get_json_object, from_json, and to_json through SQL golden files. The get_json_object and to_json files pin engine=rust.
• CometExpressionSuite to_json tests pass on the default java engine.

This PR was scaffolded with the project's brainstorming, writing-plans, and subagent-driven-development skills.