Native Iceberg scan: support abfs/abfss (Azure Data Lake Storage Gen2) locations

[djouallah]

What is the problem the feature request solves?

The native Iceberg scan declines every table whose data files live on Azure Data Lake Storage Gen2, so Comet accelerates nothing for Iceberg on Azure.

CometScanRule.icebergReadableSchemes is Set("file", "s3", "s3a", "gs", "oss"). An Iceberg table at abfss://<container>@<account>.dfs.core.windows.net/... (or OneLake's abfss://<workspace>@onelake.dfs.fabric.microsoft.com/...) is rejected at planning time and the scan runs on the JVM exactly as it would without the plugin. The comment above the set says the omission is deliberate because iceberg-rust's OpenDAL storage factory could not build the scheme.

That no longer looks true. At the iceberg-rust rev Comet pins (665c64e):

• native/Cargo.toml already enables the opendal-azdls feature on iceberg-storage-opendal.
• OpenDalStorageFactory::Azdls exists and builds an OpenDalStorage::Azdls from adls.* properties (adls.account-name, adls.account-key, adls.sas-token, adls.tenant-id, adls.client-id, adls.client-secret, adls.authority-host, adls.connection-string).
• On the JVM side, NativeConfig.scala already maps abfs and abfss to the fs.azure.* / fs.abfs.* / fs.abfss.* Hadoop config surface for the Parquet native scan.

So the backend is compiled in and the config plumbing exists for the other scan path. What is missing is the Iceberg storage factory arm and the key translation.

Observed with Spark 4.1.3, org.apache.datafusion:comet-spark-spark4.1_2.13:1.0.0, Iceberg 1.11.0 (iceberg-spark-runtime-4.1_2.13), a REST catalog (Microsoft OneLake), spark.comet.scan.icebergNative.enabled=true. Plan falls back with the scheme message; spark.comet.enabled=false reads the same table fine through HadoopFileIO + hadoop-azure.

Describe the potential solution

1. native/core/src/execution/operators/iceberg_common.rs::storage_factory_for: add "abfs" | "abfss" => Ok(Arc::new(OpenDalStorageFactory::Azdls)).
2. Translate the Hadoop ABFS keys into iceberg-rust's adls.* keys when building the catalog/FileIO properties, the way s3.* is derived from fs.s3a.* today:
   • fs.azure.account.key.<host> -> adls.account-key (+ adls.account-name from the URI host's first label)
   • fs.azure.account.oauth2.client.id / .client.secret / .client.endpoint -> adls.client-id / adls.client-secret / adls.tenant-id (tenant is the path segment of the endpoint URL) and adls.authority-host
   • fs.azure.sas.fixed.token -> adls.sas-token
3. Add abfs and abfss to icebergReadableSchemes (or, per Iceberg native scan claims schemes it cannot execute; three scheme lists disagree #5541, derive that set from the native factory so the two cannot drift again).

Two things worth getting right while there:

• Custom token providers. A lot of Azure deployments set fs.azure.account.oauth.provider.type to a class (MsiTokenProvider, WorkloadIdentityTokenProvider, CustomTokenProviderAdapter, ...) rather than a client secret. Those cannot be reproduced natively from fs.azure.* keys alone. The S3 path already has a customized_credential_load hook built on the JVM side; the equivalent here would be to obtain the bearer token via the configured Hadoop provider on the JVM and pass it through, or at minimum honour OpenDAL/reqsign's workload-identity federation from AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_FEDERATED_TOKEN_FILE. Comet Native scan in Azure fails with workload identity (ignores ABFS configs and env vars) #4747 was the same gap on the Parquet path.
• Non-core.windows.net hosts. The account should come from the URI's first host label and the endpoint from the full host, so OneLake (onelake.dfs.fabric.microsoft.com), sovereign clouds and Azurite work rather than only <account>.dfs.core.windows.net.

Additional context

Related: #5541 (the three scheme lists disagree, and abfss is one of the schemes that currently passes one check and fails another), #4747 (workload identity ignored on the native Parquet scan).

Happy to test a branch against OneLake with an OIDC-federated identity; we have a CI benchmark that runs Iceberg-on-OneLake through Spark 4.1 + Iceberg 1.11 REST and can report timings with and without Comet.