Skip to content

[SPARK-60769][SQL] Read Maven resolution settings from session configuration - #59977

Draft
qianlan717 wants to merge 7 commits into
apache:masterfrom
qianlan717:session-ivy-settings
Draft

qianlan717 wants to merge 7 commits into
apache:masterfrom
qianlan717:session-ivy-settings

Conversation

@qianlan717

@qianlan717 qianlan717 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Read Maven resolution settings from the owning SparkSession for SparkSession.addArtifact and
addArtifacts: spark.jars.ivySettings, spark.jars.ivy, spark.jars.ivyConnectTimeout, and
spark.jars.ivyReadTimeout.

This removes the Maven-specific SparkContext configuration lookups from the driver/server
ArtifactManager while preserving configuration substitution, timeout validation, session-local
caches, and the optional local Ivy repository. Values inherited from the cluster SparkConf remain
available in the session configuration.

Spark Connect routing and JVM-client Ivy resolution are intentionally handled by #59982. This PR
contains no Connect routing configuration or client-resolution changes.

Why are the changes needed?

JIRA: https://issues.apache.org/jira/browse/SPARK-60769

Artifact addition is a session-level API. Maven resolution should therefore honor the
configuration of the SparkSession that owns the artifact manager instead of always consulting the
shared SparkContext configuration. This also lets separate sessions use independent resolution
settings and caches.

Does this PR introduce any user-facing change?

Yes. SparkSession.addArtifact and addArtifacts now read Maven resolution settings from the
owning session. Cluster SparkConf values continue to work because they are inherited by the
session, while session-specific values are honored when the resolver is first used.

This PR does not change whether Spark Connect resolves a dependency on the client or server.

How was this patch tested?

Added ArtifactManagerIvySettingsSuite coverage for inherited and session-specific Ivy settings,
timeout defaults and validation, session-specific local Ivy repositories, and isolated runtime
caches.

Ran:

build/sbt 'sql/testOnly org.apache.spark.sql.artifact.ArtifactManagerIvySettingsSuite'

ArtifactManagerIvySettingsSuite: 7 tests passed. git diff --check, changed-source line-length
checks, and non-ASCII checks also passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-6)

@qianlan717 qianlan717 changed the title [SQL] Read Ivy settings from session configuration in SparkSession.addArtifact [SQL] Read Maven resolution settings from session configuration Oct 31, 2026
@qianlan717
qianlan717 marked this pull request as ready for review October 31, 2026 00:54
@qianlan717 qianlan717 changed the title [SQL] Read Maven resolution settings from session configuration [SPARK-60769][SQL] Read Maven resolution settings from session configuration Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant