GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3 - #50814
GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and newest v3#50814raulcd wants to merge 4 commits into
Conversation
|
@github-actions crossbow submit test-hdfs |
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Updates the HDFS versions used in CI/integration testing to newer v2 and v3 releases, aiming to reduce sporadic CI timeouts caused by slow downloads of older HDFS artifacts.
Changes:
- Bump HDFS versions in the CI task matrix from 2.9.2/3.2.1 to 2.10.2/3.5.0
- Update the default HDFS version used by the HDFS-enabled Docker image to 3.5.0
- Update the repo’s
.envdefault HDFS version to 3.5.0
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| dev/tasks/tasks.yml | Updates the HDFS version matrix for CI task generation. |
| ci/docker/conda-python-hdfs.dockerfile | Changes the default HDFS build arg to a newer version. |
| .env | Aligns default HDFS version with the new CI/docker default. |
|
|
|
Revision: a7c35cb Submitted crossbow builds: ursacomputing/crossbow @ actions-f937a83a40
|
|
ok, 3.4.3 worked locally but 3.5.0 doesn't, I'll investigate requirements as it just seems like a JDK version issue. |
|
@github-actions crossbow submit test-hdfs |
|
Revision: 3653540 Submitted crossbow builds: ursacomputing/crossbow @ actions-551ab1d4c1
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
ci/docker/conda-python-hdfs.dockerfile:33
- The dockerfile still defaults to
ARG jdk=11, but this PR bumps the default JDK in.envto 17 andcompose.yamlwires${JDK}into this image’s build args. Keeping a different default in the Dockerfile can be surprising when building it directly (without compose) and may lead to using Java 11 with Hadoop 3.5.0 unintentionally. Consider aligning the Dockerfile default to 17.
# installing libhdfs (JNI)
ARG hdfs=3.5.0
|
@github-actions crossbow submit test-hdfs testspark |
|
Revision: 34191f1 Submitted crossbow builds: ursacomputing/crossbow @ actions-b83945da96
|
Rationale for this change
When the issue was originally opened, the test-conda-python-3.10-hdfs-2.9.2 failed sporadically on a PR when upgrading some dependencies due to time out rebuilding the docker image.
A big chunk of time was spent on the job just downloading HDFS 2.9.2.
HDFS 2.9.2 was released on 2018. We test 2.9.2 and 3.2.1, released on 2019.
What changes are included in this PR?
Bump to newer versions, latests are 3.5.0 (2026) and from the v2 series, 2.10.2 (2022).
Bump default JDK to 17. The only other place where we currently use the JDK version on env was for Spark where we already are manually using
JDK=17on tasks.yml so no real change for any other job.Are these changes tested?
Yes via the archery jobs
Are there any user-facing changes?
No