Conversation
luhenry
commented
Aug 25, 2026
ray's C++ core is built with bazel, and neither of the two toolchains upstream's ci/build/build-manylinux-forge.sh downloads exists for riscv64: bazelisk publishes no riscv64 binary (bazel itself only ships linux-x86_64/arm64), and node.js publishes no riscv64 release at all. So the workflow splits upstream's single manylinux script into three jobs: the dashboard (a React app, arch-independent output) is built on x86 with upstream's pinned node; bazel is bootstrapped from the dist archive on the riscv64 runner, inside the same manylinux image the wheels are built in; and the wheel job then follows upstream's per-interpreter loop. Bootstrapping bazel 7.5.0 (ray's .bazelversion) on riscv64 needs one workaround: bazel 7.x pins rules_python 0.33.2, whose PLATFORMS table has no riscv64 entry, so fetching @pythons_hub aborts the bootstrap (bazelbuild/bazel#23018). Upstream fixed riscv64 bootstrapping in bazel 8.2.0 (bazelbuild/bazel#25745); the 7.x backport request (bazelbuild/bazel#26986) is still open.
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
python/setup.py vendors psutil, colorama, aiohttp and idna into the wheel with `pip install --target`, so the build container needs the riscv64 index too, not just the test container.
ray's WORKSPACE registers a python-build-standalone CPython 3.10 for the host platform and load()s it at the top level, so it is fetched before anything is analysed. The rules_python version ray resolves lists no riscv64 platform, so the load fails with "No platform declared for host OS linux on arch riscv64" -- the same trap that stops bazel 7.x from bootstrapping there. Point it at the manylinux image's own /opt/python/cp310-cp310 instead. Keeping 3.10 matters: the pip vendored by that rules_python imports distutils, so pointing it at the interpreter each wheel is built for fails on 3.12+. Checked on aarch64 with a bazel bootstrapped by the same recipe: the patched tree analyses //:gen_ray_pkg (28166 targets configured) in quay.io/pypa/manylinux_2_39_aarch64, where the unpatched tree stops at the @python3_10 load.
rules_java 7.x maps riscv64 (and mips64) to [":include/linux"] rather
than ["include/linux"] in the jni cc_library's `includes`, so the
bootstrap compiles libcpu_profiler.so with
-isystem external/rules_java~/toolchains/:include/linux
and dies on `fatal error: jni_md.h: No such file or directory`. The typo
is fixed in rules_java 8.x and was never backported to the 7.x line that
bazel 7.5.0 pins, so patch it the same way as rules_python, with
--override_module.
abefc31 is not a commit in actions/download-artifact, so the wheel job died in "Set up job" before checking anything out. Use the v8.0.1 sha the rest of the repo pins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review feedback: `docker run ... bash <<'SCRIPT'` instead of writing the script into the workspace first, for the test step too. The smoke test becomes a nested heredoc, so nothing is staged on disk any more — which also means the test container no longer needs the checkout on its working directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The three wheels share one bazel output base so the C++ core is compiled once, but grpc's python_configure repository rule only re-resolves Python.h when one of the vars it declares in `environ` changes — PYTHON3_BIN_PATH, PYTHON3_LIB_PATH, BAZEL_SH. The /usr/local/bin/python3 symlink the loop flips is not one of them, so every _raylet.so would have been compiled against cp312's headers and failed to import on cp313/cp314. ray's own .bazelrc asks for PYTHON3_BIN_PATH by name. Assert _raylet is the compiled module in the smoke test so a regression here is caught as a wrong .so rather than a confusing runtime failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
//:gen_redis_pkg builds redis with BUILD_TLS=yes against an OpenSSL 1.1.1f that
rules_foreign_cc compiles from source, and that needs two things riscv64 does not
have: a rules_perl toolchain (its BUILD registers linux arm64/x86_64/windows and
darwin only, and relocatable-perl publishes no riscv64 build), and an OpenSSL with
a riscv64 target — 1.1.1f's Configurations have none, so `config` falls through to
linux-generic32.
Analysis failed on exactly that: "No matching toolchains found for types
@@rules_perl//:toolchain_type" while resolving @@rules_perl//:current_toolchain.
RAY_BUILD_REDIS=0 is upstream's own knob for this, and the two binaries it drops —
core/src/ray/thirdparty/redis/src/{redis-server,redis-cli} — are referenced only by
ray/_private/test_utils.py's GCS fault-tolerance helpers, not by any runtime path.
Standing up a system-perl toolchain to statically link an EOL 2020 OpenSSL,
configured as 32-bit on a 64-bit target, into a wheel we distribute is the worse
trade.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The wheel job's previous failure ran 3h43m and GitHub then dropped the job
log entirely: `gh api .../jobs/98025351929/logs` answers BlobNotFound, and
the run's log zip contains only the two short jobs. The failure was
undiagnosable, so the same tree had to be rebuilt blind.
bazel's default curses progress output is the bulk of that volume — a line
per action, rewritten continuously for hours. Upstream's setup.py already
exposes BAZEL_ARGS (python/setup.py: `bazel_flags.extend(shlex.split(BAZEL_ARGS))`),
so `--curses=no --show_progress_rate_limit=60` cuts it to roughly one
progress line a minute without touching anything else about the build.
Tee the step to build.log and upload it on failure as well, so a lost job
log can't cost another multi-hour cycle. `set -o pipefail` is load-bearing:
the default `run:` shell is `bash -e {0}` without it, and tee would
otherwise swallow the build's exit status.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RocksDB (the GCS fault-tolerance backend, bazel/rocksdb.BUILD) is built
through rules_foreign_cc's cmake() rule. rules_foreign_cc 0.9.0 downloads a
prebuilt CMake for the platforms Kitware publishes and otherwise falls back
to building CMake from source, which is the path riscv64 takes — and that
bootstrap fails, 2h50m into the build:
+ CXX=/usr/bin/gcc
Error when bootstrapping CMake:
Cannot find a C++ compiler that supports both C++11 and the specified
C++ flags.
rules_foreign_cc derives CXX from bazel's auto-detected cc toolchain, which
names a single driver (`gcc`) for both C and C++, so CMake 3.23.2's bootstrap
probe cannot link a C++ program. Upstream never reaches this code path: every
architecture they build has a prebuilt CMake to download.
The manylinux_2_39_riscv64 image already ships cmake 4.4.2 in /usr/local/bin,
which is on the fixed PATH bazel gives actions under
--incompatible_strict_action_env, so select the preinstalled toolchain rather
than fixing up the bootstrap's compiler. That also drops the from-source CMake
build from the critical path entirely.
Verified on riscv64 under emulation that the image's cmake configures RocksDB
10.6.2 with ray's own cache entries: "Configuring done / Generating done",
so cmake 4.x is not too new for this RocksDB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rules_boost selects Boost.Context's per-ABI assembly with a select() that
has no riscv64 branch and ends in "//conditions:default": [], so @boost//:context
was compiled with none. jump_fcontext, make_fcontext and ontop_fcontext were
left undefined in _raylet.so and the wheel failed on its first import with
OSError: .../ray/_raylet.so: undefined symbol: jump_fcontext
Boost 1.81.0, the version ray pins, already ships the riscv64 sources, so the
patch only has to wire them up. It goes in through ray's existing
thirdparty/patches list, next to the boost-headers.patch it already carries.
Also upload the wheels before the smoke test, so a wheel that fails to import
is still available to inspect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ray2.58.0Builds ray's C++ core — raylet, gcs_server and the
_raylet.soextension — with bazel. Upstream publishes no riscv64 wheel.Mirrors upstream's
build-wheel-manylinux2014.sh, driving the container directly rather than through cibuildwheel.Differs from upstream
rules_pythonandrules_javapatched during that bootstrap; neither has a riscv64 entry.//:gen_redis_pkgdropped; it needs a perl toolchainrules_perlhas no riscv64 target for.Matrix: cp312/cp313/cp314 — upstream publishes no free-threaded wheel.
Testing
ray.init(), a remote task,ray.shutdown(). Upstream's suite needs a cluster.License: ✅
Patches
0001-Build-with-a-manylinux-interpreter-not-a-hermetic-CPython.patch— Inappropriate [riscv64 build environment]. Upstream's hermetic CPython fetch has no riscv64 artifact. riscv64-only.0002-Build-boost.context-s-riscv64-assembly.patch— To upstream [belongs innelhage/rules_boost, which ray pins by commit]. Without it boost.context compiles with no assembly and_raylet.sofails to load onjump_fcontext. riscv64-only.