Skip to content

semgrep: add build-semgrep.yml for riscv64 wheels - #452

Merged
luhenry merged 6 commits into
mainfrom
semgrep
Aug 27, 2026
Merged

semgrep: add build-semgrep.yml for riscv64 wheels#452
luhenry merged 6 commits into
mainfrom
semgrep

Conversation

@luhenry

@luhenry luhenry commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Compiles semgrep-core, semgrep's OCaml analysis engine, and ships it inside the CLI wheel. Upstream publishes a wheel per platform but none for riscv64.

Mirrors upstream's build-test-manylinux-x86.jsonnet.

Matrix: one interpreter-agnostic py3 wheel, as upstream ships per platform.

Differs from upstream

  • Built in riscv64/ubuntu:24.04; Rocky 10 riscv64 has no libunwind-devel or patchelf.
  • opam lockfile derived from upstream's amd64 one, which differs only in the host-arch package.

Testing

  • upstream's e2e rules and targets, run through the CLI; upstream's wheel job runs no tests.

License: Wheel bundles elfutils' libdw/libelf, libgmp and libstdc++/libgcc_s, which are (L)GPL, so their sources are attached to the release.

Patches

  • 0001-setup.py-tag-riscv64-linux-wheels-as-manylinux_2_39.patch - To upstream [needs a riscv64 core build job upstream does not have]. Without it setup.py's platform table raises KeyError and no wheel is built. riscv64-only.

One wheel built; semgrep --version and semgrep-core -version both answer 1.174.0, and the e2e scan reports 2 findings across Python and JavaScript.

luhenry added 3 commits August 26, 2026 19:51
@luhenry
semgrep: add build-semgrep.yml for riscv64 wheels
0025d45 
semgrep's wheel is pysemgrep plus a ~260 MB `semgrep-core` binary compiled
from the project's own OCaml tree (300k lines of OCaml, 36 tree-sitter
grammars), so the port is a full source build rather than a cibuildwheel
run. The workflow mirrors the `semgrep-core-container` stage of upstream's
Dockerfile and their `build-test-core-x86` workflow: bootstrap opam, create
the 5.3.0 switch from their pinned opam-repository, `make
install-deps-for-semgrep-core`, `make core`, then upstream's
`scripts/build-wheels.sh` recipe for the wheel itself.

Three riscv64-specific deviations, each commented in the file:

- the build container is `riscv64/ubuntu:24.04` rather than the
  manylinux_riscv64 image, because Rocky 10 has no `libunwind-devel` on
  riscv64 and semgrep's `obackward` links libunwind. Ubuntu 24.04 is glibc
  2.39, so the wheel is tagged manylinux_2_39_riscv64;
- `scripts/pick-lockfile.sh` knows no riscv64. Upstream's linux lockfiles
  differ in exactly one line (`host-arch-x86_64` vs `host-arch-arm64`), and
  the pinned opam-repository has `host-arch-riscv64`, so the lockfile is
  derived from the amd64 one with sed;
- `cli/setup.py`'s `plat_libc_to_tag` raises KeyError on `linux_riscv64`;
  patched to add the entry (patches/semgrep/1.174.0/).

The shared libraries semgrep-core links are copied next to the binary with
an `$ORIGIN/libs` rpath, reproducing the `semgrep/bin/libs/` layout of
upstream's published manylinux wheels. Several of them are (L)GPL
(elfutils, gmp, libstdc++), so their Debian sources are collected from the
same image and published via `publish-wheels`' gpl-sources inputs.

Locally validated under QEMU riscv64: the apt dependency list installs, the
opam 2.5.2 riscv64 binary runs, `opam init --bare --disable-sandboxing`
succeeds, and both `ocaml-variants.5.3.0+options` and `host-arch-riscv64`
resolve against the opam-repository commit semgrep pins.
@luhenry
semgrep: install patch in the build container
f777541 
libs/ocaml-tree-sitter-core/scripts/download-tree-sitter applies a Makefile
backport to the tree-sitter 0.22.6 tarball, and 'patch' is not part of
build-essential.
@luhenry
semgrep: keep the wheel and the GPL-source collection off the build's…
b05ad75 
… critical path

Upload the wheel as soon as it is built and move the source-package
collection into its own step, so neither a failing test nor a missing
deb-src entry throws away a multi-hour build.
@luhenry

luhenry commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

Dry-ran the test phase against upstream's released 1.174.0 wheel on macOS/arm64 (gotcha 52), so the post-build test step is not a guess: semgrep --version -> 1.174.0, the bundled binary answers semgrep-core version: 1.174.0 (33bca4bcee), and the exact semgrep scan --metrics=off --disable-version-check --config rules/eqeq-python.yaml --config rules/eqeq-basic.yaml --json ... targets/basic/stupid.py targets/basic/stupid.js invocation reports Ran 3 rules on 2 files: 2 findings with Parsed lines: ~100.0%. The assertion script the workflow runs on that JSON prints 2 findings over ['js', 'py'] and sees an empty errors list, so the flags, the JSON shape and the two-parser coverage are all confirmed before the riscv64 job ever gets there.

@luhenry

luhenry commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

Status as of handoff: the pull_request run for b05ad75 (https://github.com/riseproject-dev/python-wheels/actions/runs/32996933164) has been sitting in the queue for ~50 minutes waiting on a free ubuntu-24.04-riscv runner and has not started executing, so nothing about the riscv64 build has been proven on real hardware yet — only the local QEMU validation described above. Once it does start, expect a long job: it builds the OCaml compiler twice (upstream 5.3.0 for the switch, then semgrep's fork via the lockfile's pin-depends, which is exactly what upstream's own flow does), ~250 opam packages, the tree-sitter runtime, 36 generated grammar parsers and 300k lines of OCaml. If that is more runner time than you want to spend right now, cancel the run and leave the dispatch for later — the workflow keeps both triggers, so it can be started again from the PR at any time.

luhenry added 3 commits August 27, 2026 02:11
@luhenry
semgrep: upgrade pip before installing the wheel under test
f1a876b 
Ubuntu 24.04 ships pip 24.0, which matches no manylinux_*_riscv64 platform
tag and rejects our own wheel as "not a supported wheel on this platform"
after a two-hour build. Verified under QEMU: pip 24.0 matches 0 riscv64
manylinux tags, pip 26.2.1 matches 27.
@luhenry
semgrep: bundle shared libraries for every binary in bin/
d4eae27 
Only semgrep-core had its libs collected and its rpath set. osemgrep is a
separate binary, so it kept an unresolved libelf.so.1 and died with exit 127
in the test container. Collect the union of every executable's deps and
patchelf all of them.
@luhenry
semgrep: give each bundled library its own $ORIGIN runpath
a094789 
patchelf writes DT_RUNPATH, which the loader only searches for the object
that carries it -- not for that object's own dependencies. semgrep-core
links libdw but not libelf, so libdw's NEEDED libelf.so.1 was looked up in
the default paths only and the wheel failed to start with

  osemgrep: error while loading shared libraries: libelf.so.1

Set $ORIGIN on every library copied into bin/libs/, which is what upstream's
own manylinux wheels carry (their libdw.so.1 has RUNPATH $ORIGIN).

Only semgrep-core is installed into bin/, so drop the loop over it: osemgrep
is that same binary re-executed under a different argv[0].
@luhenry

luhenry commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

The libelf.so.1 failure was DT_RUNPATH semantics, not a missing library: the loader only searches an object's own RUNPATH for that object's direct DT_NEEDED entries, and libelf is reached through libdw. Upstream's manylinux wheels carry RUNPATH $ORIGIN on each bundled library for exactly this; the build now does the same. Verified against the wheel the last run produced, in riscv64/ubuntu:24.04 under QEMU: semgrep --version and semgrep-core -version both answer 1.174.0, and the e2e scan reports 2 findings over ['js', 'py'] with an empty errors list.

@luhenry
luhenry merged commit f479b9c into main Aug 27, 2026
12 checks passed
@luhenry
luhenry deleted the semgrep branch August 27, 2026 16:50
@luhenry luhenry mentioned this pull request Aug 27, 2026
Open
@luhenry luhenry linked an issue Aug 27, 2026 that may be closed by this pull request
semgrep riscv64 support #629
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

semgrep riscv64 support

1 participant

@luhenry