Feat/musllinux support by suyanpanghuang · Pull Request #649 · alibaba/zvec

suyanpanghuang · 2026-08-04T03:04:52Z

Summary

Add musllinux x86_64/arm64 wheel support and the musl libc compatibility fixes required to build and publish them.

musl libc compatibility

musl is not a drop-in replacement for glibc:

Drop glibc-only <execinfo.h> include that breaks the musl build.
Move SIMD kernel buffers off the stack — musl's default 128 KB thread stack (vs glibc's 8 MB) caused SIGSEGV.
Raise the default thread stack to 8 MB via PT_GNU_STACK for executables as defense in depth.
Fix logger singleton lifetime. Static class members duplicated across shared libraries via --whole-archive caused multiple __cxa_atexit registrations; tests that repeatedly register/unregister the logger triggered double-destruction on musl (silently tolerated on glibc). Converted to Meyers singleton to guarantee single init/destroy per process.

DiskAnn fixes

Two latent bugs surfaced during musl validation:

Fix heap over-read in add_vector — brace-init invoked the initializer_list constructor, creating a 2-element vector instead of max_build_degree_ elements.
Fix off-by-one in convert_pivot_data — loop used <= instead of <.

CI/CD

Add 09-musllinux-build.yml with x86_64 and arm64 jobs, wire into 01-ci-pipeline.yml.
arm64 uses docker exec because GitHub JavaScript actions cannot run inside Alpine containers on arm64 runners.
Extend build_wheel.yml to publish musllinux wheels.

Arrow dependency

Restrict the re2 cstdint patch to Linux only.

Wheel repair: avoid patchelf ELF corruption

This is a patchelf bug (patchelf <= 0.18.0). When auditwheel repair bundles libstdc++.so.6 and libgcc_s.so.1, it invokes patchelf to rewrite _zvec.so. patchelf relocates sections while extending the program header table but fails to update R_X86_64_RELATIVE relocation addends. The protobuf protodesc_cold section is moved while its addends still point to stale addresses, so import zvec crashes during protobuf initialization. See NixOS/patchelf#652.

Fix: exclude these libraries from auditwheel bundling on musllinux. They are available via apk add libstdc++; the musllinux build images already include them, and on minimal Alpine the user installs as needed, same as any other C++ musllinux wheel.

[[tool.cibuildwheel.overrides]]
select = "*-musllinux*"
repair-wheel-command = "auditwheel repair --exclude libstdc++.so.6 --exclude libgcc_s.so.1 -w {dest_dir} {wheel}"

CLAassistant · 2026-08-04T04:04:01Z

All committers have signed the CLA.

feihongxu0824 · 2026-08-04T08:35:50Z

+# available on all musllinux targets via the base system (apk add libstdc++).
+[[tool.cibuildwheel.overrides]]
+select = "*-musllinux*"
+repair-wheel-command = "auditwheel repair --exclude libstdc++.so.6 --exclude libgcc_s.so.1 -w {dest_dir} {wheel}"


这是是需要用户在自己环境安装libstdc++吗？另外 ci里面有验证这个repair逻辑吗？

repair逻辑是我直接跑CD的buildtestpypiwheel报错了才用了这个repair。其中的工具链patchelf<= 0.18.0有bug，protobuf在elf里面有自定义的section，只要auditwheel调用patchelf就不行。我加了上面的逻辑就能正常生成whl包，CD里面就能正常生成包，但是依赖libstdc++没有跟whl一起打包了，应该需要用户有相应的包

@Cuiyus 看下这里怎么处理，我看目前c的so包是依赖libstdc++的静态库的

feihongxu0824 · 2026-08-04T09:12:26Z

      const T *feature_ptr =
          reinterpret_cast<const T *>(centroids[idx].feature());
-      for (size_t d = 0; d <= chunk_dims[chunk]; ++d) {
+      for (size_t d = 0; d < chunk_dims[chunk]; ++d) {


改了这个是为啥？也没看到单测来验证

这里我用ASAN检测会越界。
在上下文里我去看了一下，
centroid->set_feature(algorithm.centroids()[i],
chunk_dim * meta_.unit_size());
缓冲区设置的时候只有chunk_dim个元素的内容，这里的下标应该是[0,chunk_dim-1]

@richyreachy 确认一下吧

feihongxu0824 · 2026-08-04T09:30:03Z

-//! Logger
-Logger::Pointer LoggerBroker::logger_(new ConsoleLogger);
+//! Initialize default Console Logger (trivial destructor, no __cxa_atexit)
+namespace {


这么改有点复杂，建议换作：

Logger::Pointer LoggerBroker::MakeDefaultLogger() { return std::make_shared<ConsoleLogger>(); }

然后在logger.h 里面换作：

class LoggerBroker { private: static Logger::Pointer MakeDefaultLogger(); static Logger::Pointer &logger_() { static Logger::Pointer instance = MakeDefaultLogger(); return instance; } };

已经修改初始化方式

feihongxu0824 · 2026-08-04T09:33:07Z

-  uint16_t fixed_buffer_2[MAX_SPARSE_BUFFER_LENGTH];
+  // static thread_local: keeps the 256KB working set off the stack, which
+  // overflows musl's small default thread stack (128KB vs glibc's 8MB).
+  static thread_local uint16_t fixed_buffer_1[MAX_SPARSE_BUFFER_LENGTH];


有评估过改完之后多线程下内存上升的代价吗？

另外看到cmake里面已经调整了stack size？

修改了四个文件，理论上每个线程会上升1.5MB的空间。
开始cmake里面调整栈大小后，只能通过C++测试。
但是musl libc的用户调用python包的时候，已经固定了默认栈大小，不能对python包的使用者假设栈的大小被修改，所以最后还是把栈里面的大数组变成了静态threadlocal变量。
是否需要写个检测，只在musl环境这样修改

我建议加，只在musl环境这样修改

所有static threadlocal修改现在只在musl生效

- Convert Logger/IndexLogger broker statics to Meyers singletons to fix double-destruction crashes: logger.cc is linked into multiple DSOs via --whole-archive, each registering its own __cxa_atexit destructor for the same interposed object - Auto-disable LTO for the Python binding on musl, where GCC LTO conflicts with fortify-headers' extern gnu_inline wrappers - Extend arrow.patch to add missing <cstdint> include for re2 pcre.h

hnsw_rabitq_entity.h has included <execinfo.h> since the module was introduced (alibaba#69) but never calls backtrace(); musl provides no such header, so the first x86_64 Alpine build to compile hnsw_rabitq failed. Verified unused: no backtrace references in sources or git history, and no backtrace symbols in the built binaries.

The x86 sparse inner-product kernels kept two fixed working buffers on the stack per call: 512KB in the fp32 SSE paths and 256KB in the fp16 AVX/AVX512FP16 paths. musl's default pthread stack is 128KB (glibc: 8MB), so any worker thread entering these kernels on Alpine crashed with SIGSEGV (reproduced by C++ TestMultiThread cases and pytest). Make the buffers static thread_local: one lazy per-thread TLS allocation reused across calls, zero hot-path cost (address hoisted out of the loops), and correct regardless of thread stack size - including threads created inside the dlopen'ed Python extension, where musl ignores PT_GNU_STACK. The kernels write the buffers before reading and bound reads by the write cursor, so per-call semantics are unchanged.

Defense in depth for C++ executables (tests, tools, examples): link with -Wl,-z,stack-size=8388608 when the host libc is musl, detected by matching "musl" in ldd --version output (musl deliberately provides no identification macro). musl reads PT_GNU_STACK at program startup and raises the default pthread stack from 128KB to 8MB. Note this does not cover dlopen'ed DSOs such as the Python extension - musl only honors PT_GNU_STACK for binaries loaded at startup - which is why the SIMD kernels avoid large stack buffers at the source level.

…ert_pivot_data

…iner

Exclude libstdc++.so.6 and libgcc_s.so.1 from auditwheel repair so patchelf is not invoked. patchelf <= 0.18.0 moves protodesc_cold without updating R_X86_64_RELATIVE addends, breaking protobuf init. Ref: NixOS/patchelf#652

Rename logger_()/logger_level_() to LoggerInstance()/LoggerLevel() in LoggerBroker and IndexLoggerBroker. The trailing underscore looked like a member variable convention, while these are static functions. PascalCase matches the project's existing Singleton<T>::Instance() style. Also remove obsolete "now a Meyers singleton" comments from the .cc files.

Move the re2 <cstdint> workaround from arrow.patch into a dedicated arrow.alpinelinux.patch and apply it only when building on musl/Alpine. Non-musl Linux builds no longer receive this patch.

On musl, dynamic linking of the C++ runtime forces auditwheel to rewrite the ELF via patchelf so it can vendor libstdc++/libgcc. The patchelf versions in current build environments (0.17.x/0.18.x) corrupt protobuf's protodesc_cold section on musl 1.2.5 x86_64, causing init crashes. Statically link libstdc++ and libgcc for the _zvec Python extension on musl instead. This keeps _zvec self-contained, avoids patchelf entirely, and removes the need for the previous auditwheel --exclude workaround. All C++ symbols remain hidden via exports.map, so the statically linked runtime does not conflict with other Python extensions that dynamically link their own libstdc++ copies. Verified coexistence with numpy, grpcio, pandas, and scipy in python:3.10-alpine.

… stack-size hack Move musl detection to the root CMakeLists.txt and expose it as a ZVEC_ON_MUSL cache variable. Propagate this macro to zvec_ailego so the large sparse-vector fixed buffers are only allocated as static thread_local on musl; on glibc they revert to ordinary stack arrays. Also revert the earlier PT_GNU_STACK -z stack-size=8M linker flag: it only helps executables loaded at startup and is ignored for dlopen'ed Python extensions, so it did not solve the actual problem. Finally drop the pinned-image comment in pyproject.toml and keep the auditwheel --exclude removal from the previous commit.

…ove stale comment Replace the eager DefaultLoggerInitializer static objects in ailego and index_logger with Meyers singletons that create the default ConsoleLogger lazily on first access. This avoids static initialization order issues when shared objects are loaded dynamically. Also remove an outdated explanatory comment about brace-init in DiskAnnBuilderEntity::add_vector; the code uses parentheses and the warning is no longer needed.

ZVEC_ON_MUSL is now detected once in the root CMakeLists.txt, so remove the duplicate ldd-based detection in src/binding/python/CMakeLists.txt. Also add missing trailing newlines to logger.cc and index_logger.cc.

suyanpanghuang requested review from Cuiyus, chinaux, feihongxu0824, iaojnh and richyreachy as code owners August 4, 2026 03:04

github-actions Bot assigned suyanpanghuang Aug 4, 2026

richyreachy reviewed Aug 4, 2026

View reviewed changes

Comment thread CMakeLists.txt Outdated

richyreachy reviewed Aug 4, 2026

View reviewed changes

Comment thread src/binding/python/CMakeLists.txt

richyreachy reviewed Aug 4, 2026

View reviewed changes

Comment thread src/include/zvec/core/framework/index_logger.h Outdated

richyreachy reviewed Aug 4, 2026

View reviewed changes

Comment thread thirdparty/arrow/arrow.patch Outdated

feihongxu0824 reviewed Aug 4, 2026

View reviewed changes

JalinWang reviewed Aug 4, 2026

View reviewed changes

Comment thread src/core/algorithm/diskann/diskann_builder_entity.cc Outdated

suyanpanghuang requested a review from zhourrr as a code owner August 6, 2026 05:42

suyanpanghuang force-pushed the feat/musllinux-support branch from 654b115 to 11200f2 Compare August 6, 2026 06:29

suyanpanghuang added 15 commits August 6, 2026 14:46

fix(diskann): fix heap over-read in add_vector and off-by-one in conv…

54ea825

…ert_pivot_data

ci: add manually triggered musllinux x64 build and test workflow

c9cc4e4

ci: pin CCACHE_DIR so ccache and actions/cache agree inside the conta…

e0476e6

…iner

ci: add musllinux arm64 build and test job

0d3c277

ci: run musllinux build and test in the main CI pipeline

de181aa

build: publish musllinux wheels

b7d168e

fix: use portable sed -i.bak in arrow re2 patch

fe7479f

fix: apply arrow re2 cstdint patch only on Linux

729e9f8

fix(pyproject): avoid patchelf ELF corruption on musllinux

03aa59f

Exclude libstdc++.so.6 and libgcc_s.so.1 from auditwheel repair so patchelf is not invoked. patchelf <= 0.18.0 moves protodesc_cold without updating R_X86_64_RELATIVE addends, breaking protobuf init. Ref: NixOS/patchelf#652

refactor: split Arrow Alpine/musl patch into arrow.alpinelinux.patch

998e9f9

Move the re2 <cstdint> workaround from arrow.patch into a dedicated arrow.alpinelinux.patch and apply it only when building on musl/Alpine. Non-musl Linux builds no longer receive this patch.

suyanpanghuang added 4 commits August 6, 2026 14:46

build(python): remove redundant musl detection and add trailing newlines

6ec84f1

ZVEC_ON_MUSL is now detected once in the root CMakeLists.txt, so remove the duplicate ldd-based detection in src/binding/python/CMakeLists.txt. Also add missing trailing newlines to logger.cc and index_logger.cc.

suyanpanghuang force-pushed the feat/musllinux-support branch from 11200f2 to 6ec84f1 Compare August 6, 2026 07:15

suyanpanghuang linked an issue Aug 7, 2026 that may be closed by this pull request

[Feature]: Support musl libc / Alpine Linux #648

Open

Uh oh!

Conversation

suyanpanghuang commented Aug 4, 2026

Summary

musl libc compatibility

DiskAnn fixes

CI/CD

Arrow dependency

Wheel repair: avoid patchelf ELF corruption

Uh oh!

CLAassistant commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CLAassistant commented Aug 4, 2026 •

edited

Loading