feat: slim down zvec dynamic libraries - #627
Open
chinaux wants to merge 11 commits into
Open
Conversation
chinaux
requested review from
Cuiyus,
egolearner,
feihongxu0824 and
zhourrr
as code owners
July 28, 2026 11:33
chinaux
force-pushed
the
feat/slim-c-api-library
branch
2 times, most recently
from
July 29, 2026 12:04
b91f370 to
3462df8
Compare
chinaux
force-pushed
the
feat/slim-c-api-library
branch
from
August 10, 2026 12:06
6b1c2a2 to
f7efd7c
Compare
Reduce the FAT C API dylib size by ~37% through a 5-layer strategy with zero functional loss; core search code stays at -O3 for performance: - Compiler dead-code sectioning: -ffunction-sections -fdata-sections - Linker pruning: macOS -dead_strip + exported symbol list; Linux --gc-sections + version script, keeping only zvec_* C API symbols - Third-party feature trimming: Arrow disables Dataset & trims Compute kernels; RocksDB disables trace/iostats/perf contexts - Protobuf lite: proto uses LITE_RUNTIME, link libprotobuf-lite - Size-optimized third-party builds: -Os for RocksDB/glog/antlr4, MinSizeRel for Arrow
Extend the size-reduction work to the C++ all-in-one shared library and the Windows build: - libzvec C++ SDK (macOS/Linux): restrict exported symbols to zvec's own public API (namespace zvec, incl. nested zvec::ailego/core/turbo) via -exported_symbols_list / --version-script + -dead_strip / --gc-sections. Hides ~14.5k third-party symbols; libzvec.dylib 24.41MB -> 19.05MB (-21.9%), total -34.1% vs main baseline. Public C++ headers never expose third-party types, so external C++ consumers are unaffected. - Third-party libs (MSVC): favor small code (/O1 /Ob1) over speed (/O2) to mirror the -Os / MinSizeRel size optimization used on GCC/Clang, including Arrow's Release config.
…rom tests The slim compute patch removed kernels that production code paths rely on: - match_like (LIKE queries) from scalar_string_ascii/utf8 - sort_indices (ORDER BY) from vector_sort/vector_array_sort - make_struct / list_value_length from scalar_nested Restore these kernels in arrow.slim_compute.patch while keeping the rest (aggregates, temporal, round, random, etc.) removed. Arrow Dataset stays disabled: sql_expr_validator_test's dataset member was dead code and is removed; sql_expr_parser_test's scan test is rewritten with arrow::compute::ExecuteScalarExpression, preserving the original intent (verify a parsed expression evaluates correctly on real data) without depending on arrow::dataset. All 153 C++ tests pass locally (write_recovery_test flake passes in isolation).
…script The version script only exported zvec::* function/data symbols. vtable, typeinfo, VTT and guard variable symbols demangle to 'vtable for zvec::...' etc., which the zvec::* pattern does not match, so they were localized by 'local: *'. Consumers linking libzvec.so then failed with 'undefined reference to vtable for zvec::core_interface::HNSWIndexParam'. Add the mangled-name patterns (_ZTVN4zvec*, _ZTIN4zvec*, _ZTSN4zvec*, _ZTTN4zvec*, _ZGVN4zvec*), mirroring what exported_symbols_cpp.txt already does on macOS.
…ibility In linker version scripts, a quoted "zvec::*" entry is a literal name match (not a glob). GNU ld silently tolerates the never-matching entry, but lld (used by the Android NDK) enforces --no-undefined-version and fails with: ld.lld: error: version script assignment of 'global' to symbol 'zvec::*' failed: symbol not defined Keep only the unquoted glob pattern, which matches all demangled zvec:: symbols on both GNU ld and lld.
Introduces a self-contained implementation of the manifest on-disk format,
in preparation for dropping the libprotobuf/protoc dependency:
- pb_wire.h: minimal protobuf wire-format reader/writer (varint, fixed32/64,
length-delimited). Unknown fields are consumed and ignored, preserving
proto3 forward compatibility; malformed input (bad varint, out-of-range
length, groups, field number 0) is reported instead of crashing.
- manifest_enum.h: the on-disk enums, mirroring src/db/proto/zvec.proto with
identical numeric values. The CodeBooks in type_helper.h now speak these
instead of the generated protobuf enums.
- manifest_codec.{h,cc}: encodes/decodes all 16 messages directly between
zvec's C++ types and bytes, without an intermediate message object.
Format compatibility is established by two test suites:
- manifest_codec_test.cc cross-checks against libprotobuf while it is still
available: every message encoded by both implementations must produce
identical bytes, and each side must parse the other's output. This covers
the subtle cases - always-present base/quantizer_param sub-messages, the
absent quantizer_param of HNSW_RABITQ, Vamana's fixed32 alpha, empty
repeated string elements and oneof "last branch wins".
- manifest_codec_golden_test.cc pins protobuf's real output as embedded byte
arrays and does not depend on libprotobuf, so it keeps guarding the format
after the dependency is removed.
src/db/proto/zvec.proto is retained as the authoritative documentation of the
format. No behaviour change yet: Version::Save/Load still use protobuf.
Version::Load/Save no longer build an intermediate proto::Manifest; they read and write the manifest bytes with ManifestCodec instead. The on-disk format is unchanged - manifest_codec_test.cc verifies byte-for-byte equality with the protobuf implementation, and manifest_codec_golden_test.cc pins the format independently. Manifests are a few kilobytes, so the file is read/written in one go rather than streamed. ProtoConverter is intentionally kept for now: it is what the cross-check tests compare against. It is removed together with the protobuf dependency.
The manifest is now read and written entirely by ManifestCodec (added and cross-checked against protobuf in the previous commits), so libprotobuf and protoc are no longer needed. Removed: - thirdparty/protobuf submodule and its build integration - src/db/proto compilation (cc_proto_library, zvec.pb.cc, libprotobuf-lite link and zvec_proto target/deps across src and tests) - ProtoConverter and the cross-check test that depended on libprotobuf - the "build host protoc" stage from the Android and iOS CI workflows and the build_android.sh / build_ios.sh scripts, along with the now-unused GLOBAL_CC_PROTOBUF_PROTOC option. Cross-compiling no longer needs a host protoc, simplifying those pipelines noticeably. Kept: - src/db/proto/zvec.proto as the authoritative documentation of the on-disk manifest format - cc_proto_library in cmake/bazel.cmake as a generic helper (no longer used by zvec itself) The manifest on-disk format is unchanged; manifest_codec_golden_test.cc keeps guarding it. Full C++ test suite (156 tests) passes locally.
zvec.proto was retained as documentation when the protobuf dependency was
dropped, but the manifest format is now fully defined by manifest_codec.{h,cc}
(field numbers) and manifest_enum.h (enum values). Delete the file and the
src/db/proto directory, and repoint the doc comments that referenced it to the
codec instead.
Also drop the stale clang-tidy header cache entries for build/src/db/proto and
the src/db/proto/*.proto hashFiles input, which no longer exist.
meta.h uses std::numeric_limits but never included <limits>; it compiled only because type_helper.h transitively included the generated zvec.pb.h, which brought <limits> along. With protobuf gone the omission surfaced on libstdc++/GCC: meta.h:280: error: 'numeric_limits' is not a member of 'std' libc++ happens to provide it transitively, which is why only the Linux jobs failed. Also make the files touched by the codec change include what they use: <memory>/<string>/<string_view>/<vector>/<cstdint> in manifest_codec.cc and <memory> in version_manager.cc.
chinaux
force-pushed
the
feat/slim-c-api-library
branch
from
August 10, 2026 12:15
f7efd7c to
287ed09
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR substantially reduces the size of zvec's prebuilt dynamic libraries with
zero functional loss and no public API change. Core search code stays at
-O3, so query performance is unaffected.Measured on macOS arm64, Release + strip, both built from the same base commit
(
d15a37e):libzvec_c_api.dylib(C API)libzvec.dylib(C++ SDK)It also removes the protobuf/protoc build dependency entirely while keeping
the on-disk manifest format byte-for-byte compatible.
Motivation
Both dynamic libraries statically bundle heavy third-party dependencies (Arrow,
RocksDB, protobuf, ANTLR4, glog). By default they export all of those
symbols and carry code paths zvec never uses. That inflates binary size, slows
down linking, and pollutes the global symbol namespace for downstream consumers.
protobuf was a particularly poor trade: zvec only used it to persist the
collection manifest, yet it pulled in a code generator (
protoc), a submodule,and a runtime library into every build.
Approach
1. Symbol visibility & dead-code elimination
-ffunction-sections -fdata-sectionsso unreachable code can bepruned at link time.
-exported_symbols_list+-dead_strip--version-script+--gc-sectionslibzvec_c_apiexports only thezvec_*C API.libzvecexports only zvec's own public C++ API (namespacezvec, includingthe nested
zvec::ailego/zvec::core/zvec::turbo/zvec::reranker),hiding ~14.5k third-party symbols (exported symbols: 23,102 → 8,609).
Two follow-up fixes were needed to make the Linux script correct:
vtable/typeinfo/VTT/ guard-variable patterns(
_ZTVN4zvec*,_ZTIN4zvec*, ...). The demangledzvec::*glob does notmatch them, so consumers previously failed with
undefined reference to vtable for zvec::core_interface::HNSWIndexParam."zvec::*"entry: in version scripts a quoted name is aliteral match, not a glob. GNU ld silently ignores the never-matching entry,
but lld (Android NDK) fails under
--no-undefined-version.2. Drop the protobuf / protoc dependency
The manifest is now encoded by a self-contained codec instead of generated
protobuf code:
pb_wire.h— a minimal protobuf wire-format reader/writer (varint,fixed32/64, length-delimited). Unknown fields are consumed and ignored to
preserve proto3 forward compatibility; malformed input (bad varint,
out-of-range length, groups, field number 0) is reported rather than crashing.
manifest_codec.{h,cc}— converts all 16 messages directly between zvec's C++types and bytes, with no intermediate message object.
manifest_enum.h— the on-disk enum values, preserved exactly.The
thirdparty/protobufsubmodule,src/db/proto/zvec.protoand the generatedproto_converterare removed.Format compatibility is enforced by two test suites:
9a27877), a temporarymanifest_codec_test.cccross-checked both implementations: every message hadto encode to identical bytes, and each side had to parse the other's output.
This covered the subtle cases — always-present
base/quantizer_paramsub-messages, the absent
quantizer_paramofHNSW_RABITQ, Vamana'sfixed32alpha, empty strings. That test was removed together with thedependency in
d8cd772, since it cannot compile without libprotobuf.manifest_codec_golden_test.cc(kept) pins golden byte arrays that wereproduced by the old protobuf implementation. It has no libprotobuf
dependency, so it continues to guard on-disk format compatibility now that
the dependency is gone.
Existing collections therefore remain readable, and manifests written by this
build remain readable by older releases.
3. Third-party feature trimming
ARROW_DATASET=OFF; unused Compute kernels trimmed viathirdparty/arrow/arrow.slim_compute.patch(aggregates, hash-aggregates,temporal, round, random, rank, replace, select_k, statistics,
cumulative_ops, pivot).
Note on the Arrow patch: an earlier revision over-trimmed and broke production
paths, because several kernels are reached indirectly through Acero nodes and
are invisible to a source grep.
sort_indices(needed by theorder_bynodethat sorts vector scores and FTS BM25 results),
make_struct(the backingimplementation of
compute::project()),match_like(SQLLIKE) andlist_value_length(SQLarray_length()) are all restored, and each retainedkernel now carries a comment stating why it is required.
4. Size-optimized third-party builds
-Osfor RocksDB / glog / ANTLR4,MinSizeRelfor Arrow./O1 /Ob1(favor small code) instead of/O2 /Ob2, including Arrow'sRelease configuration — mirroring the
-Os/MinSizeRelbehavior above.zvec's own code is untouched and keeps its default optimization level.
Where the remaining size goes
For reference, a link-map attribution of the resulting C API library:
Compatibility
src/include/zvec/keep theirexisting declarations; only symbol visibility and unused internals changed.
pinned by golden-byte tests.
as a patch at build time, keeping submodules pristine.
Testing
macOS (arm64), Linux (x64/arm64), Windows (x64), plus Android and iOS builds.
golden-byte tests.
nm/objdumpto confirm only theintended public symbols remain, and that no third-party symbols leak.