Skip to content

feat: slim down zvec dynamic libraries - #627

Open
chinaux wants to merge 11 commits into
alibaba:mainfrom
chinaux:feat/slim-c-api-library
Open

feat: slim down zvec dynamic libraries#627
chinaux wants to merge 11 commits into
alibaba:mainfrom
chinaux:feat/slim-c-api-library

Conversation

@chinaux

@chinaux chinaux commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR substantially reduces the size of zvec's prebuilt dynamic libraries with
zero functional loss and no public API change. Core search code stays at
-O3, so query performance is unaffected.

Measured on macOS arm64, Release + strip, both built from the same base commit
(d15a37e):

Library main baseline this PR Reduction
libzvec_c_api.dylib (C API) 31.11 MB 19.19 MB -11.92 MB (-38.3%)
libzvec.dylib (C++ SDK) 30.88 MB 20.44 MB -10.44 MB (-33.8%)

It also removes the protobuf/protoc build dependency entirely while keeping
the on-disk manifest format byte-for-byte compatible.

Motivation

Both dynamic libraries statically bundle heavy third-party dependencies (Arrow,
RocksDB, protobuf, ANTLR4, glog). By default they export all of those
symbols and carry code paths zvec never uses. That inflates binary size, slows
down linking, and pollutes the global symbol namespace for downstream consumers.

protobuf was a particularly poor trade: zvec only used it to persist the
collection manifest, yet it pulled in a code generator (protoc), a submodule,
and a runtime library into every build.

Approach

1. Symbol visibility & dead-code elimination

  • Compile with -ffunction-sections -fdata-sections so unreachable code can be
    pruned at link time.
  • macOS: -exported_symbols_list + -dead_strip
  • Linux: --version-script + --gc-sections
  • libzvec_c_api exports only the zvec_* C API.
  • libzvec exports only zvec's own public C++ API (namespace zvec, including
    the nested zvec::ailego / zvec::core / zvec::turbo / zvec::reranker),
    hiding ~14.5k third-party symbols (exported symbols: 23,102 → 8,609).

The public C++ headers never expose Arrow/RocksDB/Parquet types, so hiding
those symbols is safe for external C++ consumers.

Two follow-up fixes were needed to make the Linux script correct:

  • Export the mangled vtable / typeinfo / VTT / guard-variable patterns
    (_ZTVN4zvec*, _ZTIN4zvec*, ...). The demangled zvec::* glob does not
    match them, so consumers previously failed with
    undefined reference to vtable for zvec::core_interface::HNSWIndexParam.
  • Drop the quoted "zvec::*" entry: in version scripts a quoted name is a
    literal match, not a glob. GNU ld silently ignores the never-matching entry,
    but lld (Android NDK) fails under --no-undefined-version.

2. Drop the protobuf / protoc dependency

The manifest is now encoded by a self-contained codec instead of generated
protobuf code:

  • pb_wire.h — a minimal protobuf wire-format reader/writer (varint,
    fixed32/64, length-delimited). Unknown fields are consumed and ignored to
    preserve proto3 forward compatibility; malformed input (bad varint,
    out-of-range length, groups, field number 0) is reported rather than crashing.
  • manifest_codec.{h,cc} — converts all 16 messages directly between zvec's C++
    types and bytes, with no intermediate message object.
  • manifest_enum.h — the on-disk enum values, preserved exactly.

The thirdparty/protobuf submodule, src/db/proto/zvec.proto and the generated
proto_converter are removed.

Format compatibility is enforced by two test suites:

  1. While libprotobuf was still in the tree (commit 9a27877), a temporary
    manifest_codec_test.cc cross-checked both implementations: every message had
    to encode to identical bytes, and each side had to parse the other's output.
    This covered the subtle cases — always-present base / quantizer_param
    sub-messages, the absent quantizer_param of HNSW_RABITQ, Vamana's
    fixed32 alpha, empty strings. That test was removed together with the
    dependency in d8cd772, since it cannot compile without libprotobuf.
  2. manifest_codec_golden_test.cc (kept) pins golden byte arrays that were
    produced by the old protobuf implementation. It has no libprotobuf
    dependency, so it continues to guard on-disk format compatibility now that
    the dependency is gone.

Existing collections therefore remain readable, and manifests written by this
build remain readable by older releases.

3. Third-party feature trimming

  • Arrow: ARROW_DATASET=OFF; unused Compute kernels trimmed via
    thirdparty/arrow/arrow.slim_compute.patch (aggregates, hash-aggregates,
    temporal, round, random, rank, replace, select_k, statistics,
    cumulative_ops, pivot).
  • RocksDB: trace / iostats / perf contexts disabled.

Note on the Arrow patch: an earlier revision over-trimmed and broke production
paths, because several kernels are reached indirectly through Acero nodes and
are invisible to a source grep. sort_indices (needed by the order_by node
that sorts vector scores and FTS BM25 results), make_struct (the backing
implementation of compute::project()), match_like (SQL LIKE) and
list_value_length (SQL array_length()) are all restored, and each retained
kernel now carries a comment stating why it is required.

4. Size-optimized third-party builds

  • GCC/Clang: -Os for RocksDB / glog / ANTLR4, MinSizeRel for Arrow.
  • MSVC: /O1 /Ob1 (favor small code) instead of /O2 /Ob2, including Arrow's
    Release configuration — mirroring the -Os / MinSizeRel behavior above.

zvec's own code is untouched and keeps its default optimization level.

Where the remaining size goes

For reference, a link-map attribution of the resulting C API library:

Component Size Share
Arrow stack (core + compute + acero + parquet + bundled) 9.37 MB 49.6%
zvec's own code (core + db + ailego + turbo + c_api) 5.44 MB 28.8%
RocksDB 2.93 MB 15.5%
Other third-party (snowball / ANTLR4 / lz4 / glog / roaring / FastPFOR) 1.14 MB 6.0%

Compatibility

  • No public API changes: headers under src/include/zvec/ keep their
    existing declarations; only symbol visibility and unused internals changed.
  • On-disk format unchanged: the manifest is still protobuf wire format,
    pinned by golden-byte tests.
  • Third-party sources are not modified in place: Arrow trimming is applied
    as a patch at build time, keeping submodules pristine.
  • Existing C / C++ / Python examples and tests pass unchanged.

Testing

  • Full C++ unit test suite, Python tests, and the C/C++ examples pass on
    macOS (arm64), Linux (x64/arm64), Windows (x64), plus Android and iOS builds.
  • Manifest format verified by byte-level cross-checks against libprotobuf and by
    golden-byte tests.
  • Exported symbol tables inspected with nm / objdump to confirm only the
    intended public symbols remain, and that no third-party symbols leak.
  • Library sizes verified on each platform.

@chinaux
chinaux force-pushed the feat/slim-c-api-library branch 2 times, most recently from b91f370 to 3462df8 Compare July 29, 2026 12:04
@chinaux
chinaux force-pushed the feat/slim-c-api-library branch from 6b1c2a2 to f7efd7c Compare August 10, 2026 12:06
chinaux added 11 commits August 10, 2026 20:13
Reduce the FAT C API dylib size by ~37% through a 5-layer strategy with
zero functional loss; core search code stays at -O3 for performance:

- Compiler dead-code sectioning: -ffunction-sections -fdata-sections
- Linker pruning: macOS -dead_strip + exported symbol list; Linux
  --gc-sections + version script, keeping only zvec_* C API symbols
- Third-party feature trimming: Arrow disables Dataset & trims Compute
  kernels; RocksDB disables trace/iostats/perf contexts
- Protobuf lite: proto uses LITE_RUNTIME, link libprotobuf-lite
- Size-optimized third-party builds: -Os for RocksDB/glog/antlr4,
  MinSizeRel for Arrow
Extend the size-reduction work to the C++ all-in-one shared library and
the Windows build:

- libzvec C++ SDK (macOS/Linux): restrict exported symbols to zvec's own
  public API (namespace zvec, incl. nested zvec::ailego/core/turbo) via
  -exported_symbols_list / --version-script + -dead_strip / --gc-sections.
  Hides ~14.5k third-party symbols; libzvec.dylib 24.41MB -> 19.05MB
  (-21.9%), total -34.1% vs main baseline. Public C++ headers never expose
  third-party types, so external C++ consumers are unaffected.
- Third-party libs (MSVC): favor small code (/O1 /Ob1) over speed (/O2)
  to mirror the -Os / MinSizeRel size optimization used on GCC/Clang,
  including Arrow's Release config.
…rom tests

The slim compute patch removed kernels that production code paths rely on:
- match_like (LIKE queries) from scalar_string_ascii/utf8
- sort_indices (ORDER BY) from vector_sort/vector_array_sort
- make_struct / list_value_length from scalar_nested

Restore these kernels in arrow.slim_compute.patch while keeping the rest
(aggregates, temporal, round, random, etc.) removed.

Arrow Dataset stays disabled: sql_expr_validator_test's dataset member was
dead code and is removed; sql_expr_parser_test's scan test is rewritten
with arrow::compute::ExecuteScalarExpression, preserving the original
intent (verify a parsed expression evaluates correctly on real data)
without depending on arrow::dataset.

All 153 C++ tests pass locally (write_recovery_test flake passes in
isolation).
…script

The version script only exported zvec::* function/data symbols. vtable,
typeinfo, VTT and guard variable symbols demangle to 'vtable for zvec::...'
etc., which the zvec::* pattern does not match, so they were localized by
'local: *'. Consumers linking libzvec.so then failed with
'undefined reference to vtable for zvec::core_interface::HNSWIndexParam'.

Add the mangled-name patterns (_ZTVN4zvec*, _ZTIN4zvec*, _ZTSN4zvec*,
_ZTTN4zvec*, _ZGVN4zvec*), mirroring what exported_symbols_cpp.txt already
does on macOS.
…ibility

In linker version scripts, a quoted "zvec::*" entry is a literal name
match (not a glob). GNU ld silently tolerates the never-matching entry,
but lld (used by the Android NDK) enforces --no-undefined-version and
fails with:

  ld.lld: error: version script assignment of 'global' to symbol
  'zvec::*' failed: symbol not defined

Keep only the unquoted glob pattern, which matches all demangled
zvec:: symbols on both GNU ld and lld.
Introduces a self-contained implementation of the manifest on-disk format,
in preparation for dropping the libprotobuf/protoc dependency:

- pb_wire.h: minimal protobuf wire-format reader/writer (varint, fixed32/64,
  length-delimited). Unknown fields are consumed and ignored, preserving
  proto3 forward compatibility; malformed input (bad varint, out-of-range
  length, groups, field number 0) is reported instead of crashing.
- manifest_enum.h: the on-disk enums, mirroring src/db/proto/zvec.proto with
  identical numeric values. The CodeBooks in type_helper.h now speak these
  instead of the generated protobuf enums.
- manifest_codec.{h,cc}: encodes/decodes all 16 messages directly between
  zvec's C++ types and bytes, without an intermediate message object.

Format compatibility is established by two test suites:

- manifest_codec_test.cc cross-checks against libprotobuf while it is still
  available: every message encoded by both implementations must produce
  identical bytes, and each side must parse the other's output. This covers
  the subtle cases - always-present base/quantizer_param sub-messages, the
  absent quantizer_param of HNSW_RABITQ, Vamana's fixed32 alpha, empty
  repeated string elements and oneof "last branch wins".
- manifest_codec_golden_test.cc pins protobuf's real output as embedded byte
  arrays and does not depend on libprotobuf, so it keeps guarding the format
  after the dependency is removed.

src/db/proto/zvec.proto is retained as the authoritative documentation of the
format. No behaviour change yet: Version::Save/Load still use protobuf.
Version::Load/Save no longer build an intermediate proto::Manifest; they read
and write the manifest bytes with ManifestCodec instead. The on-disk format is
unchanged - manifest_codec_test.cc verifies byte-for-byte equality with the
protobuf implementation, and manifest_codec_golden_test.cc pins the format
independently.

Manifests are a few kilobytes, so the file is read/written in one go rather
than streamed.

ProtoConverter is intentionally kept for now: it is what the cross-check tests
compare against. It is removed together with the protobuf dependency.
The manifest is now read and written entirely by ManifestCodec (added and
cross-checked against protobuf in the previous commits), so libprotobuf and
protoc are no longer needed.

Removed:
- thirdparty/protobuf submodule and its build integration
- src/db/proto compilation (cc_proto_library, zvec.pb.cc, libprotobuf-lite
  link and zvec_proto target/deps across src and tests)
- ProtoConverter and the cross-check test that depended on libprotobuf
- the "build host protoc" stage from the Android and iOS CI workflows and the
  build_android.sh / build_ios.sh scripts, along with the now-unused
  GLOBAL_CC_PROTOBUF_PROTOC option. Cross-compiling no longer needs a host
  protoc, simplifying those pipelines noticeably.

Kept:
- src/db/proto/zvec.proto as the authoritative documentation of the on-disk
  manifest format
- cc_proto_library in cmake/bazel.cmake as a generic helper (no longer used
  by zvec itself)

The manifest on-disk format is unchanged; manifest_codec_golden_test.cc keeps
guarding it. Full C++ test suite (156 tests) passes locally.
zvec.proto was retained as documentation when the protobuf dependency was
dropped, but the manifest format is now fully defined by manifest_codec.{h,cc}
(field numbers) and manifest_enum.h (enum values). Delete the file and the
src/db/proto directory, and repoint the doc comments that referenced it to the
codec instead.

Also drop the stale clang-tidy header cache entries for build/src/db/proto and
the src/db/proto/*.proto hashFiles input, which no longer exist.
meta.h uses std::numeric_limits but never included <limits>; it compiled only
because type_helper.h transitively included the generated zvec.pb.h, which
brought <limits> along. With protobuf gone the omission surfaced on
libstdc++/GCC:

  meta.h:280: error: 'numeric_limits' is not a member of 'std'

libc++ happens to provide it transitively, which is why only the Linux jobs
failed.

Also make the files touched by the codec change include what they use:
<memory>/<string>/<string_view>/<vector>/<cstdint> in manifest_codec.cc and
<memory> in version_manager.cc.
@chinaux
chinaux force-pushed the feat/slim-c-api-library branch from f7efd7c to 287ed09 Compare August 10, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant