Add support for non-byte-aligned bitstrings

[pguyot]

Continuation of:

• Reject negative dynamic segment sizes in bit syntax matching #2377

These changes are made under both the "Apache 2.0" and the "GNU Lesser General
Public License 2.1 or later" license terms (dual license).

SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

[petermm]

Follow-up PR Review — d0cfee66e

This review verifies the fixes pushed after the review of c8fd1c1ed.

Verdict

Changes still requested. Four original findings are fixed. Dynamic bitstring construction is fixed for the modern bs_create_bin path but remains unsafe in legacy construction opcodes and JIT private-append reuse. The new bit-copy implementation also has a boundary overrun, and the new bs_get_binary2 interpreter implementation mishandles some negative sizes through unsigned multiplication wraparound.

Original finding status

1. Partially fixed — dynamic bitstring construction

The ordinary bs_create_bin path is fixed in both runtimes:

• The interpreter sizes all segments with term_bit_size, validates explicit sizes in bits, and uses bitstring_copy_bits at the current bit offset (src/libAtomVM/opcodesswitch.h:5628-5663, src/libAtomVM/opcodesswitch.h:5873-5912).
• The JIT computes source sizes with term_bit_size and copies bit counts rather than byte counts (libs/jit/src/jit.erl:2827-2883, src/libAtomVM/jit.c:1726-1743).
• The new runtime tests exercise dynamic partial-bit sources and mixed unaligned segments.

The original reproduction is now correct:

Bits = <<1:1>>,
{<<Bits/bitstring>>, <<Bits/bitstring, 16#AB:8>>}.

AtomVM now returns {<<1:1>>, <<213,1:1>>}, matching OTP.

However, the legacy opcodes still accept bitstrings and truncate them to complete bytes:

• bs_append uses term_binary_size, memcpy, and src_size * 8 (src/libAtomVM/opcodesswitch.h:3448-3477).
• bs_private_append does the same through term_reuse_binary (src/libAtomVM/opcodesswitch.h:3499-3519).
• bs_put_binary maps all to term_binary_size(src) (src/libAtomVM/opcodesswitch.h:3596-3632).

These paths are used by OTP 26–27 with no_bs_create_bin. Existing legacy tests use only byte-aligned sources, so they do not cover the failure. Until the legacy operations become bit-aware, the smallest safe change is to reject partial bitstrings rather than silently truncate them:

diff --git a/src/libAtomVM/opcodesswitch.h b/src/libAtomVM/opcodesswitch.h
--- a/src/libAtomVM/opcodesswitch.h
+++ b/src/libAtomVM/opcodesswitch.h
@@
-                VERIFY_IS_BITSTRING(src, "bs_append", 0);
+                if (UNLIKELY(!term_is_binary(src))) {
+                    RAISE_ERROR(BADARG_ATOM);
+                }
@@
-                VERIFY_IS_BITSTRING(src, "bs_private_append", 0);
+                if (UNLIKELY(!term_is_binary(src))) {
+                    RAISE_ERROR(BADARG_ATOM);
+                }
@@
-                VERIFY_IS_BITSTRING(src, "bs_put_binary", 0);
+                if (UNLIKELY(!term_is_binary(src))) {
+                    RAISE_ERROR(BADARG_ATOM);
+                }

The JIT also decides whether to reuse a private_append/all source from the final result alignment only (libs/jit/src/jit.erl:2384-2421). Its reuse path then derives the initial offset from term_binary_size(src) (libs/jit/src/jit.erl:2976-3005). A partial source followed by enough bits to make the final result byte-aligned can therefore still lose the source trailing bits. Match the interpreter by requiring the reused source itself to be byte-aligned, or fall back to allocation plus a bit copy.

2. Fixed — partial-width little-endian integers

Insertion now writes complete low-order bytes first and then the remaining high-order bits; extraction implements the inverse mapping (src/libAtomVM/bitstring.c:27-60, src/libAtomVM/bitstring.c:68-126).

The new tests compare byte layouts for widths 1, 4, 7, 9, 12, 15, and 63 at offset zero and after a three-bit prefix. A focused runtime probe now returns <<188,10:4>> for dynamic <<16#ABC:12/little>>, matching OTP.

3. Fixed — JIT bs_match_string trailing capacity

The JIT now measures capacity with term_bit_size and checks bs_offset before subtracting (src/libAtomVM/jit.c:1912-1921). Tests include successful, differing, and too-short matches that consume trailing bits.

4. Fixed — ETF padding canonicalization

Serialization now masks insignificant low bits from the final byte (src/libAtomVM/external_term.c:389-410). The added test decodes a one-bit value backed by 255 and requires re-encoding with final byte 128. A focused runtime probe confirms the canonical result.

5. Fixed, test suggested — bitstring printing

term_funprint now prints the trailing Value:Size field and correctly handles a sub-byte value without an empty quoted prefix (src/libAtomVM/term.c:340-409). Runtime output now renders the tested values correctly.

There is no direct regression test for printing. Add coverage for <<1:1>>, <<255,5:7>>, and a printable byte followed by trailing bits.

New findings introduced or exposed by the fixes

1. Critical — incomplete bit copies access one byte past their buffers

bitstring_copy_bits_incomplete_bytes eagerly reads the next destination and source bytes whenever their bit indices reach zero, even when the bit just copied was the final requested bit; it then unconditionally writes the current destination byte after the loop (src/libAtomVM/bitstring.c:338-375).

Examples:

• Copying seven bits at destination offset one ends exactly at the byte boundary. The loop writes the completed byte, advances dst, reads *dst past the logical destination, and writes it again after the loop.
• Copying eight bits to an unaligned destination consumes the last source bit and then immediately reads the byte after the source.
• A zero-bit copy on the incomplete path still reads both buffers.

This helper is now used by interpreter and JIT construction, so the new bit-granular fix exposes both paths to out-of-bounds reads/writes. Heap padding can hide the issue in ordinary tests; refc binaries are allocated to their requested byte size.

The smallest robust fix is a bounded bit loop:

diff --git a/src/libAtomVM/bitstring.c b/src/libAtomVM/bitstring.c
--- a/src/libAtomVM/bitstring.c
+++ b/src/libAtomVM/bitstring.c
@@
 void bitstring_copy_bits_incomplete_bytes(uint8_t *dst, size_t bits_offset, const uint8_t *src, size_t bits_count)
 {
-    /* current prefetch/rollover implementation */
+    for (size_t i = 0; i < bits_count; ++i) {
+        size_t dst_bit = bits_offset + i;
+        uint8_t mask = (uint8_t) (1U << (7 - (dst_bit % 8)));
+        if (src[i / 8] & (uint8_t) (1U << (7 - (i % 8)))) {
+            dst[dst_bit / 8] |= mask;
+        } else {
+            dst[dst_bit / 8] &= (uint8_t) ~mask;
+        }
+    }
 }

Add exact-boundary tests for source widths 0, 7, and 8 at destination offset one, and run them under ASan.

2. High — bs_get_binary2 can accept a huge negative size after multiplication wraps

The interpreter casts a negative size to size_t and then multiplies by unit, assuming the result will remain huge and fail the capacity check (src/libAtomVM/opcodesswitch.h:4463-4486). Unsigned multiplication can instead wrap to zero or another small value.

This was reproduced against the rebuilt VM:

dyn(N, B) ->
    case B of
        <<X:N/binary-unit:64, R/bits>> -> {X, R};
        _ -> nope
    end.

dyn(-(1 bsl 58), <<1>>).

AtomVM incorrectly returns {<<>>, <<1>>}. OTP returns nope. The JIT explicitly rejects a negative value before multiplication, so this is also interpreter/JIT semantic divergence.

diff --git a/src/libAtomVM/opcodesswitch.h b/src/libAtomVM/opcodesswitch.h
--- a/src/libAtomVM/opcodesswitch.h
+++ b/src/libAtomVM/opcodesswitch.h
@@
                 size_t size_bits;
                 if (term_is_integer(size)) {
-                    size_bits = (size_t) term_to_int(size) * unit;
+                    avm_int_t size_value = term_to_int(size);
+                    if (size_value < 0
+                        || (unit != 0 && (size_t) size_value > remaining_bits / unit)) {
+                        JUMP_TO_ADDRESS(mod->labels[fail]);
+                    }
+                    size_bits = (size_t) size_value * unit;

Also guard bs_offset > term_bit_size(bs_bin) before calculating remaining_bits to avoid unsigned subtraction underflow on an invalid match state.

3. Medium — JIT rejects valid all matches with non-power-of-two units

The interpreter checks remaining_bits % unit, but the JIT implements divisibility as remaining_bits & (unit - 1) and therefore explicitly raises unsupported for non-power-of-two units (libs/jit/src/jit.erl:1390-1423). The new unit:3 test covers only a variable integer size, not all, so it misses this parity gap.

Use a remainder operation or a small divisibility primitive for the JIT and add successful/failing all cases with binary-unit:3.

Validation performed

• Compared c8fd1c1ed..d0cfee66e and inspected interpreter/JIT call paths.
• Consulted Oracle for an independent fix verification and regression review.
• cmake --build build -j4 — passed.
• build/tests/test-erlang — passed.
• Rebuilt AtomVM execution of test_bs.beam and test_binary_to_term.beam — both returned 0.
• Focused runtime probe confirmed fixes 1–5 on the modern paths.
• Focused runtime probe reproduced the negative-size wraparound finding.

[petermm]

Review of 96323ce22c88070ba74bcce6b1bf3bde34dbcaa0

Verdict: changes requested. The ordinary-sized bitstring paths are well covered, but constructor arithmetic can wrap, one matching overflow is still present in the JIT, and wide integer construction can silently produce the wrong bits. The new native boundary test is also not run by CI.

Force-push recheck

This commit is a replacement for previously reviewed 6d6e9b191603497290a9dbbe6fac4da809997173, not a descendant of it. Comparing the two trees shows that the force-push changed only src/libAtomVM/external_term.c and src/libAtomVM/utils.h; every file involved in findings 1–5 is byte-for-byte unchanged, so all five findings remain open.

The new delta extracts the SMALL_BIG_EXT slow path into a NOINLINE parse_bigint helper. That is a sound fix: it preserves the prior conversion and allocation behavior while removing the approximately 1 KiB INTN_MAX_RES_LEN temporary from every recursive parse_external_terms stack frame. I found no regression in that extraction. A separate large-length issue in the newly added BIT_BINARY_EXT path is recorded as finding 6.

Findings

1. High — Reject constructor sizes before scaling or accumulating them

Location: src/libAtomVM/opcodesswitch.h:5791-5814 (also the new bs_init_bits, bs_append, and bs_private_append size calculations around lines 3453-3459, 3521-3523, and 3589-3591); JIT equivalents in libs/jit/src/jit.erl:2752-2785 and 2838-2863.

bs_create_bin multiplies the Erlang-controlled signed size before checking it:

avm_int_t size_in_bits = signed_size_value * segment_unit;

That is signed-overflow undefined behavior. It then repeats unchecked multiplication while accumulating binary_size, and rounds with (binary_size + 7) / 8, which can also wrap. The other newly bit-granular construction paths similarly add an Erlang-controlled size to the source size and round without overflow checks.

This is observable without allocating a large binary. On a 64-bit build:

make(B, N) -> <<B:N/binary-unit:64>>.

make(<<>>, 1 bsl 58).

AtomVM at this commit returns <<>>; BEAM rejects the empty source with badarg. I reproduced the AtomVM result locally. The product 2^58 * 64 wraps to zero, so both the source-capacity check and allocation are bypassed. More complex segment lists can make the wrapped allocation size disagree with the second-pass writes.

Use checked, unsigned arithmetic after the negative-size check, and use the source capacity to avoid multiplying before comparison. For example:

diff --git a/src/libAtomVM/opcodesswitch.h b/src/libAtomVM/opcodesswitch.h
@@
-avm_int_t size_in_bits = signed_size_value * segment_unit;
-if ((size_t) size_in_bits > term_bit_size(src)) {
+size_t src_bits = term_bit_size(src);
+if (segment_unit != 0
+    && (size_t) signed_size_value > src_bits / (size_t) segment_unit) {
     if (fail == 0) {
         RAISE_ERROR(BADARG_ATOM);
     } else {
         JUMP_TO_LABEL(mod, fail);
     }
 }
+size_t size_in_bits = (size_t) signed_size_value * (size_t) segment_unit;
@@
-binary_size += segment_unit * segment_size;
+size_t segment_bits;
+if (segment_unit != 0 && segment_size > SIZE_MAX / (size_t) segment_unit) {
+    RAISE_ERROR(SYSTEM_LIMIT_ATOM);
+}
+segment_bits = (size_t) segment_unit * segment_size;
+if (binary_size > SIZE_MAX - segment_bits) {
+    RAISE_ERROR(SYSTEM_LIMIT_ATOM);
+}
+binary_size += segment_bits;
@@
-size_t binary_bytes = (binary_size + 7) / 8;
+if (binary_size > SIZE_MAX - 7) {
+    RAISE_ERROR(SYSTEM_LIMIT_ATOM);
+}
+size_t binary_bytes = (binary_size + 7) / 8;

The exact exception atom should follow the VM's established oversized-allocation convention. The same checked helper/path should be used by bs_init_bits, bs_append, bs_private_append, both bs_create_bin passes, and the JIT size pass so the allocation and write calculations cannot diverge.

Add a regression that covers both binary and integer construction with positive multiplication overflow; the current tests cover large negative values, but not this case.

2. High — Use the JIT's overflow-aware scaler in bs_get_binary2

Location: libs/jit/src/jit.erl:1415-1434.

The interpreter now correctly calls bs_scaled_size_bits, but the JIT's dynamic bs_get_binary2 branch still multiplies first and compares the wrapped result:

BSt3 = MMod:mul(BSt2, SizeValReg, Unit),
cond_jump_to_label({RemainingReg, '<', SizeValReg}, Fail, MMod, BSt3)

For a unit of 64 and N = 1 bsl 58, the scaled size wraps to zero on a 64-bit target. The JIT can therefore match an empty slice while the interpreter and BEAM fail:

match(B, N) ->
    case B of
        <<X:N/binary-unit:64, Rest/bits>> -> {X, Rest};
        _ -> nope
    end.

%% Expected: nope
match(<<1>>, 1 bsl 58).

There is already an overflow-aware helper used by the integer, float, and skip match paths. Reuse it here:

diff --git a/libs/jit/src/jit.erl b/libs/jit/src/jit.erl
@@
-BSt3 = MMod:mul(BSt2, SizeValReg, Unit),
+BSt3 = scale_size_by_unit(SizeValReg, Unit, Fail, MMod, BSt2),
 cond_jump_to_label({RemainingReg, '<', SizeValReg}, Fail, MMod, BSt3)

Add the positive-overflow call above to test_dynamic_size_extraction/0; it specifically protects interpreter/JIT parity.

3. Medium — Sign-extend negative small integers wider than 64 bits

Location: src/libAtomVM/bitstring.c:90-125; called from src/libAtomVM/opcodesswitch.h:5928-5936 and src/libAtomVM/jit.c:1699-1704.

The newly accepted non-byte-aligned width exposes an existing truncation assumption in bitstring_insert_any_integer. For big-endian fields wider than 64 bits, it skips all leading bits and writes only the low 64. For little-endian fields, bytes after the first eight are written from a value that has already shifted to zero. Both behaviors zero-extend a negative small/int64 value instead of sign-extending it.

make(V, N) -> <<V:N>>.

%% Expected: <<1:1, 16#FFFFFFFFFFFFFFFF:64>>
make(-1, 65).

I reproduced this at the reviewed commit: AtomVM prints <<127,255,255,255,255,255,255,255,1:1>>, whose first field bit is zero. Before this commit, the 65-bit result was rejected as unsupported, so this is a regression in the newly enabled path. <<-1:65/little>> has the analogous missing high-order bit.

The complete fix is to fill bits beyond the low 64 with the value's sign. If that is intentionally out of scope, the safe minimal fix is to reject newly enabled non-byte-aligned integer widths above 64 rather than silently returning the wrong value:

diff --git a/src/libAtomVM/bitstring.c b/src/libAtomVM/bitstring.c
@@
 bool bitstring_insert_any_integer(uint8_t *dst, avm_int_t offset, avm_int64_t value,
     size_t n, enum BitstringFlags bs_flags)
 {
+    if (n > 64 && ((offset & 0x7) != 0 || (n & 0x7) != 0)) {
+        return false;
+    }
     // SignedInteger flag does not affect insertion (caller handles sign extension)

That fallback matches this commit's explicit unsupported handling for non-byte-aligned bignum segments. Prefer a sign-extension implementation for full BEAM parity, with big- and little-endian tests for -1, another negative value, and a positive value at widths 65 and 71.

4. Medium — Do not feed an unaligned destination to the whole-byte bignum writer

Location: src/libAtomVM/opcodesswitch.h:3660-3679 (OP_BS_PUT_INTEGER).

The old-opcode path now permits a bitstring builder to have a partial-byte offset, but its bignum branch still computes ctx->bs_offset / 8 and calls intn_to_integer_bytes, which only writes whole byte-aligned fields. A construction such as a one-bit prefix followed by a >64-bit integer writes the integer at the beginning of the containing byte, overwriting/misplacing the prefix. The bs_create_bin implementation already detects and rejects exactly this case at lines 5937-5943; the old opcode needs the same guard.

diff --git a/src/libAtomVM/opcodesswitch.h b/src/libAtomVM/opcodesswitch.h
@@
 } else {
+    if (size_value < 0
+        || (unit != 0 && (size_t) size_value > SIZE_MAX / unit)) {
+        RAISE_ERROR(BADARG_ATOM);
+    }
+    size_t field_bits = (size_t) size_value * (size_t) unit;
+    if ((ctx->bs_offset % 8) != 0 || (field_bits % 8) != 0) {
+        TRACE("bs_put_integer: non-byte-aligned big integer segment unsupported\n");
+        RAISE_ERROR(UNSUPPORTED_ATOM);
+    }
     const intn_digit_t *big_src_value = NULL;

This should eventually call a bit-granular bignum writer for BEAM parity, but rejecting is consistent with the limitation documented and tested for bs_create_bin in this commit.

5. Low — Register test-bitstring in CI instead of only compiling it

Location: tests/CMakeLists.txt:24-108; .github/workflows/build-and-test.yaml:795-876 (and the macOS/FreeBSD native-test steps).

The commit adds and builds test-bitstring, but no workflow executes it and the project does not register native executables with CTest. Consequently, the guard-buffer and exact-allocation assertions protecting bitstring_copy_bits never gate a PR. The test passes when run manually (build/tests/test-bitstring), confirming this is registration rather than a test failure.

At minimum, add it beside the other native tests in the main workflow, including the Valgrind run that gives the exact-allocation case its intended value:

diff --git a/.github/workflows/build-and-test.yaml b/.github/workflows/build-and-test.yaml
@@
+    - name: "Test: test-bitstring with valgrind"
+      if: matrix.library-arch == ''
+      working-directory: build
+      run: valgrind --error-exitcode=1 ./tests/test-bitstring
+
+    - name: "Test: test-bitstring"
+      working-directory: build
+      run: ./tests/test-bitstring
+
     - name: "Test: test-enif with valgrind"

Also run it alongside the other standalone native executables on macOS and FreeBSD.

6. Medium — Keep BIT_BINARY_EXT lengths in range and calculate them without 32-bit wrap

Location: src/libAtomVM/external_term.c:175-185, 238, 395-410, 870-880, and 1371-1387.

The new serializer returns an int, but the bitstring branch returns 6 + nbytes, where nbytes is size_t. compute_external_size converts that result back to size_t for allocation. Once the encoded size exceeds INT_MAX, the conversion can produce a negative int and then a huge allocation size. Independently, ETF's length field is 32 bits, but WRITE_32_UNALIGNED silently truncates nbytes > UINT32_MAX while memcpy still copies the full size_t length.

The decode side has a smaller, direct arithmetic bug. binary_size and BIT_BINARY_EXT_BASE_SIZE participate in 32-bit unsigned arithmetic here:

*eterm_size = BIT_BINARY_EXT_BASE_SIZE + binary_size;

For an otherwise complete BIT_BINARY_EXT with binary_size == UINT32_MAX, the consumed length is reported as 5 rather than UINT32_MAX + 6. Nested parsing or binary_to_term(..., [used]) can therefore continue at the wrong offset. The old BINARY_EXT path has related large-size limitations, but this commit newly adds them to bitstrings.

At minimum, prevent the decode-side wrap explicitly:

diff --git a/src/libAtomVM/external_term.c b/src/libAtomVM/external_term.c
@@
-*eterm_size = 6 + binary_size;
+*eterm_size = (size_t) BIT_BINARY_EXT_BASE_SIZE + (size_t) binary_size;
@@
-*eterm_size = BIT_BINARY_EXT_BASE_SIZE + binary_size;
+*eterm_size = (size_t) BIT_BINARY_EXT_BASE_SIZE + (size_t) binary_size;

The coherent serializer fix is broader: make internal serialized-size results checked size_t values, reject binary/bitstring payloads larger than UINT32_MAX, and propagate that failure through both the size and write passes. Computing the byte ceiling as total_bits / 8 + (total_bits % 8 != 0) also avoids total_bits + 7 wrapping.

Review notes and validation

• Reviewed the updated commit against its first parent and compared it directly with the previously reviewed tree across the interpreter, JIT code generator and C primitives, sub-binary representation, ETF encoding/decoding, BIF/NIF guards, comparison/printing, tests, and CI registration.
• Asked Oracle for an independent recheck focused on the disposition of the five existing findings, the force-push delta, memory safety, overflow, BEAM parity, and interpreter/JIT divergence; its candidate findings were then checked against the source before inclusion.
• Locally rebuilt the tree successfully and ran:
  • build/tests/test-bitstring — pass
  • build/tests/test-erlang test_bs — pass
  • build/tests/test-erlang test_no_bs_create_bin — pass
  • build/tests/test-erlang test_binary_to_term — pass
• Added standalone repro modules outside the worktree to confirm findings 1 and 3 on AtomVM. No repository source was modified for those repros.
• I did not include a proposed JIT private_append finding from the independent review: compiler disassembly shows the partial-source path uses append, while private_append is reserved for writable accumulator states, so the suggested trigger was not representative enough for a high-confidence PR finding.