perf: remove two hot-path memory indirections (Callgrind rec_array lookup, allocator free-list scan)

[codspeed-hq]

Summary

Flamegraph analysis of the Callgrind walltime benchmarks pointed at two dependent-load stalls that Valgrind pays on its two hottest paths — once per basic block execution, and once per arena allocation. Both are removed without changing any observable behaviour.

1. CLG_(setup_bbcc): skip the recursion-level lookup when it is a no-op

--separate-recs defaults to 2, so the separate_recursions > 1 block in CLG_(setup_bbcc) runs on every basic block execution. It ended with:

idx = level - 1;
if (bbcc->rec_array[idx]) bbcc = bbcc->rec_array[idx];
else                      bbcc = clone_bbcc(...);
CLG_ASSERT(bbcc->rec_array[bbcc->rec_index] == bbcc);

rec_array lives in a separate allocation (CLG_(new_recursion)), so rec_array[idx] is a cold pointer chase, and the assertion adds two more dependent loads on the result. In the profile of python3 testdata/test.py, no-inline those three instructions carried 9.4 % of total run time (4.09 % + 2.98 % + 2.37 % self time), and 8 % in full-with-inline.

Every BBCC satisfies the invariant rec_array[rec_index] == itself — established in clone_bbcc (both branches), in CLG_(get_bbcc)'s fresh-BBCC path and in CLG_(setup_bbcc)'s own rec_array[0] = bbcc. So when idx == bbcc->rec_index — the normal, non-recursive case — the lookup provably returns bbcc again. Guarding the block with if (idx != bbcc->rec_index) skips the pointer chase and the assertion entirely; rec_index sits in the same cache line as rec_array, in a struct already touched a few lines above.

2. VG_(arena_malloc): replace the linear free-list scan with a bitmap

The allocator's first step is "find the lowest non-empty free list at or above my size class", implemented as a linear walk over the N_MALLOC_LISTS == 112 list heads — up to 14 cache lines touched per allocation, almost all of them NULL. That loop cost 4.4 % of run time in the same profile.

The source comment already proposed fixing this with a shortcut array. This patch uses the simpler and cheaper variant: a two-word bitmap (freelist_used) with bit i set iff freelist[i] != NULL. Only two sites transition a list between empty and non-empty (mkFreeBlock and unlinkBlock), so maintenance is two masked bit operations; the query becomes a mask plus __builtin_ctzll. The other two writers of freelist[...] (swizzle, and the "step one along" path inside the search) keep the head non-NULL, so the bitmap stays exact. The old if (NULL == b) continue; becomes a vg_assert(b) — a live self-check that the bitmap and the lists agree.

Correctness

Verified against an unpatched build of the same commit, configured and built identically:

• Callgrind output is byte-for-byte identical on 8 configurations — default, --separate-recs=1, --separate-recs=3, --separate-recs=10, --skip-direct-rec=no, --separate-callers=3, /bin/echo, and --cache-sim=yes --read-inline-info=yes — including all event counts and the summary: line. The recursion configurations are exercised with a fixture using both direct (fib) and mutual (ping/pong) recursion. (The only textual difference is each build's own install path appearing in a cob= line, as expected.)
• 22/22 Callgrind regression tests pass (0 stderr/stdout/post failures) — the same set CI runs.
• 292 Memcheck regression tests produce exactly the same 4 failures as the unpatched build (gone_abrt_xml, sem, vcpu_bz2), all pre-existing sandbox/environment artefacts. This is a direct stress test of the allocator change, since every Memcheck run hammers VG_(arena_malloc)/VG_(arena_free).
• The build produces no new compiler warnings.

Measurement

Base and head were measured in this sandbox with codspeed run --mode walltime over 12 valgrind.codspeed benchmarks (20 measured rounds each, 2 s warmup), covering echo, python3 testdata/test.py, stress-ng and llsc_tzconvert_bench across the no-inline, inline, full-with-inline and cycle-estimation configurations.

Comparison	Overall impact
base → head (pair 1)	+2.24 %
base → head (pair 2)	+1.83 %
base → base (control)	+0.03 %
head → head (control)	−0.37 %

The improvement is consistent across the whole matrix rather than concentrated in one benchmark, which matches the fact that both changes sit on paths every Callgrind run takes. The two same-build controls bound this sandbox's noise floor well below the measured effect. Individual benchmarks stay under CodSpeed's per-benchmark significance threshold; the aggregate impact is the meaningful signal here.

[codspeed-hq]

Merging this PR will not alter performance

✅ 84 untouched benchmarks
⏩ 60 skipped benchmarks¹

---

_{Comparing codspeed-optim-remove-two-hot-path-memory-indirections-per-bb-rec-1785567510152 (71b3e5e) with master (ae6bf15)}

Footnotes

1. 60 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Greptile Summary

The PR removes two hot-path pointer scans while preserving existing selection behavior.

• Skips the Callgrind recursion-array lookup when the selected recursion index already matches the current BBCC.
• Tracks non-empty allocator free lists in a per-arena bitmap and jumps directly to the next eligible list.
• Updates the bitmap on empty/non-empty transitions and initializes it with each arena.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete behavioral, build, or security regression identified.

The recursion fast path relies on an invariant established by BBCC construction, while every current freelist empty-state transition updates the new bitmap and the lookup arithmetic remains valid across supported word widths.

Important Files Changed

Filename	Overview
callgrind/bbcc.c	Avoids a redundant recursion-array dereference when the requested recursion index already equals the BBCC's self index; the established BBCC initialization paths preserve that invariant.
coregrind/m_mallocfree.c	Adds and consistently maintains a per-arena non-empty-free-list bitmap, replacing the allocator's linear list-head scan without changing block selection semantics.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Allocation request] --> B[Map size to starting free-list index]
    B --> C[Mask freelist_used bitmap]
    C --> D{Non-empty list found?}
    D -- No --> E[Allocate a new superblock]
    D -- Yes --> F[Scan selected free list]
    F --> G{Suitable block found?}
    G -- Yes --> H[Unlink and allocate block]
    G -- No --> I[Query next set bitmap bit]
    I --> D
    J[mkFreeBlock: empty to non-empty] --> K[Set bitmap bit]
    L[unlinkBlock: non-empty to empty] --> M[Clear bitmap bit]

Loading

_{Reviews (1): Last reviewed commit: "perf: remove two hot-path memory indirec..." | Re-trigger Greptile}