Skip to content

Add incremental JSON Lines export for parameter sweeps - #1032

Open
Supersynergy wants to merge 3 commits into
sharkdp:masterfrom
Supersynergy:perf/json-lines-export
Open

Supersynergy wants to merge 3 commits into
sharkdp:masterfrom
Supersynergy:perf/json-lines-export

Conversation

@Supersynergy

@Supersynergy Supersynergy commented Nov 12, 2026 •

Copy link
Copy Markdown

For the Hyperfine 2.1 milestone.

Large parameter sweeps with file-based JSON export serialize all completed benchmarks again after each command. A 512-command sweep therefore serializes 131,328 result entries. This adds an opt-in --export-json-lines FILE that writes 512 entries for the same sweep. Consumers can read each completed benchmark while the next one is running.

Each line is a compact schema-v2 JSON document with a single result, with the same fields as the snapshot export. The writer flushes after each line. The existing snapshot exporter keeps its current behavior. Both formats can be requested together using separate paths.

Measurements

End-to-end release builds on Apple Silicon, compared with master at c516152. Six paired trials per sweep size, alternating which binary ran first. Each variant spawned a real process twice. Timings below cover the entire invocation through the final write to a regular file.

Variants Snapshot JSON, median JSON Lines, median Median paired speedup
8 17.0 ms 16.3 ms 1.04×
512 1.799 s 0.893 s 2.03×

Reproduce the larger workload with the baseline binary using the first command and the proposed binary using the second.

hyperfine --style=none -r 2 -P i 1 512 --export-json snapshot.json '/usr/bin/true {i}'
hyperfine --style=none -r 2 -P i 1 512 --export-json-lines records.jsonl '/usr/bin/true {i}'

This measures exporter overhead on short commands. In a four-variant Python top-k retrieval sweep (two runs per variant), both formats took about 0.46 s. With JSON Lines as the only exporter, the scheduler retains the first benchmark for comparisons and releases later measurements after flushing. In a real sweep of 128 commands with 64 process launches each, median peak RSS across three executions dropped from 8.00 to 6.86 MiB compared with the first JSON Lines commit. An isolated deterministic executor sweep (512 variants, 128 samples each) went from 11.47 to 2.72 MiB. Parameter expansion still allocates the command list up front.

Verification

133 tests passed locally, including a regression test comparing both exports from the same run after a later command fails. The test checks that old contents are truncated and that completed records retain their order, measurements and statistical summaries. Release build, rustfmt and clippy also passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant