Repository navigation
Add incremental JSON Lines export for parameter sweeps - #1032
Open
Supersynergy wants to merge 3 commits into
Open
Supersynergy wants to merge 3 commits into
Supersynergy wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For the Hyperfine 2.1 milestone.
Large parameter sweeps with file-based JSON export serialize all completed benchmarks again after each command. A 512-command sweep therefore serializes 131,328 result entries. This adds an opt-in
--export-json-lines FILEthat writes 512 entries for the same sweep. Consumers can read each completed benchmark while the next one is running.Each line is a compact schema-v2 JSON document with a single result, with the same fields as the snapshot export. The writer flushes after each line. The existing snapshot exporter keeps its current behavior. Both formats can be requested together using separate paths.
Measurements
End-to-end release builds on Apple Silicon, compared with master at c516152. Six paired trials per sweep size, alternating which binary ran first. Each variant spawned a real process twice. Timings below cover the entire invocation through the final write to a regular file.
Reproduce the larger workload with the baseline binary using the first command and the proposed binary using the second.
This measures exporter overhead on short commands. In a four-variant Python top-k retrieval sweep (two runs per variant), both formats took about 0.46 s. With JSON Lines as the only exporter, the scheduler retains the first benchmark for comparisons and releases later measurements after flushing. In a real sweep of 128 commands with 64 process launches each, median peak RSS across three executions dropped from 8.00 to 6.86 MiB compared with the first JSON Lines commit. An isolated deterministic executor sweep (512 variants, 128 samples each) went from 11.47 to 2.72 MiB. Parameter expansion still allocates the command list up front.
Verification
133 tests passed locally, including a regression test comparing both exports from the same run after a later command fails. The test checks that old contents are truncated and that completed records retain their order, measurements and statistical summaries. Release build, rustfmt and clippy also passed.