Skip to content

dd: introduce fast-stream skip#12396

Closed
oech3 wants to merge 1 commit into
uutils:mainfrom
oech3:s-discard
Closed

dd: introduce fast-stream skip#12396
oech3 wants to merge 1 commit into
uutils:mainfrom
oech3:s-discard

Conversation

@oech3
Copy link
Copy Markdown
Contributor

@oech3 oech3 commented May 20, 2026

Assuming the situation downloading part of large file e.g.
curl * | dd skip=* bs=1M.

$ hyperfine "time head -c 1G /dev/zero |dd skip=1GiB bs=1M" "time head -c 1G /dev/zero |target/release/dd skip=1GiB bs=1M"
Benchmark 1: time head -c 1G /dev/zero |dd skip=1GiB bs=1M
  Time (mean ± σ):     407.0 ms ±  14.0 ms    [User: 10.6 ms, System: 788.1 ms]
  Range (min … max):   394.2 ms … 433.5 ms    10 runs
 
Benchmark 2: time head -c 1G /dev/zero |target/release/dd skip=1GiB bs=1M
  Time (mean ± σ):     235.7 ms ±   8.2 ms    [User: 3.2 ms, System: 451.5 ms]
  Range (min … max):   224.2 ms … 248.5 ms    13 runs
 
Summary
  time head -c 1G /dev/zero |target/release/dd skip=1GiB bs=1M ran
    1.73 ± 0.08 times faster than time head -c 1G /dev/zero |dd skip=1GiB bs=1M

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 20, 2026

GNU testsuite comparison:

Skip an intermittent issue tests/misc/io-errors (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/tail/tail-n0f is now passing!

@oech3 oech3 marked this pull request as ready for review May 20, 2026 02:52
@oech3 oech3 marked this pull request as draft May 20, 2026 02:52
@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented May 20, 2026

GNU test failed: tests/dd/no-allocate.

@collinfunk
Copy link
Copy Markdown

This violates POSIX requirements:

skip=n
Skip n input blocks (using the specified input block size) before starting to copy. On seekable files, the implementation shall read the blocks or seek past them; on non-seekable files, the blocks shall be read and the data shall be discarded.

@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented May 20, 2026

Cannot we consider splice length == block size?
[Edit] Actually, I was overreading (more than buf_size).

@oech3 oech3 force-pushed the s-discard branch 5 times, most recently from b97bcbc to b600b3e Compare May 20, 2026 05:20
@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented May 20, 2026

It seems tests/dd/no-allocate.sh does not allow skipping stream under ulimit with 0 allocation...

@oech3 oech3 closed this May 21, 2026
@oech3 oech3 deleted the s-discard branch May 21, 2026 06:50
@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented May 22, 2026

I don't think posix requires block should be existing on user space and read/write means read/write syscalls.
Also I think we can use sealed memfd for dd to exactly control block size. Rust std::io::copy's dev said splice to memfd is not zero-copy (thus, changing input during sinding data is safe), but still removes context switch.

Moreover, memfd is able to mmap and edit them for dd conv=.
@collinfunk ANy thought.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants