Skip to content

Fix catastrophic performance - #1087

Open
Dreamsorcerer wants to merge 7 commits into
pyinvoke:mainfrom
Dreamsorcerer:patch-2
Open

Dreamsorcerer wants to merge 7 commits into
pyinvoke:mainfrom
Dreamsorcerer:patch-2

Conversation

@Dreamsorcerer

@Dreamsorcerer Dreamsorcerer commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Fixes #819.

Without patching invoke, no matter how much hardware is thrown at it, it will never exceed a throughput of 100 B/s (yes, bytes). This is a multi-pronged approach to fix the problem.

  • When the output is not an actual terminal, use the chunk size instead of falling back to 1 byte chunks.
  • If we received data, don't sleep for 10ms between chunks when there's probably more data to be processed.
  • Increase the default chunk size to 64 KiB, which better matches other typical default sizes.
  • Add a chunk size parameter to allow the user to use larger chunk sizes.

The first 3 points improve performance from 100 B/s to atleast comfortably over 10 MiB/s. Using a custom 2 MiB chunk size via point 4, I was able to get an extra ~6% performance on top of that.


Combined with #915, this will finally allow me to stop patching invoke and just use it as installed library.

Comment thread invoke/runners.py Outdated
Comment thread invoke/runners.py Outdated
Comment thread tests/runners.py Outdated
Co-authored-by: Sam Bull <aa6bs0@sambull.org>
Comment thread tests/runners.py
Comment thread invoke/runners.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Awful performance due to reading bytes 1 at a time

1 participant