Repository navigation
Conversation
jat255
added
py
jat255
added this pull request to stack #416
October 10, 2026 01:34
Contributor
|
Preview root: https://posit-dev.github.io/commons/pr-415/ Python site preview: https://posit-dev.github.io/commons/pr-415/py/ Built from the latest commit on this branch. The R links in it point at the published R site, which no pull request rebuilds. |
On macOS, Python 3.14's asyncio treats waitid()'s report of a stopped child as an exit and calls a blocking waitpid() on the event-loop thread, so model code running os.kill(os.getpid(), signal.SIGSTOP) froze the host until the worker really exited, which nothing could then make it do. Each local worker session now starts a thread that waits for the worker to stop or exit without reaping it, and SIGKILLs the process group if it stopped. The blocked waitpid() returns at once and the driver reports a crash. This runs on every platform with os.waitid, so a stopped worker is reported the same way everywhere rather than as a call timeout on Linux. macOS before 3.13 has no os.waitid, and keeps the timeout; its asyncio does not freeze. The write-bound test used SIGSTOP to make the worker stop reading; it now uses a stand-in worker that announces itself ready and never reads.
jat255
force-pushed
the
jat255/d5sr-stopped-worker-freeze
branch
from
October 10, 2026 01:51
adf4cf8 to
c705930
Compare
jat255
marked this pull request as ready for review
October 10, 2026 01:51
jat255
removed
the
needs-manual-review
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR stops model code from freezing the host by stopping its own worker (
os.kill(os.getpid(), signal.SIGSTOP)). Each local worker now gets a thread that kills the worker's process group when it stops, and the call reports a crash.Stacked on #411.
Agent-written details
Why the host froze. On macOS, Python 3.14's asyncio treats
waitid()'s report of a stopped child as an exit. It then calls a blockingwaitpid()on the event-loop thread, which waits until the child really exits. Killing the stopped worker ends that wait. The reasoning is in the docstring of_kill_when_stopped(diff).Same behaviour on every platform. The guard runs wherever
os.waitidexists (Linux, and macOS from 3.13). A stopped worker is therefore reported as a crash on every platform, rather than as a call timeout on Linux. macOS on 3.11 and 3.12 has noos.waitid, so there a stopped worker still runs into the call timeout. asyncio on those versions does not freeze.Rejected alternatives:
Tests. The write-bound test used SIGSTOP to make the worker stop reading. It now uses a stand-in worker that announces it's ready and never reads stdin. The new self-stop test hangs on 3.14 without the fix and passes with it. ruff, pyrefly, and the full suite pass, and the driver tests also pass on 3.14.
kata: d5sr