Skip to content

test(eval): give srt a short runtime directory - #442

Open
gmemuriuki wants to merge 1 commit into
feat/agent-eval-retentionfrom
fix/agent-eval-srt-runtime-dir
Open

gmemuriuki wants to merge 1 commit into
feat/agent-eval-retentionfrom
fix/agent-eval-srt-runtime-dir

Conversation

@gmemuriuki

@gmemuriuki gmemuriuki commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

SRT creates its proxy sockets below TMPDIR. When TMPDIR is inside a deeply nested trial directory, the socket path can exceed Unix's limit and every grading command fails before it starts.

This gives each sandbox a short, private runtime directory under /tmp and removes it when the command finishes. The evaluation directory can therefore live at an ordinary user-selected path without affecting SRT's socket paths.

The current runner is sequential. The implementation documents that assumption because concurrent sandboxes will need stronger separation between their runtime directories.

Stack: #436 → #437 → #438 → #439 → #441 → #442 → #443

Disclosure questions

AI disclosure.

  • The AI tool authored large parts of the code

Questions for reviewers.

Does the runtime-directory lifecycle make the present sequential assumption clear enough?

@gmemuriuki
gmemuriuki force-pushed the fix/agent-eval-srt-runtime-dir branch from fe488e7 to 126a625 Compare November 16, 2026 11:31
@gmemuriuki
gmemuriuki force-pushed the fix/agent-eval-srt-runtime-dir branch from 126a625 to db85f75 Compare November 16, 2026 12:15
@jackh726
jackh726 force-pushed the fix/agent-eval-srt-runtime-dir branch 2 times, most recently from 46b1101 to 4f9c0e5 Compare November 16, 2026 14:55
@gmemuriuki
gmemuriuki force-pushed the fix/agent-eval-srt-runtime-dir branch from 4f9c0e5 to 4f47d2e Compare November 16, 2026 16:59
Bind each sandbox's own tmp/ at /tmp/symposium-srt in the outer bwrap
and make that srt's TMPDIR, so its proxy sockets fit Linux's 107-byte
limit however deep the evaluation root. Each namespace maps the alias
to its own sandbox, so concurrent sandboxes never share it.

Run the command through env with its sandbox's tmp/ as TMPDIR, over
the /tmp/claude srt would set, and drop the evaluation root length
check.
@jackh726
jackh726 force-pushed the fix/agent-eval-srt-runtime-dir branch from 4f47d2e to a1c94d7 Compare November 16, 2026 17:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants