Repository navigation
test(eval): run one trial end to end - #439
gmemuriuki wants to merge 3 commits into
Conversation
| self.provenance.deadlines.agent, | ||
| ); | ||
| let changes = launch.snapshot.changes(); | ||
| if matches!(execution.ended, Ended::NotLaunched(_)) { |
There was a problem hiding this comment.
Baseline & Symsposium cases where something fails on bootstraping are taken into account differently, so the result is not the same. No preferences about which one should we use, just that we should converge.
Claude reproduction:
If the sandbox fails to start, the baseline counts it as a failed session but the Symposium arm excludes it.At the top of the stack, under real srt, a symlink left at /tmp/symposium-srt after preflight made bwrap fail with Destination is not a directory. The baseline trial recorded failed (session). The Symposium trial recorded setup_failure, same error. A claude missing inside the sandbox (exit 127) goes the same way.
NotLaunched only comes from ProcessError::Spawn, which with the srt backend means the outer bwrap couldn't be executed at all. If srt or bwrap starts and then fails, we get a non-zero exit and an empty transcript, which ends up as Ended::Failed(Exited) and counts in the success rate. The Symposium arm runs init in the same sandbox first, so the same fault becomes a setup failure there. Infrastructure problems after preflight therefore only lower the baseline, which skews the delta toward Symposium. The RFD counts "the trial never launched its client" as a setup failure, so treating a session with no system/init line as not launched, or probing the sandbox in ready() for both arms, would match it.
fa67504 to
b255b32
Compare
d7c7765 to
70da744
Compare
4da58c8 to
3cb9ce0
Compare
3cb9ce0 to
8567019
Compare
Create a trial's empty root before filling it, so a trial whose preparation fails still has a place for its result. Only a prepared Trial can be set up, launched or graded; a failed one comes back as NotPrepared with its root kept. Foreign inputs are still refused before anything is created, and create_trial stays as the two steps together.
Record a client that could not be launched as reaching launch: its session is kept, ending not_launched with the spawn error, nothing is graded, and the outcome is a setup failure. A result refuses an ungraded session whose client launched, and a graded one whose client never did. Time the agent around the backend's run alone, so transcript reading is not agent time, and let each ClientRunner name its client.
8567019 to
0e373d9
Compare
|
Another review from Codex: Strongly consider fixingSymposium-owned skills can leak into the grading submission
Consequently, those Symposium-created files are treated as agent edits. They are falsely reported under I’d derive ownership from every skill the task’s plugins can install, independently of installation assertions, and add an end-to-end case with a |
Run a trial from its allocated root to a written result.json: prepare, set up Symposium, check before launch, run the session, gather Symposium's evidence, grade without its own skill copies, and decide expectations. Only allocation, parts that cannot make one result, or a result that cannot be written stop the runner. Discover ownership from every skill the task's plugins provide, independently of installation and usage assertions. Compare candidate copies against trusted sources and require exact content and generated metadata. Keep used-only skills and unnamed siblings out of grading even when their paths are editable; altered copies remain agent edits. Run every command through a private Sandbox, implemented only by srt outside tests, and record a grading sandbox that cannot be prepared as a grading exclusion. Take the client and model from the runner alone. A Node stand-in for Claude Code runs whole trials under real srt. Co-authored-by: Codex <codex@openai.com>
0e373d9 to
a63cc4c
Compare
|
Done. |
What does this PR do?
The evaluator has the individual pieces of a trial, but nothing yet runs them in the production order.
This adds the trial driver. It allocates the result directory first, prepares the project, runs Symposium setup, checks the environment, launches the client, gathers evidence, grades the submission, and writes
result.json. Failures after allocation are recorded in that result instead of losing the trial altogether.
A client counts as launched only after its matching session introduction is observed. A wrapper that exits before then is a setup failure in both the baseline and Symposium arms. Damaged or foreign transcript evidence remains unreadable or mismatched rather than being mistaken for a client
that never launched.
Portable tests exercise every failure phase and the launch boundary. A real-SRT test runs the whole journey with a staged stand-in client that executes the hooks registered by Symposium.
Stack: #436 → #437 → #438 → #439 → #441 → #442 → #443
Disclosure questions
AI disclosure.
Questions for reviewers.
Do the phase boundaries and failure classifications match how we will want to diagnose a trial?