Skip to content

test(eval): run one trial end to end - #439

Open
gmemuriuki wants to merge 3 commits into
mainfrom
feat/agent-eval-trial-driver
Open

gmemuriuki wants to merge 3 commits into
mainfrom
feat/agent-eval-trial-driver

Conversation

@gmemuriuki

@gmemuriuki gmemuriuki commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

The evaluator has the individual pieces of a trial, but nothing yet runs them in the production order.

This adds the trial driver. It allocates the result directory first, prepares the project, runs Symposium setup, checks the environment, launches the client, gathers evidence, grades the submission, and writes
result.json. Failures after allocation are recorded in that result instead of losing the trial altogether.

A client counts as launched only after its matching session introduction is observed. A wrapper that exits before then is a setup failure in both the baseline and Symposium arms. Damaged or foreign transcript evidence remains unreadable or mismatched rather than being mistaken for a client
that never launched.

Portable tests exercise every failure phase and the launch boundary. A real-SRT test runs the whole journey with a staged stand-in client that executes the hooks registered by Symposium.

Stack: #436 → #437 → #438 → #439 → #441 → #442 → #443

Disclosure questions

AI disclosure.

  • The AI tool authored large parts of the code

Questions for reviewers.

Do the phase boundaries and failure classifications match how we will want to diagnose a trial?

self.provenance.deadlines.agent,
);
let changes = launch.snapshot.changes();
if matches!(execution.ended, Ended::NotLaunched(_)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Baseline & Symsposium cases where something fails on bootstraping are taken into account differently, so the result is not the same. No preferences about which one should we use, just that we should converge.

Claude reproduction: If the sandbox fails to start, the baseline counts it as a failed session but the Symposium arm excludes it.

At the top of the stack, under real srt, a symlink left at /tmp/symposium-srt after preflight made bwrap fail with Destination is not a directory. The baseline trial recorded failed (session). The Symposium trial recorded setup_failure, same error. A claude missing inside the sandbox (exit 127) goes the same way.

NotLaunched only comes from ProcessError::Spawn, which with the srt backend means the outer bwrap couldn't be executed at all. If srt or bwrap starts and then fails, we get a non-zero exit and an empty transcript, which ends up as Ended::Failed(Exited) and counts in the success rate. The Symposium arm runs init in the same sandbox first, so the same fault becomes a setup failure there. Infrastructure problems after preflight therefore only lower the baseline, which skews the delta toward Symposium. The RFD counts "the trial never launched its client" as a setup failure, so treating a session with no system/init line as not launched, or probing the sandbox in ready() for both arms, would match it.

@gmemuriuki gmemuriuki self-assigned this Nov 10, 2026
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-trial-driver branch from fa67504 to b255b32 Compare November 10, 2026 08:59
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-trial-driver branch 2 times, most recently from d7c7765 to 70da744 Compare November 16, 2026 12:15
@jackh726
jackh726 force-pushed the feat/agent-eval-trial-driver branch 2 times, most recently from 4da58c8 to 3cb9ce0 Compare November 16, 2026 14:55
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-trial-driver branch from 3cb9ce0 to 8567019 Compare November 16, 2026 16:59
Base automatically changed from feat/agent-eval-grading-sandbox to main November 16, 2026 17:24
Create a trial's empty root before filling it, so a trial whose
preparation fails still has a place for its result. Only a prepared
Trial can be set up, launched or graded; a failed one comes back as
NotPrepared with its root kept.

Foreign inputs are still refused before anything is created, and
create_trial stays as the two steps together.
Record a client that could not be launched as reaching launch: its
session is kept, ending not_launched with the spawn error, nothing is
graded, and the outcome is a setup failure. A result refuses an
ungraded session whose client launched, and a graded one whose client
never did.

Time the agent around the backend's run alone, so transcript reading
is not agent time, and let each ClientRunner name its client.
@jackh726
jackh726 force-pushed the feat/agent-eval-trial-driver branch from 8567019 to 0e373d9 Compare November 16, 2026 17:24
@jackh726

Copy link
Copy Markdown
Contributor

Another review from Codex:

Strongly consider fixing

Symposium-owned skills can leak into the grading submission

phases.rs:116

copies is derived from facts.installs, but that collection contains only skills named in the installed or not_installed expectations. A valid task can name a skill only in used, and plugins may provide additional unnamed skills; Symposium can still install those copies.

Consequently, those Symposium-created files are treated as agent edits. They are falsely reported under edits_outside_editable, or—if .claude/** is editable—imported into the grading copy, potentially affecting the grade.

I’d derive ownership from every skill the task’s plugins can install, independently of installation assertions, and add an end-to-end case with a used-only skill and .claude/** editable.

Run a trial from its allocated root to a written result.json:
prepare, set up Symposium, check before launch, run the session,
gather Symposium's evidence, grade without its own skill copies, and
decide expectations. Only allocation, parts that cannot make one
result, or a result that cannot be written stop the runner.

Discover ownership from every skill the task's plugins provide,
independently of installation and usage assertions. Compare
candidate copies against trusted sources and require exact content
and generated metadata. Keep used-only skills and unnamed siblings
out of grading even when their paths are editable; altered copies
remain agent edits.

Run every command through a private Sandbox, implemented only by srt
outside tests, and record a grading sandbox that cannot be prepared
as a grading exclusion. Take the client and model from the runner
alone. A Node stand-in for Claude Code runs whole trials under real
srt.

Co-authored-by: Codex <codex@openai.com>
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-trial-driver branch from 0e373d9 to a63cc4c Compare November 17, 2026 06:27
@gmemuriuki

Copy link
Copy Markdown
Contributor Author

Done.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants