Skip to content

test(eval): run one evaluation batch - #443

Open
gmemuriuki wants to merge 2 commits into
fix/agent-eval-srt-runtime-dirfrom
feat/agent-eval-command
Open

gmemuriuki wants to merge 2 commits into
fix/agent-eval-srt-runtime-dirfrom
feat/agent-eval-command

Conversation

@gmemuriuki

@gmemuriuki gmemuriuki commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

The evaluator can run one trial internally, but there is no command that turns a task into a complete evaluation batch.

This adds cargo xtask agent-eval. Before launching a client, the command checks the credential, SRT installation, staged Claude Code and cargo-agents binaries, and every task's grading preparation under the production sandbox. It also probes the staged-and-renamed publication path used for retained results.

A regression task runs the Symposium arm of round 1. A capability task runs baseline and Symposium arms that both belong to round 1. Round identity is represented separately from execution order so later
alternation cannot break the pairing.

The command tests cover preflight failures, staging, both task kinds, and complete batches under real SRT with the stand-in client. A live Claude Code run still requires credentials supported by the runner and is not claimed by these deterministic tests.

Stack: #436 → #437 → #438 → #439 → #441 → #442 → #443

Disclosure questions

AI disclosure.

  • The AI tool authored large parts of the code

Questions for reviewers.

Is the command doing enough validation before it begins a paid or long-running batch?

let staged = stage_tools(&mut batch, request, checkout, &host.path)?;
let build = symposium::recorded(&staged.cargo_agents, staged.symposium_revision, &scratch)?;
let claude = version::preflight(&batch, &staged.claude, &scratch)?;
let inputs = batch.snapshot(&checkout.evals, &request.task)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there're some missing stuff to check before running the session and failing. I've tested this with an environment without fmt and says that Symposium failed which is kind of strange.

Claude reproduction: Some things grading depends on are only checked after the paid session.

Two ways to see it, both run under real srt with the stand-in client:

  • With the default eval root /tmp/symposium-eval, create /tmp/.cargo/config.toml and run a batch. Preparation passes, the session runs, then grading refuses because Cargo would read that config. The trial is a grading_exclusion and the command exits 0.
  • Use a toolchain without rustfmt (e.g. rustup --profile minimal). The session completes, the format-on-stop hook fails with no such command: fmt, and grading fails at cargo fmt --version. Since formatting is a regression task, the failed hook expectation wins and the trial is reported failed (expectation), so a machine problem shows up as a Symposium regression.

prepare checks the credential, srt, staging, versions and the task, but cargo fmt --version, the ambient Cargo config check and the starter's cargo metadata only run inside grading, after execute. results/ is also first created by Keeper::new after trial 1. Running those model-free checks once here, and creating the results directory up front, would catch all of these before any paid session.

@gmemuriuki gmemuriuki self-assigned this Nov 9, 2026
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-command branch from 55c660f to 6a605f3 Compare November 10, 2026 09:00
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-command branch from 6a605f3 to 9679c36 Compare November 16, 2026 11:31
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-command branch from 9679c36 to e59759c Compare November 16, 2026 12:15
@jackh726
jackh726 force-pushed the feat/agent-eval-command branch 2 times, most recently from 8bc2bf3 to dec8412 Compare November 16, 2026 14:55
@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-command branch from dec8412 to 89e64d3 Compare November 16, 2026 16:59
Add the agent-eval command that prepares one batch and runs the task's
required arms through the existing trial driver.

Stage exact tool copies and preflight credentials, srt, Claude Code,
the cargo-agents build, and every task's grading preparation before
launching a client. Keep each named preflight workspace and its logs.

Probe staged result publication before spending model tokens. Retain
each completed result and record the provenance needed to reproduce it.
Install the deterministic Claude stand-in the way npm installs the
real client, then run regression and capability tasks through the
command path.

Check the successful regression result and prove that both capability
arms are retained as one round with matching result metadata.
@jackh726
jackh726 force-pushed the feat/agent-eval-command branch from 89e64d3 to 7286247 Compare November 16, 2026 17:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants