Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion md/design/agent-eval.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,9 @@ A `Batch` is one run of the runner: `<eval-root>/<batch>/`, holding the staged t
- **Environment.** `Trial::environment` is the whole environment of a trial's processes: `HOME`, `PATH`, `TMPDIR`, `CARGO_HOME`, `EVAL_EVIDENCE_DIR`, `SYMPOSIUM_HOOK_TRACE` (`evidence/hook-trace.jsonl`), and the API key. `PATH` is the trial's `bin/`, the toolchain's `bin/`, then `/usr/bin` and `/bin`. The real key is set on `srt` only, which masks it. The key's `Debug` never prints it, and neither does `Invocation`'s, which lists the environment's names only, since the key travels on as a plain value.
- **Launching (Linux only).** `Srt` names the Node that runs `srt` and its `dist/cli.js`, since `srt` finds Node through `env node`, which the trial's `PATH` may not satisfy. `Srt::preflight` runs before a batch, so an unsuitable machine fails setup rather than the first trial: Node must be 22.12 or later, `srt` the pinned `SRT_VERSION`, and `true` must run through the whole stack (outer `bwrap`, Node, `srt`, a policy file with the same network proxy and TLS termination as a trial's), which proves `srt`'s own dependencies and the nested namespace without this code listing them. A tool that cannot start, answer in time, or exit successfully is reported by name, apart from one that reports the wrong version. `Srt::for_trial` refuses a trial whose `srt.json` has not been written. `Sandboxed` is the `grade::process::Backend` for one trial: it wraps a command as `srt --settings <run>/srt.json -- <program> <args>` and hands it to `Local`, which keeps the deadline, kills the process tree, and logs. `srt` runs the command through `bash -c` with `srt`'s own environment, so the trial's environment is set on `srt`. An outer `bwrap --dev-bind / / --dir /tmp/claude --bind <run>/tmp /tmp/claude` gives each trial its own `/tmp/claude` and changes nothing else; `srt` builds its sandbox inside, with its own PID namespace, so killing the tree reaches processes that left the process group. Three ignored tests run real `srt` under the generated policy: one trial cannot see another's temporary files, the real API key appears in no environment the sandbox can read, and a process that calls `setsid` still dies at the deadline. Run them on Linux with `cargo test -p symposium-eval -- --ignored`.
- **Smoke test.** `run::smoke` is RFD step 2's no-model smoke test under real `srt` and the generated policy, one `sh` script per trial that stops at the first failed check and names it. Each trial builds and tests its project offline; cannot read a home (a shell profile, and `~/.cargo/bin/cargo-agents`, also run by absolute path), a hidden check, a solution, the checkout's manifest, or another trial; and leaves its fixtures, its policy, the staged `cargo-agents`, the registry, and another trial's project unchanged. Only a Symposium trial can run the staged `cargo-agents`. The home is a decoy the test creates and adds to the denied user data roots, so the test does not depend on what the machine's home holds; the machine's own files are checked too when they exist. Paths reach the script as arguments, so any name works. Writes are checked on the host, not inside the sandbox, because `srt` covers a read-denied directory with an empty writable tmpfs, so a write there succeeds and is thrown away. Its tools are staged the way the runner stages them, and run by name from each trial's `bin/`: the machine's `node`, a small npm-style package run through that staged `node`, and a stand-in for `cargo-agents`, which a baseline trial cannot find on its `PATH`. Staging and launching the real clients and `cargo-agents` with `--version` comes with the adapters.
- **Running a trial.** `run::trial::Driver::run(number, arm, runner)` runs one trial end to end and writes its `result.json` in the trial's root, never over an existing one. It allocates the root first, so every later failure has a place to be recorded; then prepares the trial, writes its policy, sets up Symposium (Symposium trials only), snapshots the config, runs the pre-launch checks, and runs the session (`client::execute`). A failure before launch is `Reached::Setup` with its step; a client that never launched is `Reached::Session`, ungraded. Otherwise, for a Symposium trial, `expect::decide::gather` reads Symposium's evidence before grading, since `expect::ownership::symposium_copies` needs its installs to leave Symposium's own skill copies out of the agent's edits (`grade::Submitted::trial`); then grading runs, and `decide` decides the expectations. A grading sandbox that cannot be prepared is `GradeError::Sandbox`, so the trial is a grading exclusion. Only an allocation failure, parts that cannot make one result (`ResultError`), or a result that cannot be written are the runner's error (`TrialError`). The trial's client and requested model are named by `runner.client()` and `runner.model()` alone, so a result cannot record a model other than the one launched. Commands run through a private `Sandbox` trait (`for_trial`, `for_grading`), implemented only by `Srt` outside tests, so no run can go unconfined; tests play `init` and the client through their own. Two ignored Linux tests each run a `formatting` trial under real `srt`, with the `cargo-agents` built from the checkout and a Node stand-in for Claude Code (`claude-stand-in.js`), staged as an npm package and read by the real adapter: it runs the hooks `init` registered, and leaves `double` unformatted, so only the Symposium trial, whose `format-on-stop` hook formats it, succeeds.
- **Running a trial.** `run::trial::Driver::run(round, arm, runner)` runs one trial end to end and writes its `result.json` in the trial's root, never over an existing one. It allocates the root first, so every later failure has a place to be recorded; then prepares the trial, writes its policy, sets up Symposium (Symposium trials only), snapshots the config, runs the pre-launch checks, and runs the session (`client::execute`). A failure before launch is `Reached::Setup` with its step; a client that never launched is `Reached::Session`, ungraded. Otherwise, for a Symposium trial, `expect::decide::gather` reads Symposium's evidence before grading, since `expect::ownership::symposium_copies` needs its installs to leave Symposium's own skill copies out of the agent's edits (`grade::Submitted::trial`); then grading runs, and `decide` decides the expectations. A grading sandbox that cannot be prepared is `GradeError::Sandbox`, so the trial becomes a grading exclusion. Only an allocation failure, parts that cannot make one result (`ResultError`), or a result that cannot be written are the runner's error (`TrialError`). The trial's client and requested model are named by `runner.client()` and `runner.model()` alone, so a result cannot record a model other than the one launched. Commands run through a private `Sandbox` trait (`for_trial`, `for_grading`), implemented only by `Srt` outside tests, so no run can go unconfined; tests play `init` and the client through their own. Two ignored Linux tests each run a `formatting` trial under real `srt`, with the `cargo-agents` built from the checkout and a Node stand-in for Claude Code (`claude-stand-in.js`), staged as an npm package and read by the real adapter: it runs the hooks `init` registered, and leaves `double` unformatted, so only the Symposium trial, whose `format-on-stop` hook formats it, succeeds.
- **Retention.** `Driver::run` returns a `Done`: the canonical evaluation-root anchor, the trial's root and configuration, whether every process tree was stopped, and, once the client launched, its transcript, standard error, and last response (`Execution::stderr` names the second). `Done::retain(destination)` copies what the RFD's "Retained results" lists into a destination that must not exist yet: `result.json`, `srt.json`, the project as `submission/` without `target/`, grading's logs as `grading/`, `transcript.jsonl`, `stderr.log`, and `response.txt`, plus, for a Symposium trial, `hook-trace.jsonl`, `evidence/`, and `symposium.log`, Symposium's per-process logs joined in name order, each under a `== <file> ==` header. A part the trial never reached is absent. The agent could write much of this, so every source path is lexical and relative to the trial, and the anchor, trial root, every parent, and every recursive entry are inspected without following links immediately before a read. A symlink, Windows reparse point, special file, or optional tree that cannot be inspected is left out and named in `retention-skipped.txt`; only `NotFound` means missing. A name that is not UTF-8 stops its whole tree from being listed. Retention refuses to read after an unsupervised session or any setup, session, or grading timeout whose post-deadline work did not prove the process tree stopped. Grading checks both its outer infrastructure error and every completed step in a full or partial report, so a later error cannot hide an earlier uncertain timeout; a timeout with no post-deadline error remains safe to retain. The anchor is canonical and outside the sandbox's writable namespace; `srt` denies the evaluation root and reopens only the trial directories it owns, so the submission cannot replace the anchor or its ancestors. These checks prevent symlink and reparse traversal. They do not establish physical ownership against hard links or mount points: the sandbox prevents submissions from reaching a host file from which to make a hard link and gives them no mount capability. What a reached phase always leaves is required, and a missing, linked, or non-regular one is an error: `result.json`, which a `Done` promises was written, and, once the client launched, its transcript and standard error, which the backend creates as it starts the client. The joined Symposium log ends each file with a newline, so every header starts its own line. Everything is copied into a temporary sibling (`.retaining-*`) that is renamed to the destination only once complete, so a failed retention leaves nothing that could be taken for a result, and can be retried.
- **The command.** `cargo xtask agent-eval <task> --client claude --model <model>` (`command`) runs one batch. A regression task runs round 1's Symposium trial; a capability task runs round 1's baseline trial followed by its Symposium trial. The plan represents the round separately from execution order, so later alternation cannot change which trials form a pair. Preparation checks the credential, isolation stack, staged Claude version, and `cargo-agents` build before launching an agent. It also prepares every distinct task through grading's production `srt` policy in a retained `preflight/<task>/` workspace: the trusted starter and fixtures are copied, both Cargo metadata checks run around placement of every hidden check, and a task with `fmt_check` runs `cargo fmt --version` from the same toolchain its hook and grader use. A failure stops the batch before any client launches, and its workspace and logs are not removed or reused. This proves that trusted grading preparation works; it cannot prove that a later submission will build or that every later check will finish. The command then probes the retained-results directory by writing a staging directory and atomically renaming it in place; later filesystem changes can still make retention fail. The batch is named by its UTC start time and random suffix and runs under `--eval-root`, or `<temp>/symposium-eval` by default. Each retained trial is written below `results/<task>/claude/<batch>/` and printed from its retained result. A trial outcome is not a command error; the command fails only when preparation fails or a trial has no result. Enforced trials require Linux. Ignored Linux tests run regression and capability batches with the Node stand-in installed as npm installs it.
- **macOS.** The RFD makes enforced mode the default on macOS too, but there trials would share `/tmp/claude`, and with no PID namespace a process that leaves the process group outlives the deadline. Until both are solved, the runner has no macOS backend.
- **Tests.** `Policy::can_read` and `can_write` model `srt`'s documented precedence (an `allowRead` reopens a less specific `denyRead`, never a more specific one), after Codex's `WritableRoot::is_path_writable`. The isolation invariants are tested through them: a trial cannot read another trial, the checkout, or a home; only a Symposium trial reads `tools/symposium`; a trial writes only its working directories. The policy tests never read the developer's home or the platform's roots: `Machine::with_exact_roots` takes them as arguments. The model leaves out `srt`'s built-in rules, which is exactly where the `/tmp/claude` gap lives, so these are the guarantees the policy is designed to give; only the no-model smoke test under `srt` (RFD step 2) shows they hold, and `run` lands with that caller.

Expand Down
1 change: 1 addition & 0 deletions symposium-eval/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ publish = false

[dependencies]
anyhow = "1"
chrono = "0.4"
clap = { version = "4", features = ["derive"] }
globset = "0.4"
proc-macro2 = { version = "1", features = ["span-locations"] }
Expand Down
8 changes: 8 additions & 0 deletions symposium-eval/src/client/claude.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ impl Claude {
/// # Errors
///
/// As [`Trial::exploratory_arguments`].
#[cfg_attr(
not(test),
expect(dead_code, reason = "--exploratory needs its own design")
)]
pub(crate) fn exploratory(
model: String,
version: Version,
Expand All @@ -73,6 +77,10 @@ impl Claude {
}
}

#[cfg_attr(
not(test),
expect(dead_code, reason = "only tests ask for the session")
)]
pub(crate) fn session(&self) -> Uuid {
self.session
}
Expand Down
8 changes: 8 additions & 0 deletions symposium-eval/src/client/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,10 @@ pub(crate) enum SessionIntroduction {
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) enum HookEvidence {
/// The client's output does not report hook runs.
#[cfg_attr(
not(test),
expect(dead_code, reason = "a client that reports no hooks comes with Codex")
)]
Unsupported,
Reported(HookRuns),
}
Expand Down Expand Up @@ -292,6 +296,10 @@ impl Summary {
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum Usage {
/// The client's output never reports usage.
#[cfg_attr(
not(test),
expect(dead_code, reason = "a client that reports no usage comes with Codex")
)]
Unsupported,
Reported(Tokens),
/// The client reports usage, but the transcript does not show this session's.
Expand Down
37 changes: 37 additions & 0 deletions symposium-eval/src/command/batch_name.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
//! A new batch's name: when it started, in UTC, so batches sort by time, and a few
//! random digits, so two started in the same second are unlikely to collide. A collision
//! is still safe: [`crate::run::Batch::create`] refuses to reuse an existing batch.

use chrono::{DateTime, Utc};
use uuid::Uuid;

use crate::task::BatchName;

/// A name such as `20261106T183005Z-a3f9`.
pub(crate) fn new_batch_name(now: DateTime<Utc>) -> BatchName {
let random = Uuid::new_v4().simple().to_string();
let name = format!("{}-{}", now.format("%Y%m%dT%H%M%SZ"), &random[..4]);
BatchName::new(name).expect("a timestamp and hex digits are a portable name")
}

#[cfg(test)]
mod tests {
use chrono::TimeZone;

use super::*;

#[test]
fn a_batch_name_contains_its_start_time_and_a_random_suffix() {
let now = Utc.with_ymd_and_hms(2026, 11, 6, 18, 30, 5).unwrap();

let name = new_batch_name(now);

let (time, random) = name.as_str().split_once('-').unwrap();
assert_eq!(time, "20261106T183005Z");
assert_eq!(random.len(), 4);
assert!(
random.chars().all(|digit| digit.is_ascii_hexdigit()),
"{name}"
);
}
}
Loading
Loading