diff --git a/Cargo.lock b/Cargo.lock index 4b0c357f9..c473382b7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2636,6 +2636,7 @@ name = "symposium-eval" version = "0.1.0" dependencies = [ "anyhow", + "chrono", "clap", "globset", "insta", diff --git a/md/design/agent-eval.md b/md/design/agent-eval.md index 1a1d34ac3..645631402 100644 --- a/md/design/agent-eval.md +++ b/md/design/agent-eval.md @@ -94,8 +94,9 @@ A `Batch` is one run of the runner: `//`, holding the staged t - **Environment.** `Trial::environment` is the whole environment of a trial's processes: `HOME`, `PATH`, `TMPDIR`, `CARGO_HOME`, `EVAL_EVIDENCE_DIR`, `SYMPOSIUM_HOOK_TRACE` (`evidence/hook-trace.jsonl`), and the API key. `PATH` is the trial's `bin/`, the toolchain's `bin/`, then `/usr/bin` and `/bin`. The real key is set on `srt` only, which masks it. The key's `Debug` never prints it, and neither does `Invocation`'s, which lists the environment's names only, since the key travels on as a plain value. - **Launching (Linux only).** `Srt` names the Node that runs `srt` and its `dist/cli.js`, since `srt` finds Node through `env node`, which the trial's `PATH` may not satisfy. `Srt::preflight` runs before a batch, so an unsuitable machine fails setup rather than the first trial: Node must be 22.12 or later, `srt` the pinned `SRT_VERSION`, and `true` must run through the whole stack (outer `bwrap`, Node, `srt`, a policy file with the same network proxy and TLS termination as a trial's), which proves `srt`'s own dependencies and the nested namespace without this code listing them. A tool that cannot start, answer in time, or exit successfully is reported by name, apart from one that reports the wrong version. `Srt::for_trial` refuses a trial whose `srt.json` has not been written. `Sandboxed` is the `grade::process::Backend` for one trial: it wraps a command as `srt --settings /srt.json -- ` and hands it to `Local`, which keeps the deadline, kills the process tree, and logs. `srt` runs the command through `bash -c` with `srt`'s own environment, so the trial's environment is set on `srt`. An outer `bwrap --dev-bind / / --dir /tmp/claude --bind /tmp /tmp/claude` gives each trial its own `/tmp/claude` and changes nothing else; `srt` builds its sandbox inside, with its own PID namespace, so killing the tree reaches processes that left the process group. Three ignored tests run real `srt` under the generated policy: one trial cannot see another's temporary files, the real API key appears in no environment the sandbox can read, and a process that calls `setsid` still dies at the deadline. Run them on Linux with `cargo test -p symposium-eval -- --ignored`. - **Smoke test.** `run::smoke` is RFD step 2's no-model smoke test under real `srt` and the generated policy, one `sh` script per trial that stops at the first failed check and names it. Each trial builds and tests its project offline; cannot read a home (a shell profile, and `~/.cargo/bin/cargo-agents`, also run by absolute path), a hidden check, a solution, the checkout's manifest, or another trial; and leaves its fixtures, its policy, the staged `cargo-agents`, the registry, and another trial's project unchanged. Only a Symposium trial can run the staged `cargo-agents`. The home is a decoy the test creates and adds to the denied user data roots, so the test does not depend on what the machine's home holds; the machine's own files are checked too when they exist. Paths reach the script as arguments, so any name works. Writes are checked on the host, not inside the sandbox, because `srt` covers a read-denied directory with an empty writable tmpfs, so a write there succeeds and is thrown away. Its tools are staged the way the runner stages them, and run by name from each trial's `bin/`: the machine's `node`, a small npm-style package run through that staged `node`, and a stand-in for `cargo-agents`, which a baseline trial cannot find on its `PATH`. Staging and launching the real clients and `cargo-agents` with `--version` comes with the adapters. -- **Running a trial.** `run::trial::Driver::run(number, arm, runner)` runs one trial end to end and writes its `result.json` in the trial's root, never over an existing one. It allocates the root first, so every later failure has a place to be recorded; then prepares the trial, writes its policy, sets up Symposium (Symposium trials only), snapshots the config, runs the pre-launch checks, and runs the session (`client::execute`). A failure before launch is `Reached::Setup` with its step; a client that never launched is `Reached::Session`, ungraded. Otherwise, for a Symposium trial, `expect::decide::gather` reads Symposium's evidence before grading, since `expect::ownership::symposium_copies` needs its installs to leave Symposium's own skill copies out of the agent's edits (`grade::Submitted::trial`); then grading runs, and `decide` decides the expectations. A grading sandbox that cannot be prepared is `GradeError::Sandbox`, so the trial is a grading exclusion. Only an allocation failure, parts that cannot make one result (`ResultError`), or a result that cannot be written are the runner's error (`TrialError`). The trial's client and requested model are named by `runner.client()` and `runner.model()` alone, so a result cannot record a model other than the one launched. Commands run through a private `Sandbox` trait (`for_trial`, `for_grading`), implemented only by `Srt` outside tests, so no run can go unconfined; tests play `init` and the client through their own. Two ignored Linux tests each run a `formatting` trial under real `srt`, with the `cargo-agents` built from the checkout and a Node stand-in for Claude Code (`claude-stand-in.js`), staged as an npm package and read by the real adapter: it runs the hooks `init` registered, and leaves `double` unformatted, so only the Symposium trial, whose `format-on-stop` hook formats it, succeeds. +- **Running a trial.** `run::trial::Driver::run(round, arm, runner)` runs one trial end to end and writes its `result.json` in the trial's root, never over an existing one. It allocates the root first, so every later failure has a place to be recorded; then prepares the trial, writes its policy, sets up Symposium (Symposium trials only), snapshots the config, runs the pre-launch checks, and runs the session (`client::execute`). A failure before launch is `Reached::Setup` with its step; a client that never launched is `Reached::Session`, ungraded. Otherwise, for a Symposium trial, `expect::decide::gather` reads Symposium's evidence before grading, since `expect::ownership::symposium_copies` needs its installs to leave Symposium's own skill copies out of the agent's edits (`grade::Submitted::trial`); then grading runs, and `decide` decides the expectations. A grading sandbox that cannot be prepared is `GradeError::Sandbox`, so the trial becomes a grading exclusion. Only an allocation failure, parts that cannot make one result (`ResultError`), or a result that cannot be written are the runner's error (`TrialError`). The trial's client and requested model are named by `runner.client()` and `runner.model()` alone, so a result cannot record a model other than the one launched. Commands run through a private `Sandbox` trait (`for_trial`, `for_grading`), implemented only by `Srt` outside tests, so no run can go unconfined; tests play `init` and the client through their own. Two ignored Linux tests each run a `formatting` trial under real `srt`, with the `cargo-agents` built from the checkout and a Node stand-in for Claude Code (`claude-stand-in.js`), staged as an npm package and read by the real adapter: it runs the hooks `init` registered, and leaves `double` unformatted, so only the Symposium trial, whose `format-on-stop` hook formats it, succeeds. - **Retention.** `Driver::run` returns a `Done`: the canonical evaluation-root anchor, the trial's root and configuration, whether every process tree was stopped, and, once the client launched, its transcript, standard error, and last response (`Execution::stderr` names the second). `Done::retain(destination)` copies what the RFD's "Retained results" lists into a destination that must not exist yet: `result.json`, `srt.json`, the project as `submission/` without `target/`, grading's logs as `grading/`, `transcript.jsonl`, `stderr.log`, and `response.txt`, plus, for a Symposium trial, `hook-trace.jsonl`, `evidence/`, and `symposium.log`, Symposium's per-process logs joined in name order, each under a `== ==` header. A part the trial never reached is absent. The agent could write much of this, so every source path is lexical and relative to the trial, and the anchor, trial root, every parent, and every recursive entry are inspected without following links immediately before a read. A symlink, Windows reparse point, special file, or optional tree that cannot be inspected is left out and named in `retention-skipped.txt`; only `NotFound` means missing. A name that is not UTF-8 stops its whole tree from being listed. Retention refuses to read after an unsupervised session or any setup, session, or grading timeout whose post-deadline work did not prove the process tree stopped. Grading checks both its outer infrastructure error and every completed step in a full or partial report, so a later error cannot hide an earlier uncertain timeout; a timeout with no post-deadline error remains safe to retain. The anchor is canonical and outside the sandbox's writable namespace; `srt` denies the evaluation root and reopens only the trial directories it owns, so the submission cannot replace the anchor or its ancestors. These checks prevent symlink and reparse traversal. They do not establish physical ownership against hard links or mount points: the sandbox prevents submissions from reaching a host file from which to make a hard link and gives them no mount capability. What a reached phase always leaves is required, and a missing, linked, or non-regular one is an error: `result.json`, which a `Done` promises was written, and, once the client launched, its transcript and standard error, which the backend creates as it starts the client. The joined Symposium log ends each file with a newline, so every header starts its own line. Everything is copied into a temporary sibling (`.retaining-*`) that is renamed to the destination only once complete, so a failed retention leaves nothing that could be taken for a result, and can be retried. +- **The command.** `cargo xtask agent-eval --client claude --model ` (`command`) runs one batch. A regression task runs round 1's Symposium trial; a capability task runs round 1's baseline trial followed by its Symposium trial. The plan represents the round separately from execution order, so later alternation cannot change which trials form a pair. Preparation checks the credential, isolation stack, staged Claude version, and `cargo-agents` build before launching an agent. It also prepares every distinct task through grading's production `srt` policy in a retained `preflight//` workspace: the trusted starter and fixtures are copied, both Cargo metadata checks run around placement of every hidden check, and a task with `fmt_check` runs `cargo fmt --version` from the same toolchain its hook and grader use. A failure stops the batch before any client launches, and its workspace and logs are not removed or reused. This proves that trusted grading preparation works; it cannot prove that a later submission will build or that every later check will finish. The command then probes the retained-results directory by writing a staging directory and atomically renaming it in place; later filesystem changes can still make retention fail. The batch is named by its UTC start time and random suffix and runs under `--eval-root`, or `/symposium-eval` by default. Each retained trial is written below `results//claude//` and printed from its retained result. A trial outcome is not a command error; the command fails only when preparation fails or a trial has no result. Enforced trials require Linux. Ignored Linux tests run regression and capability batches with the Node stand-in installed as npm installs it. - **macOS.** The RFD makes enforced mode the default on macOS too, but there trials would share `/tmp/claude`, and with no PID namespace a process that leaves the process group outlives the deadline. Until both are solved, the runner has no macOS backend. - **Tests.** `Policy::can_read` and `can_write` model `srt`'s documented precedence (an `allowRead` reopens a less specific `denyRead`, never a more specific one), after Codex's `WritableRoot::is_path_writable`. The isolation invariants are tested through them: a trial cannot read another trial, the checkout, or a home; only a Symposium trial reads `tools/symposium`; a trial writes only its working directories. The policy tests never read the developer's home or the platform's roots: `Machine::with_exact_roots` takes them as arguments. The model leaves out `srt`'s built-in rules, which is exactly where the `/tmp/claude` gap lives, so these are the guarantees the policy is designed to give; only the no-model smoke test under `srt` (RFD step 2) shows they hold, and `run` lands with that caller. diff --git a/symposium-eval/Cargo.toml b/symposium-eval/Cargo.toml index fcdaf61aa..63366b05c 100644 --- a/symposium-eval/Cargo.toml +++ b/symposium-eval/Cargo.toml @@ -9,6 +9,7 @@ publish = false [dependencies] anyhow = "1" +chrono = "0.4" clap = { version = "4", features = ["derive"] } globset = "0.4" proc-macro2 = { version = "1", features = ["span-locations"] } diff --git a/symposium-eval/src/client/claude.rs b/symposium-eval/src/client/claude.rs index bf7a7e4e8..5b859932f 100644 --- a/symposium-eval/src/client/claude.rs +++ b/symposium-eval/src/client/claude.rs @@ -54,6 +54,10 @@ impl Claude { /// # Errors /// /// As [`Trial::exploratory_arguments`]. + #[cfg_attr( + not(test), + expect(dead_code, reason = "--exploratory needs its own design") + )] pub(crate) fn exploratory( model: String, version: Version, @@ -73,6 +77,10 @@ impl Claude { } } + #[cfg_attr( + not(test), + expect(dead_code, reason = "only tests ask for the session") + )] pub(crate) fn session(&self) -> Uuid { self.session } diff --git a/symposium-eval/src/client/mod.rs b/symposium-eval/src/client/mod.rs index 5648958a3..982314491 100644 --- a/symposium-eval/src/client/mod.rs +++ b/symposium-eval/src/client/mod.rs @@ -81,6 +81,10 @@ pub(crate) enum SessionIntroduction { #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) enum HookEvidence { /// The client's output does not report hook runs. + #[cfg_attr( + not(test), + expect(dead_code, reason = "a client that reports no hooks comes with Codex") + )] Unsupported, Reported(HookRuns), } @@ -292,6 +296,10 @@ impl Summary { #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum Usage { /// The client's output never reports usage. + #[cfg_attr( + not(test), + expect(dead_code, reason = "a client that reports no usage comes with Codex") + )] Unsupported, Reported(Tokens), /// The client reports usage, but the transcript does not show this session's. diff --git a/symposium-eval/src/command/batch_name.rs b/symposium-eval/src/command/batch_name.rs new file mode 100644 index 000000000..815eee0c0 --- /dev/null +++ b/symposium-eval/src/command/batch_name.rs @@ -0,0 +1,37 @@ +//! A new batch's name: when it started, in UTC, so batches sort by time, and a few +//! random digits, so two started in the same second are unlikely to collide. A collision +//! is still safe: [`crate::run::Batch::create`] refuses to reuse an existing batch. + +use chrono::{DateTime, Utc}; +use uuid::Uuid; + +use crate::task::BatchName; + +/// A name such as `20261106T183005Z-a3f9`. +pub(crate) fn new_batch_name(now: DateTime) -> BatchName { + let random = Uuid::new_v4().simple().to_string(); + let name = format!("{}-{}", now.format("%Y%m%dT%H%M%SZ"), &random[..4]); + BatchName::new(name).expect("a timestamp and hex digits are a portable name") +} + +#[cfg(test)] +mod tests { + use chrono::TimeZone; + + use super::*; + + #[test] + fn a_batch_name_contains_its_start_time_and_a_random_suffix() { + let now = Utc.with_ymd_and_hms(2026, 11, 6, 18, 30, 5).unwrap(); + + let name = new_batch_name(now); + + let (time, random) = name.as_str().split_once('-').unwrap(); + assert_eq!(time, "20261106T183005Z"); + assert_eq!(random.len(), 4); + assert!( + random.chars().all(|digit| digit.is_ascii_hexdigit()), + "{name}" + ); + } +} diff --git a/symposium-eval/src/command/claude.rs b/symposium-eval/src/command/claude.rs new file mode 100644 index 000000000..4720d94f0 --- /dev/null +++ b/symposium-eval/src/command/claude.rs @@ -0,0 +1,181 @@ +//! The `claude` to stage, as it is installed: a native install is one executable, and an +//! npm install is a link to a Node script inside its package, staged whole with the +//! `node` that runs it. + +use std::ffi::OsStr; +use std::fs::File; +use std::io::Read as _; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, bail}; + +use super::host; +use crate::run::stage::Tool; + +/// What staging `claude` takes. +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct ClaudeTools { + pub(crate) claude: Tool, + /// The `node` an npm install runs through. + pub(crate) node: Option, +} + +/// The `claude` on `path`, as it was installed. +pub(crate) fn claude_tools(path: &OsStr) -> anyhow::Result { + let installed = host::resolved(&host::require(path, "claude")?)?; + if !is_node_script(&installed)? { + return Ok(ClaudeTools { + claude: Tool::Executable(installed), + node: None, + }); + } + let Some(package) = package_root(&installed) else { + bail!( + "`{}` is a Node script outside any package", + installed.display() + ); + }; + let entry = installed + .strip_prefix(&package) + .expect("the package holds the script") + .to_owned(); + Ok(ClaudeTools { + claude: Tool::NodePackage { package, entry }, + node: Some(Tool::Executable(host::require(path, "node")?)), + }) +} + +/// Whether `file` starts with a `#!` line that runs `node`. +fn is_node_script(file: &Path) -> anyhow::Result { + let mut start = Vec::new(); + File::open(file) + .and_then(|opened| opened.take(256).read_to_end(&mut start)) + .with_context(|| format!("cannot read `{}`", file.display()))?; + let first_line = start + .split(|byte| *byte == b'\n') + .next() + .unwrap_or_default(); + Ok(first_line.starts_with(b"#!") && String::from_utf8_lossy(first_line).contains("node")) +} + +/// The nearest directory above `file` with a `package.json`. +fn package_root(file: &Path) -> Option { + file.ancestors() + .skip(1) + .find(|dir| dir.join("package.json").is_file()) + .map(Path::to_owned) +} + +#[cfg(all(test, unix))] +mod tests { + use std::ffi::OsStr; + use std::fs; + use std::os::unix::fs::symlink; + + use tempfile::TempDir; + + use super::*; + use crate::run::test_support::executable; + + /// A Claude Code installation laid out as the production installers lay it out. + struct ClaudeInstall { + _temp: TempDir, + bin: PathBuf, + target: PathBuf, + package: Option, + } + + impl ClaudeInstall { + fn native() -> Self { + let temp = TempDir::new().unwrap(); + let target = temp.path().join("versions/2.1.318"); + fs::create_dir_all(target.parent().unwrap()).unwrap(); + fs::write(&target, b"\x7fELF").unwrap(); + Self::link(temp, target, None) + } + + fn npm() -> Self { + let temp = TempDir::new().unwrap(); + let package = temp + .path() + .join("lib/node_modules/@anthropic-ai/claude-code"); + fs::create_dir_all(&package).unwrap(); + fs::write(package.join("package.json"), "{}").unwrap(); + let target = package.join("cli.js"); + executable(&target, "#!/usr/bin/env node\n"); + Self::link(temp, target, Some(package)) + } + + fn unpackaged_node_script() -> Self { + let temp = TempDir::new().unwrap(); + let target = temp.path().join("claude.js"); + executable(&target, "#!/usr/bin/env node\n"); + Self::link(temp, target, None) + } + + fn link(temp: TempDir, target: PathBuf, package: Option) -> Self { + let bin = temp.path().join("bin"); + fs::create_dir(&bin).unwrap(); + symlink(&target, bin.join("claude")).unwrap(); + executable(&bin.join("node"), "#!/bin/sh\n"); + Self { + _temp: temp, + bin, + target, + package, + } + } + + fn path(&self) -> &OsStr { + self.bin.as_os_str() + } + + fn resolved_target(&self) -> PathBuf { + fs::canonicalize(&self.target).unwrap() + } + + fn resolved_package(&self) -> PathBuf { + fs::canonicalize(self.package.as_ref().unwrap()).unwrap() + } + } + + #[test] + fn a_native_install_is_one_executable() { + let install = ClaudeInstall::native(); + + let tools = claude_tools(install.path()).unwrap(); + + assert_eq!( + tools, + ClaudeTools { + claude: Tool::Executable(install.resolved_target()), + node: None + } + ); + } + + #[test] + fn an_npm_install_is_its_package_and_node() { + let install = ClaudeInstall::npm(); + + let tools = claude_tools(install.path()).unwrap(); + + assert_eq!( + tools, + ClaudeTools { + claude: Tool::NodePackage { + package: install.resolved_package(), + entry: "cli.js".into(), + }, + node: Some(Tool::Executable(install.bin.join("node"))), + }, + ); + } + + #[test] + fn a_node_script_outside_a_package_is_refused() { + let install = ClaudeInstall::unpackaged_node_script(); + + assert!(claude_tools(install.path()).is_err()); + } +} diff --git a/symposium-eval/src/command/credential.rs b/symposium-eval/src/command/credential.rs new file mode 100644 index 000000000..9ae83d411 --- /dev/null +++ b/symposium-eval/src/command/credential.rs @@ -0,0 +1,76 @@ +//! The credential trials reach the model with: an API key or a subscription token from +//! the runner's environment, which `srt` masks so no trial can read it. + +use std::ffi::OsString; + +use thiserror::Error; + +use crate::run::ApiKey; + +/// The variables Claude Code reads a credential from. Exactly one must be set, so the +/// runner never guesses which the developer meant. +const CLAUDE_CREDENTIALS: [&str; 2] = ["ANTHROPIC_API_KEY", "CLAUDE_CODE_OAUTH_TOKEN"]; + +#[derive(Debug, Error, PartialEq, Eq)] +pub(crate) enum CredentialError { + #[error("set {} or {} for Claude Code", CLAUDE_CREDENTIALS[0], CLAUDE_CREDENTIALS[1])] + Missing, + #[error("both {} and {} are set; unset one", CLAUDE_CREDENTIALS[0], CLAUDE_CREDENTIALS[1])] + Ambiguous, +} + +/// The one credential `var` holds for Claude Code. An empty variable counts as unset. +pub(crate) fn claude(var: impl Fn(&str) -> Option) -> Result { + let mut set = CLAUDE_CREDENTIALS.into_iter().filter_map(|name| { + var(name) + .filter(|value| !value.is_empty()) + .map(|value| (name, value)) + }); + match (set.next(), set.next()) { + (Some((name, value)), None) => Ok(ApiKey::new(name, value)), + (None, _) => Err(CredentialError::Missing), + (Some(_), Some(_)) => Err(CredentialError::Ambiguous), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn credential_from(vars: &[(&str, &str)]) -> Result { + claude(|name| { + vars.iter() + .find(|(set, _)| *set == name) + .map(|(_, value)| OsString::from(value)) + }) + } + + #[test] + fn exactly_one_nonempty_credential_is_used() { + for name in CLAUDE_CREDENTIALS { + let credential = credential_from(&[(name, "secret")]).unwrap(); + + assert_eq!(credential.name(), name); + } + } + + #[test] + fn an_absent_or_empty_credential_is_missing() { + for vars in [&[][..], &[("ANTHROPIC_API_KEY", "")][..]] { + let error = credential_from(vars).unwrap_err(); + + assert_eq!(error, CredentialError::Missing, "{vars:?}"); + } + } + + #[test] + fn two_credentials_are_ambiguous() { + let error = credential_from(&[ + ("ANTHROPIC_API_KEY", "key"), + ("CLAUDE_CODE_OAUTH_TOKEN", "token"), + ]) + .unwrap_err(); + + assert_eq!(error, CredentialError::Ambiguous); + } +} diff --git a/symposium-eval/src/command/host.rs b/symposium-eval/src/command/host.rs new file mode 100644 index 000000000..5273d8a9b --- /dev/null +++ b/symposium-eval/src/command/host.rs @@ -0,0 +1,69 @@ +//! What the runner finds on the machine it runs on: tools on its `PATH`, and the active +//! Rust toolchain. + +use std::ffi::OsStr; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use anyhow::{Context, bail}; + +use crate::run::srt::Srt; + +/// The first file called `name` in a directory `path` lists. +pub(crate) fn on_path(path: &OsStr, name: &str) -> Option { + std::env::split_paths(path) + .map(|dir| dir.join(name)) + .find(|candidate| candidate.is_file()) +} + +/// The first file called `name` on `path`, or an error naming it. +pub(crate) fn require(path: &OsStr, name: &str) -> anyhow::Result { + on_path(path, name).with_context(|| format!("`{name}` is not on PATH")) +} + +/// The installed `srt`, with the `node` that runs it and the `bwrap` around it. npm +/// links `srt` on `PATH` to its package's `dist/cli.js`. +pub(crate) fn srt(path: &OsStr) -> anyhow::Result { + let cli = require(path, "srt")?; + let cli = std::fs::canonicalize(&cli) + .with_context(|| format!("cannot resolve `{}`", cli.display()))?; + Ok(Srt::new( + &require(path, "node")?, + &cli, + &require(path, "bwrap")?, + )?) +} + +/// The active toolchain's sysroot, as `rustc` reports it. +pub(crate) fn sysroot() -> anyhow::Result { + rustc(&["--print", "sysroot"]).map(PathBuf::from) +} + +/// `rustc --version`, recorded with each result. +pub(crate) fn toolchain() -> anyhow::Result { + rustc(&["--version"]) +} + +/// The operating system and architecture, as each result records them. +pub(crate) fn os() -> String { + format!("{} {}", std::env::consts::OS, std::env::consts::ARCH) +} + +fn rustc(args: &[&str]) -> anyhow::Result { + let output = Command::new("rustc") + .args(args) + .output() + .context("cannot run `rustc`")?; + if !output.status.success() { + bail!("`rustc {}` failed", args.join(" ")); + } + Ok(String::from_utf8(output.stdout) + .context("`rustc` printed something other than UTF-8")? + .trim() + .to_owned()) +} + +/// `path` resolved, so what is staged and hashed is the file a link names. +pub(crate) fn resolved(path: &Path) -> anyhow::Result { + std::fs::canonicalize(path).with_context(|| format!("cannot resolve `{}`", path.display())) +} diff --git a/symposium-eval/src/command/mod.rs b/symposium-eval/src/command/mod.rs new file mode 100644 index 000000000..7f5443363 --- /dev/null +++ b/symposium-eval/src/command/mod.rs @@ -0,0 +1,459 @@ +//! `cargo xtask agent-eval `: one batch of a task on one client and model (RFD, +//! "What happens when you run an eval"). +//! +//! A regression task asks whether Symposium delivered, so it runs one Symposium trial. A +//! capability task asks whether Symposium helps, so it runs one round: a baseline trial, +//! then a Symposium trial. More trials, alternating order, and `benchmark.json` come with +//! `--trials`. +//! +//! Everything a trial could fail on is checked before the first one starts: the +//! credential, `srt` and its stack, the client's version, and the `cargo-agents` under +//! evaluation. Each trial's result is then kept under +//! `results////`. An outcome, even a failed one, is not an error. + +mod batch_name; +mod claude; +mod credential; +pub(crate) mod host; +#[cfg(test)] +mod preflight_tests; +#[cfg(all(test, target_os = "linux"))] +mod real; +mod report; +mod symposium; + +use std::collections::BTreeSet; +use std::ffi::{OsStr, OsString}; +use std::fs; +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use anyhow::{Context, bail}; +use chrono::Utc; + +use crate::client::claude::version::Version; +use crate::client::claude::{Claude, version}; +use crate::evals::Evals; +use crate::grade::{self, GradeLimits}; +use crate::result::{Arm, Deadlines, IsolationMode, Provenance, SymposiumBuild}; +use crate::run::stage::{Audience, StagedTool, Tool}; +use crate::run::trial::{Driver, Sandbox}; +use crate::run::{ApiKey, Batch, Inputs, Machine, SRT_VERSION}; +use crate::task::{TaskKind, TaskName, ToolName}; + +/// The only host a Claude Code trial reaches. +const CLAUDE_API_HOST: &str = "api.anthropic.com"; + +/// The deadlines every trial runs with. +const DEADLINES: Deadlines = Deadlines { + setup: Duration::from_secs(120), + agent: Duration::from_secs(600), + grading: Duration::from_secs(600), +}; + +/// What the developer asked for. +#[derive(Debug)] +pub(crate) struct Request { + pub(crate) task: TaskName, + pub(crate) model: String, + pub(crate) symposium_bin: Option, + pub(crate) eval_root: Option, +} + +/// Where the runner reads tasks and keeps results. +#[derive(Debug)] +pub(crate) struct Checkout { + pub(crate) root: PathBuf, + pub(crate) evals: Evals, + pub(crate) results: PathBuf, +} + +impl Checkout { + /// The checkout this crate is built in: its `evals/`, and `results/` beside it. + pub(crate) fn in_repository() -> Self { + let root = Path::new(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("symposium-eval lives inside the repository") + .to_owned(); + Self { + evals: Evals::in_repository(), + results: root.join("results"), + root, + } + } +} + +/// A batch ready to run trials: everything checked and staged. +struct Ready { + batch: Batch, + inputs: Inputs, + machine: Machine, + srt: crate::run::srt::Srt, + api_key: ApiKey, + provenance: Provenance, + claude: version::Version, + build: SymposiumBuild, +} + +/// What preparation found on the developer's machine before creating a batch. +struct PreparedHost { + api_key: ApiKey, + path: OsString, + srt: crate::run::srt::Srt, + machine: Machine, +} + +/// The tools copied into a batch, together with the source revision of Symposium when +/// one can be named truthfully. +struct StagedTools { + claude: StagedTool, + cargo_agents: StagedTool, + symposium_revision: Option, +} + +/// One configuration planned for a round. Execution order may change without changing +/// the round that pairs a baseline with its Symposium trial. +#[derive(Debug)] +struct PlannedTrial { + round: usize, + arm: Arm, +} + +/// Runs `request` from `checkout`, with `var` reading the runner's environment. +/// +/// # Errors +/// +/// When the machine, the credential, or a tool is not fit to run trials, or a trial has +/// no result. A trial that failed is an outcome, not an error. +pub(crate) fn run( + request: &Request, + checkout: &Checkout, + var: impl Fn(&str) -> Option, +) -> anyhow::Result<()> { + if !cfg!(target_os = "linux") { + bail!("trials run under srt, which this runner supports on Linux only"); + } + let ready = prepare(request, checkout, &var)?; + let batch = ready + .batch + .root() + .file_name() + .context("a batch has a name")?; + let results = checkout + .results + .join(request.task.as_str()) + .join("claude") + .join(batch); + let failed = after_task_preflights( + [&ready.inputs], + |inputs| preflight_task(&ready.batch, inputs, &ready.machine, &ready.srt), + || { + probe_publication(&results)?; + Ok(run_trials(&ready, &request.model, &results)) + }, + )?; + println!("results in {}", results.display()); + if failed > 0 { + bail!("{failed} trial(s) have no result; see above"); + } + Ok(()) +} + +/// Checks the machine and credential, then creates and stages the batch. +fn prepare( + request: &Request, + checkout: &Checkout, + var: &impl Fn(&str) -> Option, +) -> anyhow::Result { + let host = inspect_host(checkout, var)?; + let mut batch = create_batch(request)?; + let scratch = scratch(&batch, "preflight")?; + host.srt.preflight(&scratch)?; + + let staged = stage_tools(&mut batch, request, checkout, &host.path)?; + let build = symposium::recorded(&staged.cargo_agents, staged.symposium_revision, &scratch)?; + let claude = version::preflight(&batch, &staged.claude, &scratch)?; + let inputs = batch.snapshot(&checkout.evals, &request.task)?; + + Ok(Ready { + batch, + inputs, + machine: host.machine, + srt: host.srt, + api_key: host.api_key, + provenance: provenance(claude)?, + claude, + build, + }) +} + +/// Preflights every distinct task before `launch` can spend model tokens. +fn after_task_preflights<'a, T>( + inputs: impl IntoIterator, + mut preflight: impl FnMut(&Inputs) -> anyhow::Result<()>, + launch: impl FnOnce() -> anyhow::Result, +) -> anyhow::Result { + let mut seen = BTreeSet::new(); + for inputs in inputs { + let task = inputs.task().name(); + if !seen.insert(task.as_str().to_owned()) { + continue; + } + preflight(inputs)?; + } + launch() +} + +/// Proves that one task's trusted grading inputs and required toolchain capabilities +/// can be prepared under the production grading sandbox. +fn preflight_task( + batch: &Batch, + inputs: &Inputs, + machine: &Machine, + sandbox: &impl Sandbox, +) -> anyhow::Result<()> { + let task = inputs.task().name(); + let trial = batch + .allocate_preflight(inputs) + .with_context(|| format!("cannot allocate grading preflight for `{task}`"))? + .prepare() + .map_err(|failed| failed.into_error()) + .with_context(|| format!("cannot prepare grading preflight for `{task}`"))?; + let dirs = trial + .prepare_grading(machine) + .with_context(|| format!("cannot prepare grading sandbox for `{task}`"))?; + let backend = sandbox + .for_grading(&trial) + .with_context(|| format!("cannot start grading sandbox for `{task}`"))?; + let limits = GradeLimits { + commands: DEADLINES.grading, + ..GradeLimits::default() + }; + grade::preflight_task(inputs.evals(), inputs.task(), &backend, &limits, &dirs) + .with_context(|| format!("grading preflight failed for `{task}`")) +} + +/// Proves that this batch's retained-results directory supports the same staged write +/// and atomic rename used to publish a trial result. +fn probe_publication(results: &Path) -> anyhow::Result<()> { + let parent = results + .parent() + .context("a results directory has a parent")?; + fs::create_dir_all(parent).with_context(|| format!("cannot create `{}`", parent.display()))?; + fs::create_dir(results).with_context(|| format!("cannot create `{}`", results.display()))?; + let staging = results.join(".publication-probe-staging"); + let published = results.join(".publication-probe-published"); + fs::create_dir(&staging).with_context(|| format!("cannot create `{}`", staging.display()))?; + fs::write(staging.join("probe"), b"publication probe\n") + .with_context(|| format!("cannot write publication probe in `{}`", staging.display()))?; + fs::rename(&staging, &published) + .with_context(|| format!("cannot publish a staged result in `{}`", results.display()))?; + fs::remove_dir_all(&published) + .with_context(|| format!("cannot remove `{}`", published.display()))?; + Ok(()) +} + +/// Checks the host state needed by every later preparation phase. +fn inspect_host( + checkout: &Checkout, + var: &impl Fn(&str) -> Option, +) -> anyhow::Result { + let api_key = credential::claude(var)?; + let path = var("PATH").context("PATH is not set")?; + let srt = host::srt(&path)?; + let machine = Machine::new(Vec::new(), &checkout.root, &host::sysroot()?)?; + Ok(PreparedHost { + api_key, + path, + srt, + machine, + }) +} + +/// Creates the fresh batch in which preparation stages its inputs. +fn create_batch(request: &Request) -> anyhow::Result { + Ok(Batch::create( + &eval_root(request)?, + &batch_name::new_batch_name(Utc::now()), + )?) +} + +/// Discovers and stages the tools the trials will run. +fn stage_tools( + batch: &mut Batch, + request: &Request, + checkout: &Checkout, + path: &OsStr, +) -> anyhow::Result { + let under_evaluation = + symposium::under_evaluation(&checkout.root, request.symposium_bin.as_deref())?; + let claude_tools = claude::claude_tools(path)?; + if let Some(node) = &claude_tools.node { + stage(batch, "node", node, Audience::Common)?; + } + let claude = stage(batch, "claude", &claude_tools.claude, Audience::Common)?; + let cargo_agents = stage( + batch, + "cargo-agents", + &Tool::Executable(under_evaluation.binary), + Audience::Symposium, + )?; + Ok(StagedTools { + claude, + cargo_agents, + symposium_revision: under_evaluation.revision, + }) +} + +/// Metadata shared by every result in the prepared batch. +fn provenance(claude: Version) -> anyhow::Result { + Ok(Provenance { + client_version: Some(claude.to_string()), + os: host::os(), + toolchain: Some(host::toolchain()?), + isolation: IsolationMode::Enforced, + srt_version: Some(SRT_VERSION.to_owned()), + deadlines: DEADLINES, + }) +} + +/// Runs each trial the task's kind asks for, keeps its results under `results`, and +/// prints its outcome. Returns how many trials have no result. +fn run_trials(ready: &Ready, model: &str, results: &Path) -> usize { + let driver = Driver { + batch: &ready.batch, + inputs: &ready.inputs, + machine: &ready.machine, + sandbox: &ready.srt, + api_host: CLAUDE_API_HOST, + api_key: &ready.api_key, + provenance: &ready.provenance, + }; + let mut failed = 0; + for PlannedTrial { round, arm } in plan(ready.inputs.task().kind(), &ready.build) { + let configuration = arm.configuration(); + let runner = Claude::enforced(model.to_owned(), ready.claude); + let kept = driver + .run(round, arm, &runner) + .map_err(anyhow::Error::from) + .and_then(|done| { + let destination = + results.join(done.root.file_name().context("a trial has a name")?); + done.retain(&destination)?; + Ok(destination) + }); + match kept { + Ok(destination) => report::trial(&destination), + Err(error) => { + failed += 1; + report::no_result(round, configuration, &error); + } + } + } + failed +} + +/// The rounds and configurations a task of `kind` runs, in execution order. +fn plan(kind: TaskKind, build: &SymposiumBuild) -> Vec { + match kind { + TaskKind::Regression => vec![PlannedTrial { + round: 1, + arm: Arm::Symposium(build.clone()), + }], + TaskKind::Capability => vec![ + PlannedTrial { + round: 1, + arm: Arm::Baseline, + }, + PlannedTrial { + round: 1, + arm: Arm::Symposium(build.clone()), + }, + ], + } +} + +/// The evaluation root asked for, or one under the system's temporary directory. +fn eval_root(request: &Request) -> anyhow::Result { + let root = match &request.eval_root { + Some(root) => std::path::absolute(root) + .with_context(|| format!("cannot resolve `{}`", root.display()))?, + None => std::env::temp_dir().join("symposium-eval"), + }; + Ok(root) +} + +/// A directory of the batch's own for checks before any trial. +fn scratch(batch: &Batch, name: &str) -> anyhow::Result { + let dir = batch.root().join(name); + fs::create_dir(&dir).with_context(|| format!("cannot create `{}`", dir.display()))?; + Ok(dir) +} + +fn stage( + batch: &mut Batch, + name: &str, + tool: &Tool, + audience: Audience, +) -> anyhow::Result { + let name = ToolName::new(name).expect("a tool name the runner chose"); + batch + .stage(name.clone(), tool, audience) + .with_context(|| format!("cannot stage `{name}`")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn symposium_build() -> SymposiumBuild { + SymposiumBuild { + version: "cargo-agents 1.0.0".to_owned(), + sha256: crate::digest::file( + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("Cargo.toml") + .as_path(), + ) + .unwrap(), + revision: None, + } + } + + #[test] + fn a_regression_task_runs_with_symposium_only() { + let plan = plan(TaskKind::Regression, &symposium_build()); + + assert!( + matches!( + plan[..], + [PlannedTrial { + round: 1, + arm: Arm::Symposium(_) + }] + ), + "{plan:?}" + ); + } + + #[test] + fn a_capability_task_pairs_both_arms_in_round_one() { + let plan = plan(TaskKind::Capability, &symposium_build()); + + assert!( + matches!( + plan[..], + [ + PlannedTrial { + round: 1, + arm: Arm::Baseline + }, + PlannedTrial { + round: 1, + arm: Arm::Symposium(_) + } + ] + ), + "{plan:?}" + ); + } +} diff --git a/symposium-eval/src/command/preflight_tests.rs b/symposium-eval/src/command/preflight_tests.rs new file mode 100644 index 000000000..42ea63368 --- /dev/null +++ b/symposium-eval/src/command/preflight_tests.rs @@ -0,0 +1,235 @@ +//! Fast command-boundary regressions use a controlled backend: the missing-rustfmt +//! case is simulated by making Cargo reject the formatter subcommand. The ignored +//! Linux command test in `real` remains the production-shaped check that successful +//! preparation and a complete journey run under real `srt`. + +use std::cell::{Cell, RefCell}; +use std::fs; +use std::path::Path; +use std::time::Duration; + +use tempfile::TempDir; + +use super::{after_task_preflights, preflight_task, probe_publication}; +use crate::grade::ProcessError; +use crate::grade::process::{Backend, Finished, Invocation, Local}; +use crate::run::test_support::{Host, inputs}; +use crate::run::trial::Sandbox; +use crate::run::{Batch, Inputs, Machine, RunError, Trial}; +use crate::task::BatchName; + +#[derive(Clone, Copy)] +enum PreflightBackend { + Local, + MissingFmt, +} + +impl Backend for PreflightBackend { + fn run( + &self, + invocation: &Invocation, + deadline: Duration, + logs: &Path, + ) -> Result { + if matches!(self, Self::MissingFmt) + && invocation.args.iter().any(|argument| argument == "fmt") + { + let mut unavailable = invocation.clone(); + unavailable.args = vec!["symposium-missing-rustfmt".into()]; + Local.run(&unavailable, deadline, logs) + } else { + Local.run(invocation, deadline, logs) + } + } +} + +struct PreflightSandbox(PreflightBackend); + +impl Sandbox for PreflightSandbox { + type Backend<'a> = PreflightBackend; + + fn for_trial(&self, _trial: &Trial) -> Result, RunError> { + Ok(self.0) + } + + fn for_grading(&self, _trial: &Trial) -> Result, RunError> { + Ok(self.0) + } +} + +fn preflight_with_launch_count( + batch: &Batch, + inputs: &Inputs, + machine: &Machine, + sandbox: &impl Sandbox, +) -> (anyhow::Result<()>, usize) { + let launches = Cell::new(0); + let result = after_task_preflights( + [inputs], + |inputs| preflight_task(batch, inputs, machine, sandbox), + || { + launches.set(launches.get() + 1); + Ok(()) + }, + ); + (result, launches.get()) +} + +#[test] +fn publication_preflight_stages_and_renames_inside_results() { + let root = TempDir::new().unwrap(); + let results = root.path().join("task/client/batch"); + + probe_publication(&results).unwrap(); + + assert!(results.is_dir()); + assert_eq!(fs::read_dir(&results).unwrap().count(), 0); +} + +#[test] +fn publication_preflight_never_reuses_a_batch_results_directory() { + let root = TempDir::new().unwrap(); + let results = root.path().join("task/client/batch"); + probe_publication(&results).unwrap(); + + let error = probe_publication(&results).unwrap_err(); + + assert!(error.to_string().contains("cannot create"), "{error:#}"); +} + +#[test] +fn every_task_passes_preflight_before_a_client_can_launch() { + let root = TempDir::new().unwrap(); + let first_batch = Batch::create(root.path(), &BatchName::new("first").unwrap()).unwrap(); + let second_batch = Batch::create(root.path(), &BatchName::new("second").unwrap()).unwrap(); + let first = inputs(&first_batch, "formatting"); + let second = inputs(&second_batch, "store-durability"); + let checked = RefCell::new(Vec::new()); + let launches = Cell::new(0); + + let result = after_task_preflights( + [&first, &second], + |inputs| { + let name = inputs.task().name().as_str(); + checked.borrow_mut().push(name.to_owned()); + if name == "store-durability" { + anyhow::bail!("injected later-task failure"); + } + Ok(()) + }, + || { + launches.set(launches.get() + 1); + Ok(()) + }, + ); + + assert!(result.is_err()); + assert_eq!(&*checked.borrow(), &["formatting", "store-durability"]); + assert_eq!(launches.get(), 0); +} + +#[test] +fn missing_rustfmt_stops_before_a_client_launches() { + let host = Host::new(); + let batch = host.batch(); + let inputs = inputs(&batch, "formatting"); + + let (result, launches) = preflight_with_launch_count( + &batch, + &inputs, + &host.machine(), + &PreflightSandbox(PreflightBackend::MissingFmt), + ); + + let error = result.unwrap_err(); + assert!( + format!("{error:#}").contains("cargo fmt --version"), + "{error:#}" + ); + assert_eq!(launches, 0); +} + +#[test] +fn ambient_grading_inputs_stop_before_a_client_launches() { + for name in [ + ".cargo/config", + ".cargo/config.toml", + "Cargo.toml", + "rustfmt.toml", + ".rustfmt.toml", + ] { + assert_ambient_input_stops_before_launch(name); + } +} + +fn assert_ambient_input_stops_before_launch(name: &str) { + let host = Host::new(); + let config = host.join(name); + fs::create_dir_all(config.parent().unwrap()).unwrap(); + fs::write(&config, "").unwrap(); + let batch = host.batch(); + let inputs = inputs(&batch, "store-durability"); + + let (result, launches) = preflight_with_launch_count( + &batch, + &inputs, + &host.machine(), + &PreflightSandbox(PreflightBackend::Local), + ); + + let error = result.unwrap_err(); + assert!( + format!("{error:#}").contains("Cargo or rustfmt would read"), + "{error:#}" + ); + assert!(format!("{error:#}").contains(&config.display().to_string())); + assert_eq!(launches, 0); +} + +#[test] +fn invalid_starter_metadata_stops_before_a_client_launches() { + let host = Host::new(); + let batch = host.batch(); + let inputs = inputs(&batch, "store-durability"); + fs::write( + inputs + .evals() + .starter_dir(inputs.task().name()) + .join("Cargo.toml"), + "this is not TOML", + ) + .unwrap(); + + let (result, launches) = preflight_with_launch_count( + &batch, + &inputs, + &host.machine(), + &PreflightSandbox(PreflightBackend::Local), + ); + + let error = result.unwrap_err(); + assert!(format!("{error:#}").contains("cargo metadata"), "{error:#}"); + assert_eq!(launches, 0); +} + +#[test] +fn publication_failure_stops_before_a_client_launches() { + let root = TempDir::new().unwrap(); + let results = root.path().join("results"); + fs::write(&results, "not a directory").unwrap(); + let launches = Cell::new(0); + + let result = after_task_preflights( + std::iter::empty(), + |_| unreachable!("there are no tasks"), + || { + probe_publication(&results)?; + launches.set(launches.get() + 1); + Ok(()) + }, + ); + + let error = result.unwrap_err(); + assert!(format!("{error:#}").contains("cannot create"), "{error:#}"); + assert_eq!(launches.get(), 0); +} diff --git a/symposium-eval/src/command/real.rs b/symposium-eval/src/command/real.rs new file mode 100644 index 000000000..537077ce4 --- /dev/null +++ b/symposium-eval/src/command/real.rs @@ -0,0 +1,185 @@ +//! The command end to end under real `srt`, with the Node stand-in for Claude Code +//! installed as npm installs it, and the `cargo-agents` built from this checkout given +//! as `--symposium-bin`: run on Linux with `cargo test -p symposium-eval -- --ignored`. + +use std::ffi::OsString; +use std::fs; +use std::os::unix::fs::symlink; +use std::path::{Path, PathBuf}; + +use serde_json::Value; +use tempfile::TempDir; + +use super::{Checkout, Request, run}; +use crate::evals::Evals; +use crate::run::test_support::{built_cargo_agents, checkout, executable}; +use crate::task::TaskName; + +const STAND_IN: &str = include_str!("../run/trial/claude-stand-in.js"); + +/// A production-shaped command invocation with a deterministic Claude Code stand-in. +struct Scenario { + _temp: TempDir, + path: OsString, + request: Request, + checkout: Checkout, +} + +impl Scenario { + fn formatting_regression() -> Self { + Self::for_task("formatting") + } + + fn store_durability_capability() -> Self { + Self::for_task("store-durability") + } + + fn for_task(task: &str) -> Self { + let temp = TempDir::new().unwrap(); + let bin = install_claude_stand_in(temp.path()); + let path = std::env::join_paths( + std::iter::once(bin).chain(std::env::split_paths(&std::env::var_os("PATH").unwrap())), + ) + .unwrap(); + let request = Request { + task: TaskName::new(task).unwrap(), + model: "a-model".to_owned(), + symposium_bin: Some(built_cargo_agents()), + eval_root: Some(temp.path().join("eval")), + }; + let checkout = Checkout { + root: checkout(), + evals: Evals::in_repository(), + results: temp.path().join("results"), + }; + Self { + _temp: temp, + path, + request, + checkout, + } + } + + fn run(&self) { + run(&self.request, &self.checkout, |name| match name { + "PATH" => Some(self.path.clone()), + "ANTHROPIC_API_KEY" => Some(OsString::from("sk-test")), + _ => None, + }) + .unwrap(); + } + + fn kept_trials(&self) -> Vec { + let batches = self + .checkout + .results + .join(self.request.task.as_str()) + .join("claude"); + let batch = only_entry(&batches, "batch"); + let mut trials: Vec<_> = fs::read_dir(batch) + .unwrap() + .map(|entry| KeptTrial::read(entry.unwrap().path())) + .collect(); + trials.sort_by(|left, right| left.name().cmp(right.name())); + trials + } + + fn only_kept_trial(&self) -> KeptTrial { + let mut trials = self.kept_trials(); + if trials.len() != 1 { + panic!("expected one trial, found {}", trials.len()); + } + trials.pop().unwrap() + } +} + +struct KeptTrial { + root: PathBuf, + result: Value, +} + +impl KeptTrial { + fn read(root: PathBuf) -> Self { + let result = serde_json::from_slice(&fs::read(root.join("result.json")).unwrap()).unwrap(); + Self { root, result } + } + + fn name(&self) -> &str { + self.root.file_name().unwrap().to_str().unwrap() + } + + fn outcome(&self) -> &str { + self.result["outcome"]["result"].as_str().unwrap() + } + + fn symposium_revision(&self) -> &Value { + &self.result["symposium"]["revision"] + } + + fn toolchain(&self) -> &str { + self.result["toolchain"].as_str().unwrap() + } + + fn has_transcript(&self) -> bool { + self.root.join("transcript.jsonl").is_file() + } + + fn round(&self) -> u64 { + self.result["trial"].as_u64().unwrap() + } +} + +fn install_claude_stand_in(root: &Path) -> PathBuf { + let package = root.join("lib/node_modules/claude-stand-in"); + fs::create_dir_all(&package).unwrap(); + fs::write(package.join("package.json"), "{}").unwrap(); + executable(&package.join("cli.js"), STAND_IN); + let bin = root.join("bin"); + fs::create_dir(&bin).unwrap(); + symlink(package.join("cli.js"), bin.join("claude")).unwrap(); + bin +} + +fn only_entry(directory: &Path, kind: &str) -> PathBuf { + let entries: Vec<_> = fs::read_dir(directory) + .unwrap() + .map(|entry| entry.unwrap().path()) + .collect(); + let [entry] = &entries[..] else { + panic!( + "expected one {kind} in `{}`, found {entries:?}", + directory.display() + ); + }; + entry.clone() +} + +#[test] +#[ignore = "needs node, srt, and bubblewrap installed, and builds cargo-agents"] +fn a_regression_task_runs_one_symposium_trial_and_keeps_it() { + let scenario = Scenario::formatting_regression(); + + scenario.run(); + + let trial = scenario.only_kept_trial(); + assert_eq!(trial.name(), "trial-1-with-symposium"); + assert_eq!(trial.outcome(), "succeeded"); + assert_eq!(trial.symposium_revision(), &Value::Null); + assert!(trial.toolchain().starts_with("rustc ")); + assert!(trial.has_transcript()); +} + +#[test] +#[ignore = "needs node, srt, and bubblewrap installed, and builds cargo-agents"] +fn a_capability_task_keeps_both_arms_in_the_same_round() { + let scenario = Scenario::store_durability_capability(); + + scenario.run(); + + let trials = scenario.kept_trials(); + assert_eq!( + trials.iter().map(KeptTrial::name).collect::>(), + ["trial-1-with-symposium", "trial-1-without-symposium"] + ); + assert!(trials.iter().all(|trial| trial.round() == 1)); +} diff --git a/symposium-eval/src/command/report.rs b/symposium-eval/src/command/report.rs new file mode 100644 index 000000000..aba6b69a4 --- /dev/null +++ b/symposium-eval/src/command/report.rs @@ -0,0 +1,78 @@ +//! One line per trial, read back from the result it kept, so what is printed is what +//! was recorded. + +use std::fs; +use std::path::Path; + +use serde_json::Value; + +use crate::run::Configuration; +use crate::run::trial::RESULT_FILE; + +/// Prints the outcome the trial kept at `kept` recorded, such as +/// `trial-1-with-symposium: failed (expectation)`. +pub(crate) fn trial(kept: &Path) { + let name = kept.file_name().unwrap_or_default().to_string_lossy(); + println!("{name}: {}", outcome(&kept.join(RESULT_FILE))); +} + +/// Prints why one configuration in `round` has no result. +pub(crate) fn no_result(round: usize, configuration: Configuration, error: &anyhow::Error) { + println!("trial-{round}-{configuration}: no result: {error:#}"); +} + +fn outcome(result: &Path) -> String { + let read = fs::read(result) + .ok() + .and_then(|bytes| serde_json::from_slice::(&bytes).ok()); + let Some(result) = read else { + return "result unreadable".to_owned(); + }; + describe(&result) +} + +/// `result`'s outcome, its failure or the phase it stopped at when there is one. +fn describe(result: &Value) -> String { + let outcome = &result["outcome"]; + let name = outcome["result"].as_str().unwrap_or("unknown"); + let detail = outcome["failure"].as_str().or_else(|| { + (name == "setup_failure") + .then(|| result["reached"]["phase"].as_str()) + .flatten() + }); + match detail { + Some(detail) => format!("{name} ({detail})"), + None => name.to_owned(), + } +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::*; + + #[test] + fn a_failure_names_what_failed() { + let result = json!({ "outcome": { "result": "failed", "failure": "expectation" } }); + + assert_eq!(describe(&result), "failed (expectation)"); + } + + #[test] + fn a_setup_failure_names_where_it_stopped() { + let result = json!({ + "reached": { "phase": "prelaunch" }, + "outcome": { "result": "setup_failure" } + }); + + assert_eq!(describe(&result), "setup_failure (prelaunch)"); + } + + #[test] + fn a_success_is_one_word() { + let result = json!({ "outcome": { "result": "succeeded" } }); + + assert_eq!(describe(&result), "succeeded"); + } +} diff --git a/symposium-eval/src/command/symposium.rs b/symposium-eval/src/command/symposium.rs new file mode 100644 index 000000000..759ac1344 --- /dev/null +++ b/symposium-eval/src/command/symposium.rs @@ -0,0 +1,203 @@ +//! The `cargo-agents` under evaluation: built from the checkout, or the binary given with +//! `--symposium-bin`. A result names the staged copy, which is what trials run, by its +//! version and hash, and by the commit it was built from only when that commit describes +//! it: the checkout's `HEAD` with nothing changed or added. A given binary's commit is +//! unknown. + +use std::path::{Path, PathBuf}; +use std::process::Command; + +use anyhow::{Context, bail}; + +use super::host; +use crate::result::SymposiumBuild; +use crate::run::stage::StagedTool; + +/// The binary to stage, and the commit it was built from, when known. +#[derive(Debug)] +pub(crate) struct UnderEvaluation { + pub(crate) binary: PathBuf, + pub(crate) revision: Option, +} + +/// `given`, or a release build from `checkout`. +pub(crate) fn under_evaluation( + checkout: &Path, + given: Option<&Path>, +) -> anyhow::Result { + Ok(match given { + Some(given) => UnderEvaluation { + binary: host::resolved(given)?, + revision: None, + }, + None => UnderEvaluation { + binary: build(checkout)?, + revision: clean_revision(checkout), + }, + }) +} + +/// What a result records of `staged`. `scratch` is the home its `--version` runs with, +/// so it reads none of the developer's configuration. +pub(crate) fn recorded( + staged: &StagedTool, + revision: Option, + scratch: &Path, +) -> anyhow::Result { + Ok(SymposiumBuild { + version: version(staged.command(), scratch)?, + sha256: staged + .digest() + .context("cannot hash the staged cargo-agents")?, + revision, + }) +} + +/// Builds `cargo-agents` from `checkout`, and returns where Cargo put it. +fn build(checkout: &Path) -> anyhow::Result { + let output = Command::new(env!("CARGO")) + .args(["build", "--release", "--locked", "--quiet"]) + .args(["--package", "symposium", "--bin", "cargo-agents"]) + .arg("--message-format=json-render-diagnostics") + .current_dir(checkout) + .output() + .context("cannot run cargo")?; + if !output.status.success() { + bail!( + "cannot build cargo-agents:\n{}", + String::from_utf8_lossy(&output.stderr) + ); + } + String::from_utf8_lossy(&output.stdout) + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .find(|message| message["target"]["name"] == "cargo-agents") + .and_then(|message| message["executable"].as_str().map(PathBuf::from)) + .context("cargo did not report the cargo-agents executable") +} + +/// `HEAD`, when nothing in `checkout` is changed or untracked; otherwise the build is +/// not that commit's, so none is named. +fn clean_revision(checkout: &Path) -> Option { + let git = |args: &[&str]| { + let output = Command::new("git") + .args(args) + .current_dir(checkout) + .output() + .ok()?; + output + .status + .success() + .then(|| String::from_utf8(output.stdout).ok()) + .flatten() + }; + let status = git(&["status", "--porcelain"])?; + if !status.trim().is_empty() { + return None; + } + git(&["rev-parse", "HEAD"]).map(|head| head.trim().to_owned()) +} + +/// `binary --version`, run outside any sandbox with `scratch` as its home and nothing +/// else in its environment. +fn version(binary: &Path, scratch: &Path) -> anyhow::Result { + let output = Command::new(binary) + .arg("--version") + .env_clear() + .env("HOME", scratch) + .env("PATH", "/usr/bin:/bin") + .current_dir(scratch) + .output() + .with_context(|| format!("cannot run `{} --version`", binary.display()))?; + if !output.status.success() { + bail!("`{} --version` failed", binary.display()); + } + Ok(String::from_utf8_lossy(&output.stdout).trim().to_owned()) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use tempfile::TempDir; + + use super::*; + + struct Repository { + root: TempDir, + } + + impl Repository { + fn committed() -> Self { + let repository = Self { + root: TempDir::new().unwrap(), + }; + repository.git(&["init", "--quiet"]); + repository.write("lib.rs", ""); + repository.git(&["add", "."]); + repository.git(&["commit", "--quiet", "-m", "start"]); + repository + } + + fn path(&self) -> &Path { + self.root.path() + } + + fn write(&self, path: &str, contents: &str) { + fs::write(self.path().join(path), contents).unwrap(); + } + + fn head(&self) -> String { + self.git_output(&["rev-parse", "HEAD"]) + } + + fn git(&self, args: &[&str]) { + let status = self.git_command(args).status().unwrap(); + assert!(status.success(), "git {args:?}"); + } + + fn git_output(&self, args: &[&str]) -> String { + let output = self.git_command(args).output().unwrap(); + assert!(output.status.success(), "git {args:?}"); + String::from_utf8(output.stdout).unwrap().trim().to_owned() + } + + fn git_command(&self, args: &[&str]) -> Command { + let mut command = Command::new("git"); + command + .args(["-c", "user.name=test", "-c", "user.email=test@example.com"]) + .args(args) + .current_dir(self.path()); + command + } + } + + #[test] + fn a_clean_checkout_names_its_commit() { + let repository = Repository::committed(); + + let revision = clean_revision(repository.path()); + + assert_eq!(revision.as_deref(), Some(repository.head().as_str())); + } + + #[test] + fn a_changed_checkout_names_no_commit() { + let repository = Repository::committed(); + repository.write("lib.rs", "changed"); + + let revision = clean_revision(repository.path()); + + assert_eq!(revision, None); + } + + #[test] + fn an_untracked_file_names_no_commit() { + let repository = Repository::committed(); + repository.write("new.rs", ""); + + let revision = clean_revision(repository.path()); + + assert_eq!(revision, None); + } +} diff --git a/symposium-eval/src/grade/mod.rs b/symposium-eval/src/grade/mod.rs index 32753434f..178df9d81 100644 --- a/symposium-eval/src/grade/mod.rs +++ b/symposium-eval/src/grade/mod.rs @@ -308,6 +308,39 @@ pub(crate) fn grade( } } +/// Checks the trusted task inputs and the grading toolchain without importing or +/// running a submission. +/// +/// This is the same preparation [`grade`] performs for every trial: it copies the +/// starter and fixtures, resolves the starter with Cargo, places and verifies every +/// hidden check, and verifies `cargo fmt` when the task requires formatting. +/// +/// # Errors +/// +/// When trusted inputs cannot be copied, a required command cannot be run or does not +/// succeed, or the starter and hidden checks do not have the expected Cargo targets. +pub(crate) fn preflight_task( + evals: &Evals, + task: &Task, + backend: &impl Backend, + limits: &GradeLimits, + dirs: &GradingDirs, +) -> Result<(), GradeError> { + fs::create_dir_all(&dirs.logs) + .map_err(|source| ImportError::Write { + path: dirs.logs.clone(), + source, + }) + .map_err(GradeError::from)?; + let mut commands = Commands { + backend, + budget: Budget::new(limits.commands), + logs: &dirs.logs, + }; + prepare(evals, task, &PROJECT_SCAFFOLD, dirs, &mut commands, limits)?; + Ok(()) +} + fn try_grade( evals: &Evals, task: &Task, @@ -384,7 +417,7 @@ fn prepare( let placed = inspect_starter(&cargo, "metadata-checks", ©, commands, limits.metadata)?; verify_check_targets(task.checks(), &placed.tests)?; if task.fmt_check() { - preflight( + preflight_command( commands, &cargo.fmt_version(), "cargo fmt --version", @@ -558,7 +591,7 @@ fn inspect_starter( commands: &mut Commands<'_, impl Backend>, cap: Duration, ) -> Result { - let finished = preflight(commands, &cargo.metadata(log_name), "cargo metadata", cap)?; + let finished = preflight_command(commands, &cargo.metadata(log_name), "cargo metadata", cap)?; cargo::starter_targets(finished.stdout.log(), ©.project()) } @@ -602,7 +635,7 @@ fn verify_check_targets( /// Runs a command that must succeed on the trusted starter. Its failure is the task's or /// the machine's, never the submission's. -fn preflight( +fn preflight_command( commands: &mut Commands<'_, impl Backend>, invocation: &Invocation, step: &str, @@ -1114,6 +1147,29 @@ fn locks_grading_tmp() { ) } + #[test] + fn task_preflight_checks_the_required_formatter() { + let fixture = TaskFixture::new("fmt_check = true\n"); + let root = TempDir::new().unwrap(); + + let error = preflight_task( + fixture.evals(), + fixture.task(), + &FailsOn { argument: "fmt" }, + &GradeLimits::default(), + &GradingDirs::under(root.path()), + ) + .unwrap_err(); + + assert!( + matches!( + error, + GradeError::Process { ref step, .. } if step == "cargo fmt --version" + ), + "{error:?}" + ); + } + #[test] fn an_infrastructure_failure_keeps_steps_that_already_completed() { let grading = grade_with_failure("--test"); @@ -1175,7 +1231,7 @@ fn locks_grading_tmp() { }; let invocation = Cargo::new(logs.path().to_owned(), logs.path()).metadata("metadata"); - let error = preflight( + let error = preflight_command( &mut commands, &invocation, "cargo metadata", diff --git a/symposium-eval/src/lib.rs b/symposium-eval/src/lib.rs index 00ede6ad2..75a3bf60c 100644 --- a/symposium-eval/src/lib.rs +++ b/symposium-eval/src/lib.rs @@ -6,35 +6,29 @@ //! is `md/rfds/skill-evaluation/`; `md/design/agent-eval.md` maps it onto this //! crate's modules. -#[expect(dead_code, reason = "the runner starts sessions later in RFD step 4")] mod client; -#[expect(dead_code, reason = "the runner records these later in RFD step 4")] +mod command; mod digest; mod evals; -#[expect(dead_code, reason = "the runner decides these later in RFD step 4")] mod expect; mod grade; mod jsonl; -#[cfg_attr( - not(test), - expect(dead_code, reason = "the runner decides these later in RFD step 4") -)] mod outcome; -#[expect(dead_code, reason = "the runner writes these later in RFD step 4")] mod result; -#[expect(dead_code, reason = "the runner launches trials later in RFD step 2")] mod run; mod skill_md; mod task; mod tree; use std::ffi::OsString; +use std::path::PathBuf; use anyhow::{Context, bail}; -use clap::Parser; +use clap::{Parser, ValueEnum}; +use crate::command::{Checkout, Request}; use crate::evals::{Evals, TaskEntry}; -use crate::task::{Task, TaskError, TaskErrors}; +use crate::task::{Task, TaskError, TaskErrors, TaskName}; #[derive(Debug, Parser)] #[command( @@ -44,8 +38,33 @@ use crate::task::{Task, TaskError, TaskErrors}; )] struct Cli { /// Validate every task under evals/ without running an agent - #[arg(long)] + #[arg(long, conflicts_with = "task")] check_tasks: bool, + + /// The task to run, a directory under evals/tasks/ + #[arg(required_unless_present = "check_tasks", requires_all = ["client", "model"])] + task: Option, + + /// The client to run the task on + #[arg(long, value_enum)] + client: Option, + + /// The model to ask the client for, as the client names it + #[arg(long)] + model: Option, + + /// Evaluate this cargo-agents instead of building one from the checkout + #[arg(long, value_name = "PATH")] + symposium_bin: Option, + + /// Where batches run, outside the checkout [default: the system's temporary directory] + #[arg(long, value_name = "DIR")] + eval_root: Option, +} + +#[derive(Debug, Clone, Copy, ValueEnum)] +enum ClientArg { + Claude, } /// `args` is everything after `agent-eval`. Clap handles `--help` and invalid @@ -65,7 +84,24 @@ where if cli.check_tasks { return check_tasks(&Evals::in_repository()); } - bail!("running a task is not implemented yet; use --check-tasks") + let request = request(cli)?; + command::run(&request, &Checkout::in_repository(), |name| { + std::env::var_os(name) + }) +} + +/// What `cli` asks to run. Clap has already required a task, client, and model. +fn request(cli: Cli) -> anyhow::Result { + let (Some(task), Some(ClientArg::Claude), Some(model)) = (cli.task, cli.client, cli.model) + else { + bail!("a task needs --client and --model"); + }; + Ok(Request { + task: TaskName::new(task)?, + model, + symposium_bin: cli.symposium_bin, + eval_root: cli.eval_root, + }) } fn check_tasks(evals: &Evals) -> anyhow::Result<()> { @@ -131,6 +167,49 @@ mod tests { Cli::command().debug_assert(); } + fn parses(args: &[&str]) -> bool { + Cli::try_parse_from(std::iter::once("agent-eval").chain(args.iter().copied())).is_ok() + } + + #[test] + fn a_task_needs_a_client_and_a_model() { + assert!(!parses(&[])); + assert!(!parses(&["formatting"])); + assert!(!parses(&["formatting", "--client", "claude"])); + assert!(!parses(&["formatting", "--model", "haiku"])); + assert!(parses(&[ + "formatting", + "--client", + "claude", + "--model", + "haiku" + ])); + } + + #[test] + fn only_a_client_with_an_adapter_is_accepted() { + assert!(!parses(&[ + "formatting", + "--client", + "codex", + "--model", + "m" + ])); + } + + #[test] + fn checking_tasks_runs_none() { + assert!(parses(&["--check-tasks"])); + assert!(!parses(&[ + "--check-tasks", + "formatting", + "--client", + "claude", + "--model", + "m" + ])); + } + #[test] fn checking_no_tasks_is_an_error() { let root = TempDir::new().unwrap(); diff --git a/symposium-eval/src/result/mod.rs b/symposium-eval/src/result/mod.rs index 2a7ac78c9..1498dd4d5 100644 --- a/symposium-eval/src/result/mod.rs +++ b/symposium-eval/src/result/mod.rs @@ -61,6 +61,7 @@ pub(crate) struct Planned<'a> { pub(crate) client: Client, pub(crate) model: &'a str, pub(crate) arm: Arm, + /// The round this arm belongs to, shared by paired baseline and Symposium trials. pub(crate) trial: usize, } @@ -106,6 +107,7 @@ pub(crate) struct SymposiumBuild { #[serde(rename_all = "snake_case")] pub(crate) enum IsolationMode { Enforced, + #[expect(dead_code, reason = "--exploratory needs its own design")] Exploratory, } diff --git a/symposium-eval/src/run/allocate.rs b/symposium-eval/src/run/allocate.rs index a34602049..67729770c 100644 --- a/symposium-eval/src/run/allocate.rs +++ b/symposium-eval/src/run/allocate.rs @@ -22,6 +22,7 @@ pub(crate) struct NewTrial<'a> { /// A trial whose preparation failed, with the root it keeps for its result. #[derive(Debug)] pub(crate) struct NotPrepared { + #[cfg_attr(not(test), expect(dead_code, reason = "only tests read it"))] root: PathBuf, error: RunError, } @@ -53,11 +54,40 @@ impl Batch { Ok(NewTrial { trial, inputs }) } + /// Allocates the named workspace used to prove that `inputs` can be prepared for + /// grading before any client is launched. + /// + /// Preflights live under `preflight/` rather than the numbered trial + /// namespace. A failed preflight is retained for diagnosis and cannot be reused. + /// + /// # Errors + /// + /// When `inputs` belong to another batch, the named workspace already exists, or + /// its directory cannot be created. + pub(crate) fn allocate_preflight<'a>( + &self, + inputs: &'a Inputs, + ) -> Result, RunError> { + if inputs.batch() != self.root { + return Err(RunError::ForeignInputs(inputs.batch().to_owned())); + } + let parent = self.root.join("preflight"); + create_dirs([parent.clone()])?; + let trial = Trial { + batch: self.clone(), + root: parent.join(inputs.task().name().as_str()), + configuration: Configuration::Baseline, + }; + create_fresh(&trial.root)?; + Ok(NewTrial { trial, inputs }) + } + /// Allocates trial `number` and prepares it from `inputs`. /// /// # Errors /// /// As [`Self::allocate_trial`] and [`NewTrial::prepare`]. + #[cfg(test)] pub(crate) fn create_trial( &self, number: usize, @@ -128,10 +158,12 @@ impl NewTrial<'_> { impl NotPrepared { /// The root the trial was allocated, where its result goes. + #[cfg(test)] pub(crate) fn root(&self) -> &Path { &self.root } + #[cfg(test)] pub(crate) fn error(&self) -> &RunError { &self.error } @@ -229,4 +261,43 @@ mod tests { assert!(trial.project().join("Cargo.lock").is_file()); assert!(trial.project().join(".claude").is_dir()); } + + #[test] + fn a_task_preflight_has_its_own_named_workspace() { + let root = TempDir::new().unwrap(); + let batch = batch(&root); + let inputs = store_durability(&batch); + + let new = batch.allocate_preflight(&inputs).unwrap(); + + assert_eq!( + new.root(), + batch + .root() + .join("preflight") + .join(inputs.task().name().as_str()) + ); + assert_eq!(fs::read_dir(new.root()).unwrap().count(), 0); + } + + #[test] + fn a_failed_task_preflight_is_kept_and_never_reused() { + let root = TempDir::new().unwrap(); + let batch = batch(&root); + let inputs = store_durability(&batch); + fs::remove_dir_all(inputs.evals().starter_dir(inputs.task().name())).unwrap(); + let failed = batch + .allocate_preflight(&inputs) + .unwrap() + .prepare() + .unwrap_err(); + + let again = batch.allocate_preflight(&inputs); + + assert!(failed.root().is_dir()); + assert!( + matches!(&again, Err(RunError::Create { source, .. }) if source.kind() == io::ErrorKind::AlreadyExists), + "{again:?}" + ); + } } diff --git a/symposium-eval/src/run/client.rs b/symposium-eval/src/run/client.rs index c49571b0f..eababf906 100644 --- a/symposium-eval/src/run/client.rs +++ b/symposium-eval/src/run/client.rs @@ -7,6 +7,10 @@ #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum Client { ClaudeCode, + #[cfg_attr( + not(test), + expect(dead_code, reason = "the Codex adapter comes in RFD step 7") + )] Codex, } diff --git a/symposium-eval/src/run/error.rs b/symposium-eval/src/run/error.rs index 6f608bb3f..6e7161d6c 100644 --- a/symposium-eval/src/run/error.rs +++ b/symposium-eval/src/run/error.rs @@ -43,10 +43,22 @@ pub(crate) enum RunError { Digest(#[source] DigestError), #[error("cannot write the `srt` policy")] Policy(#[source] serde_json::Error), + #[cfg_attr( + not(test), + expect(dead_code, reason = "--exploratory needs its own design") + )] #[error("cannot write the client's sandbox settings")] ClientSettings(#[source] serde_json::Error), + #[cfg_attr( + not(test), + expect(dead_code, reason = "--exploratory needs its own design") + )] #[error("`{}` would be read as a pattern by the client's sandbox", .0.display())] PatternInPath(PathBuf), + #[cfg_attr( + not(test), + expect(dead_code, reason = "--exploratory needs its own design") + )] #[error("`{}` is not valid Unicode, which the client's config needs", .0.display())] NotUnicode(PathBuf), #[error("`{}` is a baseline trial, which has no Symposium to set up", .0.display())] diff --git a/symposium-eval/src/run/mod.rs b/symposium-eval/src/run/mod.rs index acd26fe68..b3c3a1cb6 100644 --- a/symposium-eval/src/run/mod.rs +++ b/symposium-eval/src/run/mod.rs @@ -8,6 +8,7 @@ //! // //! tools/common/ staged clients, readable by every trial //! tools/symposium/ staged cargo-agents, readable by Symposium trials only +//! preflight// retained model-free grading preparation //! trial---symposium/ //! ``` //! @@ -26,6 +27,10 @@ mod allocate; pub(crate) mod client; mod env; mod error; +#[cfg_attr( + not(test), + expect(dead_code, reason = "--exploratory needs its own design") +)] mod exploratory; mod grading; mod inputs; @@ -36,7 +41,7 @@ mod runtime_tmp; pub(crate) mod setup; #[cfg(all(test, target_os = "linux"))] mod smoke; -mod srt; +pub(crate) mod srt; pub(crate) mod stage; #[cfg(test)] pub(crate) mod test_support; @@ -131,6 +136,11 @@ impl Batch { Ok(batch) } + /// The batch's own directory, named for the batch. + pub(crate) fn root(&self) -> &Path { + &self.root + } + /// The canonical evaluation root, which holds other batches and their trials. pub(crate) fn eval_root(&self) -> &Path { &self.eval_root diff --git a/symposium-eval/src/run/policy.rs b/symposium-eval/src/run/policy.rs index 9dabe0005..6e744d250 100644 --- a/symposium-eval/src/run/policy.rs +++ b/symposium-eval/src/run/policy.rs @@ -399,6 +399,7 @@ impl Policy { /// Whether `srt` lets the trial write `path`. `srt`'s own rules are not modelled: it /// always allows writing its shared temporary directory and never `.git/hooks` or /// shell startup files. + #[cfg(test)] pub(crate) fn can_write(&self, path: &Path) -> bool { most_specific(&self.filesystem.allow_write, path).is_some() } diff --git a/symposium-eval/src/run/stage.rs b/symposium-eval/src/run/stage.rs index 3dee120ab..3222557c8 100644 --- a/symposium-eval/src/run/stage.rs +++ b/symposium-eval/src/run/stage.rs @@ -29,7 +29,7 @@ pub(crate) enum Audience { } /// How a tool is installed on the developer's machine. -#[derive(Debug, Clone)] +#[derive(Debug, Clone, PartialEq, Eq)] pub(crate) enum Tool { /// One executable file, such as a native client or `node`. Executable(PathBuf), @@ -94,7 +94,7 @@ impl Batch { name: ToolName, tool: &Tool, audience: Audience, - ) -> Result<&StagedTool, RunError> { + ) -> Result { if self.staged.iter().any(|staged| staged.name == name) { return Err(RunError::StagedTwice(name.to_string())); } @@ -127,13 +127,14 @@ impl Batch { (destination.join(entry), Copy::Package(destination)) } }; - self.staged.push(StagedTool { + let staged = StagedTool { name, audience, command, copy, - }); - Ok(self.staged.last().expect("just pushed")) + }; + self.staged.push(staged.clone()); + Ok(staged) } /// Links every tool `trial` may run into its `bin/`. diff --git a/symposium-eval/src/run/test_support.rs b/symposium-eval/src/run/test_support.rs index 11c132268..d4fe2e5aa 100644 --- a/symposium-eval/src/run/test_support.rs +++ b/symposium-eval/src/run/test_support.rs @@ -177,7 +177,7 @@ impl RealSrt { audience: super::stage::Audience, ) -> super::stage::StagedTool { let name = crate::task::ToolName::new(name).unwrap(); - self.batch.stage(name, tool, audience).unwrap().clone() + self.batch.stage(name, tool, audience).unwrap() } pub(crate) fn srt(&self) -> &super::srt::Srt { @@ -364,10 +364,8 @@ fn installed_srt() -> super::srt::Srt { /// The first file called `name` in a directory on `PATH`. #[cfg(target_os = "linux")] pub(crate) fn on_path(name: &str) -> PathBuf { - std::env::split_paths(&std::env::var_os("PATH").unwrap()) - .map(|dir| dir.join(name)) - .find(|path| path.is_file()) - .unwrap_or_else(|| panic!("`{name}` is not on PATH")) + let path = std::env::var_os("PATH").unwrap(); + crate::command::host::on_path(&path, name).unwrap_or_else(|| panic!("`{name}` is not on PATH")) } /// The active toolchain's sysroot, as `rustc` reports it. diff --git a/symposium-eval/src/run/trial/mod.rs b/symposium-eval/src/run/trial/mod.rs index 0b44bf138..19cfce081 100644 --- a/symposium-eval/src/run/trial/mod.rs +++ b/symposium-eval/src/run/trial/mod.rs @@ -71,6 +71,7 @@ pub(crate) struct SessionOutput { } impl Done { + #[cfg(test)] pub(crate) fn result(&self) -> PathBuf { self.root.join(RESULT_FILE) } @@ -120,7 +121,7 @@ struct Stopped { } impl Driver<'_, S> { - /// Runs trial `number` of `arm` with `runner`'s client, and writes its result. + /// Runs `arm` in `round` with `runner`'s client, and writes its result. /// /// # Errors /// @@ -128,14 +129,14 @@ impl Driver<'_, S> { /// result cannot be written. Anything else that goes wrong is in the result. pub(crate) fn run( &self, - number: usize, + round: usize, arm: Arm, runner: &impl ClientRunner, ) -> Result { let configuration = arm.configuration(); let new = self .batch - .allocate_trial(number, configuration, self.inputs) + .allocate_trial(round, configuration, self.inputs) .map_err(TrialError::Allocate)?; let root = new.root().to_owned(); let path = root.join(RESULT_FILE); @@ -145,7 +146,7 @@ impl Driver<'_, S> { client: runner.client(), model: runner.model(), arm, - trial: number, + trial: round, }; let ran = self.ran(new, runner); let result = TrialResult::new(planned, self.provenance.clone(), ran.reached()) diff --git a/symposium-eval/src/task/mod.rs b/symposium-eval/src/task/mod.rs index f0d8f337c..427f739f9 100644 --- a/symposium-eval/src/task/mod.rs +++ b/symposium-eval/src/task/mod.rs @@ -166,10 +166,6 @@ impl Task { &self.name } - #[cfg_attr( - not(test), - expect(dead_code, reason = "the runner records this later in RFD step 4") - )] pub(crate) fn kind(&self) -> TaskKind { self.kind } @@ -205,10 +201,6 @@ impl Task { } /// The directory in `evals/` that provides `skill`, when one does. - #[cfg_attr( - not(test), - expect(dead_code, reason = "the runner decides these later in RFD step 4") - )] pub(crate) fn skill_source(&self, skill: &SkillName) -> Option<&Path> { self.skill_sources .iter() @@ -217,10 +209,6 @@ impl Task { } /// The plugin hooks that must leave a successful record, each with its event. - #[cfg_attr( - not(test), - expect(dead_code, reason = "the runner decides these later in RFD step 4") - )] pub(crate) fn expected_hooks(&self) -> &[ExpectedHook] { &self.expected_hooks }