diff --git a/md/design/agent-eval.md b/md/design/agent-eval.md index 811a64f91..8fe9b5814 100644 --- a/md/design/agent-eval.md +++ b/md/design/agent-eval.md @@ -54,7 +54,7 @@ Grading never runs a submission in place. `grade` (in `grade/mod.rs`) builds a * - The submission is listed recursively without following links (`tree::walk`). Files and directories are both imported, so an empty directory survives grading. A link or special file at, inside, or above an editable root (a pattern's literal prefix, such as `src`) invalidates the submission; anywhere else it is only reported. - Protected files are never imported, even when `editable` matches them. Their names compare without regard to ASCII case on every platform. - Every other change is reported as an outside edit (added, modified, deleted, or type changed), sorted by path; a directory with contents is reported through them, and an empty one by itself; files are compared in fixed-size chunks, so a large one costs time, not memory. The top-level `target/` is ignored, since the agent runs Cargo in its project. -- Only the agent's changes are its edits. `grade` takes a `Submitted`, built only as `Submitted::trial` or `Submitted::bare` (a task's solution), so the runner's additions come from one place: the empty directories every trial's starter gets (`grade::PROJECT_SCAFFOLD`, today `.claude/`), which the trusted copy gets too, so they are the starting state, and the skill copies shown to be wholly Symposium's, which are neither imported, even where `editable` matches them, nor reported, and neither is a directory that holds nothing else. `expect::ownership` decides that from the copy `expect::installed` found matching, but more strictly, since a copy that matches as sync reads it, links followed, can still hold an entry the agent made: the copy must be a real directory whose literal tree is the source's, real files and directories only and no others, with an empty `.symposium` and a `.gitignore` holding `*`, which sync writes over any the source has. Any other copy, a skill the task does not name, and anything the agent put in the scaffolding remain the agent's, imported or reported according to `editable`. +- Only the agent's changes are its edits. `grade` takes a `Submitted`, built only as `Submitted::trial` or `Submitted::bare` (a task's solution), so the runner's additions come from one place: the empty directories every trial's starter gets (`grade::PROJECT_SCAFFOLD`, today `.claude/`), which the trusted copy gets too, so they are the starting state, and the skill copies shown to be wholly Symposium's, which are neither imported, even where `editable` matches them, nor reported, and neither is a directory that holds nothing else. `expect::ownership` discovers candidates for every skill provided by the task's plugins, independently of installation and usage assertions. It checks each candidate against every trusted source for that name: the copy must be a real directory whose literal tree is the source's, real files and directories only and no others, with an empty `.symposium` and a `.gitignore` holding `*`, which sync writes over any the source has. Any other copy, a skill none of the task's plugins provides, and anything the agent put in the scaffolding remain the agent's, imported or reported according to `editable`. - The starter and fixtures are trusted, but a link or special file in them is still a task error rather than something to copy. Commands run through a `grade::process::Backend`: grading decides which command runs and what its result means, and the backend decides how the process starts and stops, so the `srt` backend can replace `Local` without changing grading policy. `Local` contains the whole process tree: on Unix a process group, killed at the deadline and again once the command exits; on Windows a kill-on-close job object the command is assigned to as it starts, closed on every path. A guard owns each command from the moment it starts, so every return path stops and reaps it, and no wait is unbounded. Full output streams to log files while a bounded tail stays in memory, and a failure to start, contain, stop, or collect a command is an infrastructure error. `GradeLimits` caps each command and all of grading's commands together: a command gets the smaller of its cap and the budget left, and only time spent running commands is charged. File work (copying, listing, comparing) is not counted; it gets entry-count and size quotas when grading accepts untrusted submissions. @@ -78,7 +78,7 @@ Both have a `solution/`, so `--check-tasks` shows each hidden check passes again ## Trials -A `Batch` is one run of the runner: `//`, holding the staged tools and its trials. A `Trial` is one agent session, created only by `Batch::create_trial` and carrying its batch, with a fresh copy of the starter, the fixtures, and, for a Symposium trial, the task's plugins as its registry. +A `Batch` is one run of the runner: `//`, holding the staged tools and its trials. A `Trial` is one agent session, carrying its batch, with a fresh copy of the starter, the fixtures, and, for a Symposium trial, the task's plugins as its registry. It is made in two steps (`run::allocate`): `Batch::allocate_trial` creates only its empty root and returns a `NewTrial`, which knows where its result goes, and `NewTrial::prepare` fills it and returns the `Trial`. Only a `Trial` can be set up, launched or graded, so a trial whose preparation failed never runs; it comes back as `NotPrepared`, keeping its root, so the runner can still record why it stopped. `Batch::create_trial` is the two together. - **Fresh.** The batch and trial directories are created with `create_dir`, so one that already exists is an error. A restarted run can never inherit an earlier attempt's home, project, or evidence. A trial whose setup failed keeps its directory and its number is not retried; `srt.json` is written last, so a trial without one never ran. On Unix the batch directory is `0700`, as Nix creates its build directories. - **Paths.** A batch name is a `BatchName`, one portable path component like a task name, so it cannot leave the evaluation root. The evaluation root must be absolute and is kept both as given and resolved. @@ -87,13 +87,14 @@ A `Batch` is one run of the runner: `//`, holding the staged t - **Temporary files.** `srt` lets every sandbox write `/tmp/claude` whatever the policy says, and a `denyRead` of it does not hide what is already there ([sandbox-runtime#654](https://github.com/anthropics/sandbox-runtime/issues/654), [#611](https://github.com/anthropics/sandbox-runtime/issues/611)), so the policy alone cannot keep one trial out of what another left there. The runner does not rely on environment-variable overrides to isolate it; the launcher closes it instead (below). `CLAUDE_CODE_TMPDIR` as Claude Code's own setting belongs to the Claude adapter. - **Staging.** `Batch::stage(name, tool, audience)` copies a tool into the batch once, before any trial is created: into `tools/common/` for every trial, or `tools/symposium/` for Symposium trials only. A `Tool` is one `Executable` (a native client, or `node`; a link such as `~/.local/bin/claude` is resolved and its target copied) or a `NodePackage`, a package directory run through an `entry`, a regular file inside it whose `env node` line finds the staged `node`; a missing or unusable entry fails staging rather than the first trial that runs it. A package is copied exactly (`grade::copy_exact`), `target/` included, since the grading copy's rule that `target/` is build output belongs to submissions, not installed packages; links and special files are still refused. Tools are copied, not linked, since `srt` matches read rules by the name a file is opened under, so a link into the developer's home would not run. Each trial's `bin/` then links every tool its configuration may run (only on Unix: elsewhere creating a trial that needs a link fails, rather than leaving `bin/` empty), so a baseline trial's `bin/` has no `cargo-agents`; the policy keeps it from reading `tools/symposium/` too, so either layer alone stops it. Which client is installed, and how, is the adapters' to say. `StagedTool::digest` hashes the staged copy, the file or the whole package, since that is what trials run; the result records only `cargo-agents`'s, which the RFD asks for. - **Digests.** `digest` gives SHA-256 digests a result can name its inputs by. A file's is of its bytes alone, as `sha256sum` prints it. A tree's covers each entry by its `/`-joined relative path, sorted by bytes: its kind (directory or file), and a file's bytes, each length written before what it measures so no two trees encode alike. Timestamps, permissions and absolute paths are left out, so a copy elsewhere has the same digest; a link or special file is refused rather than followed or skipped. Bytes are as checked out, so a checkout that rewrites line endings hashes differently. Each encoding begins with its name and version (`symposium-eval tree v1`), and a test pins a value computed by an independent implementation, so changing the encoding means naming a new version. The RFD's `content_hash` is the tree digest of the batch's `inputs/`, whose paths (`tasks//...`, `fixtures//...`, `plugins//...`) already say what each file belongs to. A tree's root must be a real directory, since listing a linked root would follow it, and a file must hold exactly the length read before hashing it, so one that changes meanwhile is an error rather than a digest of part of it. -- **Inputs.** `Batch::snapshot` copies the task, its fixtures and its plugins into the batch's `inputs/` once (`grade::copy_exact`), laid out as `evals/` is, loads the task again from the copy, and hashes the copy. `Batch::create_trial` takes only an `Inputs`, and refuses one recorded for another batch (`ForeignInputs`), so no trial can be made from live `evals/` or a mix of copies; grading takes the same `Inputs`' `evals()`. Tests reuse a batch's copy through `test_support::inputs`; so a change to `evals/` during a batch reaches no trial, and the content hash names what every trial ran. `inputs/` is under the evaluation root, which the policy denies, and no trial's `allowRead` reopens it. +- **Inputs.** `Batch::snapshot` copies the task, its fixtures and its plugins into the batch's `inputs/` once (`grade::copy_exact`), laid out as `evals/` is, loads the task again from the copy, and hashes the copy. `Batch::allocate_trial` takes only an `Inputs`, which `prepare` fills the trial from, and refuses one recorded for another batch (`ForeignInputs`) before creating anything, so no trial can be made from live `evals/` or a mix of copies; grading takes the same `Inputs`' `evals()`. Tests reuse a batch's copy through `test_support::inputs`; so a change to `evals/` during a batch reaches no trial, and the content hash names what every trial ran. `inputs/` is under the evaluation root, which the policy denies, and no trial's `allowRead` reopens it. - **Symposium setup.** `Trial::set_up_symposium(client, backend, env, deadline)` writes the trial's `~/.symposium/config.toml` (`auto-sync` on, `hook-scope = "global"`, `auto-update = "off"`, both built-in registries off, and one `[[registry]]` named `eval` at the trial's `registry/`), serialized from typed data so no path can break the TOML, then runs the staged `cargo-agents init --add-agent ` through `backend`, inside the trial's sandbox. `init` writes the config's agent and registers hooks in the trial's home without syncing; global scope matters because project-scoped hooks exist only after a sync, which would install skills before the behavior under test. A baseline trial is refused. `Client` names the two clients and the name `init` takes for each. Every trial's project starts with an empty `.claude/`: `srt` write-protects `.claude/commands` in its working directory, and where `.claude/` is missing it protects the whole directory instead, so a sync could not install a Claude Code skill. - **Pre-launch checks.** `Trial::check_before_launch(client, env)` runs just before a client starts, on the host, which is what the sandbox sees. Every trial must have no skill in the client's skill directory, in the project or the home, and nothing in the client's machine-wide managed-settings directory (`/etc/claude-code`, `/etc/codex`). A baseline trial must also have no `cargo-agents` in any directory of `env`'s `PATH`, no `~/.symposium`, and no hook or MCP configuration file that names `cargo-agents`. A failure is `RunError::NotReady`, naming the path and the check (`Unready`). `Client` (`run/client.rs`) holds where each client keeps these files. - **Exploratory mode.** Without `srt`, `Trial::exploratory_arguments(client, machine)` returns the arguments that confine the client to its own sandbox. For Claude Code it writes `claude-settings.json` at the trial root, outside every writable directory since Claude Code reloads a changed settings file, and returns `--settings --permission-mode acceptEdits`. The file turns the sandbox on strictly (`failIfUnavailable`, no unsandboxed retries, no network) and gives commands the trial's `Reach`, the same read and write rules as its `srt` policy (`Reach::for_trial` builds both), plus `blockReadsOutsideWorkingDirectories` for the file tools. Claude Code reads those paths as globs, so a path with `*`, `?`, or a bracket is refused. Those guarantees need Claude Code 2.1.285 or later: only from then does disabling unsandboxed retries through `--settings` stop a repository's settings from loosening the sandbox, so the client preflight must refuse older versions. For Codex it returns `--sandbox workspace-write` with the trial's cargo home, home and evidence as `sandbox_workspace_write.writable_roots`, passed through `--config` as a TOML array; `--add-dir` does not make a root writable on some versions (openai/codex issue 18448). Its reads stay open. In both clients hooks run outside the sandbox, and the API key is in the client's environment, which is why exploratory results are never compared. Setup runs `init` through the local backend. - **Environment.** `Trial::environment` is the whole environment of a trial's processes: `HOME`, `PATH`, `TMPDIR`, `CARGO_HOME`, `EVAL_EVIDENCE_DIR`, `SYMPOSIUM_HOOK_TRACE` (`evidence/hook-trace.jsonl`), and the API key. `PATH` is the trial's `bin/`, the toolchain's `bin/`, then `/usr/bin` and `/bin`. The real key is set on `srt` only, which masks it. The key's `Debug` never prints it, and neither does `Invocation`'s, which lists the environment's names only, since the key travels on as a plain value. - **Launching (Linux only).** `Srt` names the Node that runs `srt` and its `dist/cli.js`, since `srt` finds Node through `env node`, which the trial's `PATH` may not satisfy. `Srt::preflight` runs before a batch, so an unsuitable machine fails setup rather than the first trial: Node must be 22.12 or later, `srt` the pinned `SRT_VERSION`, and `true` must run through the whole stack (outer `bwrap`, Node, `srt`, a policy file with the same network proxy and TLS termination as a trial's), which proves `srt`'s own dependencies and the nested namespace without this code listing them. A tool that cannot start, answer in time, or exit successfully is reported by name, apart from one that reports the wrong version. `Srt::for_trial` refuses a trial whose `srt.json` has not been written. `Sandboxed` is the `grade::process::Backend` for one trial: it wraps a command as `srt --settings /srt.json -- ` and hands it to `Local`, which keeps the deadline, kills the process tree, and logs. `srt` runs the command through `bash -c` with `srt`'s own environment, so the trial's environment is set on `srt`. An outer `bwrap --dev-bind / / --dir /tmp/claude --bind /tmp /tmp/claude` gives each trial its own `/tmp/claude` and changes nothing else; `srt` builds its sandbox inside, with its own PID namespace, so killing the tree reaches processes that left the process group. Three ignored tests run real `srt` under the generated policy: one trial cannot see another's temporary files, the real API key appears in no environment the sandbox can read, and a process that calls `setsid` still dies at the deadline. Run them on Linux with `cargo test -p symposium-eval -- --ignored`. - **Smoke test.** `run::smoke` is RFD step 2's no-model smoke test under real `srt` and the generated policy, one `sh` script per trial that stops at the first failed check and names it. Each trial builds and tests its project offline; cannot read a home (a shell profile, and `~/.cargo/bin/cargo-agents`, also run by absolute path), a hidden check, a solution, the checkout's manifest, or another trial; and leaves its fixtures, its policy, the staged `cargo-agents`, the registry, and another trial's project unchanged. Only a Symposium trial can run the staged `cargo-agents`. The home is a decoy the test creates and adds to the denied user data roots, so the test does not depend on what the machine's home holds; the machine's own files are checked too when they exist. Paths reach the script as arguments, so any name works. Writes are checked on the host, not inside the sandbox, because `srt` covers a read-denied directory with an empty writable tmpfs, so a write there succeeds and is thrown away. Its tools are staged the way the runner stages them, and run by name from each trial's `bin/`: the machine's `node`, a small npm-style package run through that staged `node`, and a stand-in for `cargo-agents`, which a baseline trial cannot find on its `PATH`. Staging and launching the real clients and `cargo-agents` with `--version` comes with the adapters. +- **Running a trial.** `run::trial::Driver::run(number, arm, runner)` runs one trial end to end and writes its `result.json` in the trial's root, never over an existing one. It allocates the root first, so every later failure has a place to be recorded; then prepares the trial, writes its policy, sets up Symposium (Symposium trials only), snapshots the config, runs the pre-launch checks, and runs the session (`client::execute`). A failure before launch is `Reached::Setup` with its step; a client that never launched is `Reached::Session`, ungraded. Otherwise, for a Symposium trial, `expect::decide::gather` reads Symposium's evidence before grading, since `expect::ownership::symposium_copies` needs its installs to leave Symposium's own skill copies out of the agent's edits (`grade::Submitted::trial`); then grading runs, and `decide` decides the expectations. A grading sandbox that cannot be prepared is `GradeError::Sandbox`, so the trial is a grading exclusion. Only an allocation failure, parts that cannot make one result (`ResultError`), or a result that cannot be written are the runner's error (`TrialError`). The trial's client and requested model are named by `runner.client()` and `runner.model()` alone, so a result cannot record a model other than the one launched. Commands run through a private `Sandbox` trait (`for_trial`, `for_grading`), implemented only by `Srt` outside tests, so no run can go unconfined; tests play `init` and the client through their own. Two ignored Linux tests each run a `formatting` trial under real `srt`, with the `cargo-agents` built from the checkout and a Node stand-in for Claude Code (`claude-stand-in.js`), staged as an npm package and read by the real adapter: it runs the hooks `init` registered, and leaves `double` unformatted, so only the Symposium trial, whose `format-on-stop` hook formats it, succeeds. - **macOS.** The RFD makes enforced mode the default on macOS too, but there trials would share `/tmp/claude`, and with no PID namespace a process that leaves the process group outlives the deadline. Until both are solved, the runner has no macOS backend. - **Tests.** `Policy::can_read` and `can_write` model `srt`'s documented precedence (an `allowRead` reopens a less specific `denyRead`, never a more specific one), after Codex's `WritableRoot::is_path_writable`. The isolation invariants are tested through them: a trial cannot read another trial, the checkout, or a home; only a Symposium trial reads `tools/symposium`; a trial writes only its working directories. The policy tests never read the developer's home or the platform's roots: `Machine::with_exact_roots` takes them as arguments. The model leaves out `srt`'s built-in rules, which is exactly where the `/tmp/claude` gap lives, so these are the guarantees the policy is designed to give; only the no-model smoke test under `srt` (RFD step 2) shows they hold, and `run` lands with that caller. @@ -101,11 +102,13 @@ A `Batch` is one run of the runner: `//`, holding the staged t ## Client sessions -`client::execute(runner, backend, trial, env, prompt, skills, deadline)` runs one session and never returns an error: whatever goes wrong is how the session `Ended`, and the `Execution` also carries each skill's `Load`. A `ClientRunner` only builds the client's `Invocation` and reads its transcript, once, in `ClientRunner::read`, which returns a `Reading`: the completion (or why there is none, as `Unusable`) and the loads. The `Backend` sandboxes, logs, and stops the process, so a stand-in backend tests every ending without a client. A timed-out transcript is read too, since it can still show a mismatch or a load. A transcript is not read when the client never started, when the backend lost hold of it (`Unsupervised`), or when its output stayed open after the deadline, since whatever still holds the pipe may still be writing; its loads are then all `Unknown`. +The trial driver checks `Ended::containment_is_certain` before comparing configuration, gathering evidence, or importing the submission. An `Unsupervised` session, or a timeout with a cleanup error, instead reaches `Uncontained`: its result reports `reached.phase = "uncontained_session"`, a session failure, and no code or expectations. The result constructor rejects grading that state or using it for a contained session. This preserves the failed trial in outcome accounting without reading paths a surviving process may still replace. + +`client::execute(runner, backend, trial, env, prompt, skills, deadline)` runs one session and never returns an error: whatever goes wrong is how the session `Ended`, and the `Execution` also carries each skill's `Load`. `Execution::elapsed` is the agent's time: measured around `Backend::run` alone, so reading the transcript does not count. A `ClientRunner` names its client (`ClientRunner::client`, the only place a trial's client comes from), builds the client's `Invocation` and reads its transcript, once, in `ClientRunner::read`, which returns a `Reading`: the completion (or why there is none, as `Unusable`) and the loads. The `Backend` sandboxes, logs, and stops the process, so a stand-in backend tests every ending without a client. A timed-out transcript is read too, since it can still show a mismatch or a load. A transcript is not read when the client never started, when the backend lost hold of it (`Unsupervised`), or when its output stayed open after the deadline, since whatever still holds the pipe may still be writing; its loads are then all `Unknown`. - **Transcript.** The client's standard output, which the backend writes to the trial's `session/`, such as `session/claude.stdout.log`. The runner writes it from outside the sandbox, and the policy gives the trial no access to `session/`, so the agent cannot alter it. - **Client failures.** Only `Failed` is the client's: a reported error, which wins over the exit status since its message says more, then a failed exit (`Exited`), then a successful exit without a completion event (`NoCompletion`), so exiting 0 is never enough. -- **Not the client's.** `NotLaunched` (the program could not start: a setup failure, never a trial), `Unsupervised` (the runner lost hold of a started client), and `Unreadable`, carrying a `TranscriptError`: `Read` when the runner could not read the transcript, `Malformed` with a line number when an event the adapter needs is not in a shape it knows. An unreadable transcript decides the ending before the exit status does, so a changed event format is never counted as a failed session. `Completion::Missing` is only for a transcript the adapter fully read. `Mismatched`, carrying a `ProvenanceError`, is a transcript that is not this session's, or comes from a client other than the one checked before launch. The client launched, so it is an isolation exclusion, not a setup failure, and it outranks a timeout and the exit status. `Unusable` separates the two kinds of problem. +- **Not the client's.** `NotLaunched` (the program could not start, or a wrapper exited before the adapter observed the session introduction: a setup failure, never a trial), `Unsupervised` (the runner lost hold of a started client), and `Unreadable`, carrying a `TranscriptError`: `Read` when the runner could not read the transcript, `Malformed` with a line number when an event the adapter needs is not in a shape it knows. Each adapter names its introduction (`system/init` for Claude Code), and reports only whether it observed one. Absence means `NotLaunched` only after the transcript was read reliably; malformed, oversized or unreadable output cannot prove absence. A foreign session ID remains `Mismatched` even without an introduction. A matching result without an introduction is `NotLaunched`. Provenance mismatch outranks a timeout, a timeout otherwise outranks transcript damage, and supervision failure remains `Unsupervised` because its output may still change. `Completion::Missing` is only for a transcript the adapter fully read. `Mismatched`, carrying a `ProvenanceError`, is a transcript that is not this session's, or comes from a client other than the one checked before launch. The client launched, so it is an isolation exclusion, not a setup failure. `Unusable` separates provenance and transcript problems. - **Deadlines.** `TimedOut` stands once the runner killed the tree. `Local::run` returns a timeout even when collecting the output then fails, keeping that problem in `Finished::after_deadline`. A session keeps it in `Ended::TimedOut`, and a grading step in `Outcome::TimedOut` with its causes, so the report says why the output may be incomplete while still counting the submission's timeout. A preflight timeout keeps it as the source of `GradeError::TimedOut`. ### Claude Code @@ -138,7 +141,7 @@ A `Reading` also carries `ToolActivity`: the file mutations and shell commands t `expect::trace::session_start` decides whether the client's `SessionStart` hook ran and succeeded, from the trace Symposium appends to `SYMPOSIUM_HOOK_TRACE` (`Trial::hook_trace()`): one JSON line per hook call that reached a response, with its `event`, `agent` and `exit_code`. A call that failed before responding leaves no line. The reader needs only those three fields, holds at most 8 MiB of a line through `jsonl::next_line`, the bounded line reader it shares with the transcript, reads at most 64 MiB of the whole trace, checked before reading and again while reading in case it grew, and follows no link. A larger trace is `Unknown`: each line carries a hook's input, so a long session's trace reaches megabytes, but reading more than any session leaves would be work the agent chose. A `SessionStart` of the client that exited 0 is `Succeeded` and settles it; otherwise a line it could not read is `Unknown`, inconclusive; otherwise a call that exited otherwise is `Failed`, and none at all is `Missing`, both failing, since `check_before_launch` requires the trial to start with no trace (`Unready::HookTrace`). A failed auto-sync does not change the hook's exit, which is why the sync record is decided separately. The same real-hook test pattern reads the trace this reader expects. The bounded, link-free reading is `expect::log`, shared by every log a process inside the trial writes. -`expect::hooks::hook_record` decides each `[expected] hooks` entry. Loading a task finds each entry's event in the `[[hooks]]` of its plugins' `Symposium.toml`, with the checker's own reader of `name`, `event`, `agent` and `format` (`task::hooks`), and `--check-tasks` rejects one that no plugin declares, that two declare, whose event is not one of Symposium's five, or that is limited to one agent by `agent` or a `format` other than `symposium`: a task runs on every client. A manifest that cannot be read or parsed is reported alone, since it could declare any hook or skill; `task::manifest` reads each plugin once, for hooks and skills both, and draws no conclusion when a plugin directory is missing: it lists the plugin's tree, refusing a link or special file anywhere in it, since `copy_tree` would refuse it when creating a trial, and checks the manifest against the whole schema `symposium-sdk` defines (`RawPluginManifest`, which refuses unknown fields and holds each hook to Symposium's events) before its own reader takes the fields it needs, since Symposium rejects a plugin whose manifest does not match. Symposium's own validation goes further (a registry plugin must name itself, retired fields, references to unknown installations); the checker does not repeat it. `task::skills` finds the `SKILL.md` behind each expected skill as Symposium finds a registry plugin's: every `SKILL.md` under the plugin's `[[skills]] source.path` groups, or the one at the root of a plugin without a manifest, skipping one nested in another skill or in a directory holding an entry named `.symposium`, sync's marker, of any kind, as Symposium checks only that the name exists. A skill counts only if Symposium would load it (`skill_md::loadable_name`): frontmatter that is a YAML mapping with string keys, no retired `crates` field, `name`, `description`, `depends-on` and `predicates` as strings, a `name` that is a plain directory name, and a `description` that is not empty and at most 1,024 bytes; `depends-on` and `predicates` must parse with `symposium-sdk`'s predicate syntax, the shared schema, so the checker does not keep a second parser. Otherwise a broken task would look like a Symposium failure. One `SKILL.md` reached through two groups is one provider, as sync counts it, so only distinct origins are ambiguous. A skill in `installed` or `used` needs exactly one provider, and any named skill may have at most one: with two, sync would install neither under its own name. A source must be a plain relative path to a directory in the plugin, since only the plugin is copied into the trial; the skill reader reads the plugin's one listing rather than walking each source again. `Task::skill_source` gives `gather` each skill's directory in `evals/`, and `expect::decide::installed_skills` gives the adapter the path each named skill installs at: a trial starts with no skills and each name has one provider, so the copy takes the skill's own name. Which events a client delivers is not checked, so a hook Symposium does not deliver on a client fails there as a missing record. The trace shows Symposium handled an event, not which plugin hooks ran within it, so each hook in `evals/` records its own runs: with `EVAL_EVIDENCE_DIR` set, it appends a JSON line naming itself and its exit code to `hook-records.jsonl` (`Trial::hook_records()`), one file for all hooks, so a hook's name never becomes a path. `format-on-stop` also records whether the code was already formatted. A `Stop` hook is decided by its first record (`Counted::First`), as the RFD asks, and a line before it that cannot be read could have been it, so that is `Unknown`. `check_before_launch` requires no records file (`Unready::HookRecord`). A Linux test runs Symposium's real `Stop` hook, which runs the plugin's, and reads its record. +`expect::hooks::hook_record` decides each `[expected] hooks` entry. Loading a task finds each entry's event in the `[[hooks]]` of its plugins' `Symposium.toml`, with the checker's own reader of `name`, `event`, `agent` and `format` (`task::hooks`), and `--check-tasks` rejects one that no plugin declares, that two declare, whose event is not one of Symposium's five, or that is limited to one agent by `agent` or a `format` other than `symposium`: a task runs on every client. A manifest that cannot be read or parsed is reported alone, since it could declare any hook or skill; `task::manifest` reads each plugin once, for hooks and skills both, and draws no conclusion when a plugin directory is missing: it lists the plugin's tree, refusing a link or special file anywhere in it, since `copy_tree` would refuse it when creating a trial, and checks the manifest against the whole schema `symposium-sdk` defines (`RawPluginManifest`, which refuses unknown fields and holds each hook to Symposium's events) before its own reader takes the fields it needs, since Symposium rejects a plugin whose manifest does not match. Symposium's own validation goes further (a registry plugin must name itself, retired fields, references to unknown installations); the checker does not repeat it. `task::skills` catalogues the `SKILL.md` behind every provided skill as Symposium finds a registry plugin's: every `SKILL.md` under the plugin's `[[skills]] source.path` groups, or the one at the root of a plugin without a manifest, skipping one nested in another skill or in a directory holding an entry named `.symposium`, sync's marker, of any kind, as Symposium checks only that the name exists. A skill counts only if Symposium would load it (`skill_md::loadable_name`): frontmatter that is a YAML mapping with string keys, no retired `crates` field, `name`, `description`, `depends-on` and `predicates` as strings, a `name` that is a plain directory name, and a `description` that is not empty and at most 1,024 bytes; `depends-on` and `predicates` must parse with `symposium-sdk`'s predicate syntax, the shared schema, so the checker does not keep a second parser. Otherwise a broken task would look like a Symposium failure. One `SKILL.md` reached through two groups is one provider, as sync counts it, so only distinct origins are ambiguous. A skill in `installed` or `used` needs exactly one provider, and any named skill may have at most one: with two, sync would install neither under its own name. A source must be a plain relative path to a directory in the plugin, since only the plugin is copied into the trial; the skill reader reads the plugin's one listing rather than walking each source again. `Task::provided_skills` gives ownership the complete source catalogue, including unnamed skills and distinct origins with the same name. `Task::skill_source` gives `gather` a skill's source only when it has one provider, and `expect::decide::installed_skills` gives the adapter the path each named skill installs at: a trial starts with no skills and each name has one provider, so the copy takes the skill's own name. Which events a client delivers is not checked, so a hook Symposium does not deliver on a client fails there as a missing record. The trace shows Symposium handled an event, not which plugin hooks ran within it, so each hook in `evals/` records its own runs: with `EVAL_EVIDENCE_DIR` set, it appends a JSON line naming itself and its exit code to `hook-records.jsonl` (`Trial::hook_records()`), one file for all hooks, so a hook's name never becomes a path. `format-on-stop` also records whether the code was already formatted. A `Stop` hook is decided by its first record (`Counted::First`), as the RFD asks, and a line before it that cannot be read could have been it, so that is `Unknown`. `check_before_launch` requires no records file (`Unready::HookRecord`). A Linux test runs Symposium's real `Stop` hook, which runs the plugin's, and reads its record. `expect::config::Snapshot` copies the client's hook config, in the home (`Client::hook_config`) and the project (`Client::project_config`), and Symposium's `config.toml` after `init`, and `changes` compares them after the trial, byte for byte. Claude Code applies a hook edit mid-session, so the project files matter too. A file unreadable either time is a change, since it cannot be shown unchanged. Claude Code's `.claude.json` is not watched: Claude Code rewrites it mid-session. @@ -146,7 +149,7 @@ A `Reading` also carries `ToolActivity`: the file mutations and shell commands t `outcome::decide` places a trial in the RFD's count from how far it got (`Session::NotStarted`, or `Session::Graded` with its `Ended` and `Grading`) and its expectations' reported verdicts (`Expectations::verdicts`, none for a baseline). It reads nothing, and its order is the precedence: a client that never launched (`NotStarted`, or `Ended::NotLaunched`) is a setup failure; a mismatched transcript is an isolation exclusion whatever the trial produced; a session that did not complete fails (`Failure::Session`) even when grading then breaks: a timeout, a client error, and also `Unsupervised` and `Unreadable`, since excluding a session the runner lost hold of would let a configuration that breaks supervision (a process left holding the client's output) score better by shrinking its count. For a completed session, an invalid submission or a check that did not pass fails; a regression trial also fails on a failed expectation, and grading's own failure excludes the trial only when it was still needed, so a failed expectation keeps the trial failed. Then a regression trial with an inconclusive expectation is inconclusive. A capability trial is decided by its code alone. Edits outside `editable` are reported, never graded. Cancellation is the batch's, decided before a trial runs. -`result` writes `result.json`. As cargo (`cargo metadata`, JSON messages), rustc (JSON diagnostics), rustdoc (`rustdoc-json-types`) and nextest (`nextest-metadata`) do, the schema is its own set of types, named by serde, built from the runner's by small `From` conversions, so renaming a type in the runner never changes it. Each file holds one part's record types with their conversions: `provenance`, `reached` (the phase reached and the outcome, decided there with `outcome::decide` so the two always agree), `session`, `code`, `expectations` and `concern` (a stable `reason` code and a `detail` for people); `TrialResult::new` only assembles them. `SCHEMA_VERSION` changes only when a field is removed or changes meaning. A phase's output is `null` exactly when the trial stopped before it; a value a phase reached but could not know has its own variant (`usage: unknown`), as nextest's `status: unavailable` does. Errors keep their causes and are cut to 4 KiB. A trial's configuration is an `Arm`, so only a Symposium trial can carry a `cargo-agents` build, and `TrialResult::new` refuses parts that cannot belong to one trial (`ResultError`): expectations on a graded baseline trial, none on a graded Symposium trial, or a different number of loads than skills looked for, which would otherwise drop or misname a load. `TrialResult::write` writes and syncs the file beside the destination, then names it with `persist_noclobber`, so an existing result is never replaced; on Linux and macOS, where trials run, that is a rename that refuses to replace or a hard link, so the name appears only for the complete file. Elsewhere `tempfile` promises less, and the directory is not synced, so a crash may lose the name. Tests pin each shape as an `insta` snapshot of the whole JSON, with hashes redacted. +`result` writes `result.json`. As cargo (`cargo metadata`, JSON messages), rustc (JSON diagnostics), rustdoc (`rustdoc-json-types`) and nextest (`nextest-metadata`) do, the schema is its own set of types, named by serde, built from the runner's by small `From` conversions, so renaming a type in the runner never changes it. Each file holds one part's record types with their conversions: `provenance`, `reached` (the phase reached and the outcome, decided there with `outcome::decide` so the two always agree), `session`, `code`, `expectations` and `concern` (a stable `reason` code and a `detail` for people); `TrialResult::new` only assembles them. `SCHEMA_VERSION` changes only when a field is removed or changes meaning. A phase's output is `null` exactly when the trial stopped before it. A client that could not be launched reaches `launch` (`Reached::Session`): its session is kept, ending `not_launched` with the spawn error, nothing is graded, so `code` and `expectations` are `null`, and the outcome is a setup failure; An uncontained session instead reaches `uncontained_session` with no filesystem evidence, code, or expectations, and remains a failed session in the count. `TrialResult::new` rejects grading an uncontained session or assigning that phase to a contained one (`ResultError::SessionContainment`), as well as grading a client that never launched (`ResultError::SessionPhase`). A value a phase reached but could not know has its own variant (`usage: unknown`), as nextest's `status: unavailable` does. Errors keep their causes and are cut to 4 KiB. A trial's configuration is an `Arm`, so only a Symposium trial can carry a `cargo-agents` build, and `TrialResult::new` refuses parts that cannot belong to one trial (`ResultError`): expectations on a graded baseline trial, none on a graded Symposium trial, or a different number of loads than skills looked for, which would otherwise drop or misname a load. `TrialResult::write` writes and syncs the file beside the destination, then names it with `persist_noclobber`, so an existing result is never replaced; on Linux and macOS, where trials run, that is a rename that refuses to replace or a hard link, so the name appears only for the complete file. Elsewhere `tempfile` promises less, and the directory is not synced, so a crash may lose the name. Tests pin each shape as an `insta` snapshot of the whole JSON, with hashes redacted. ## Code comments diff --git a/md/rfds/skill-evaluation/reference.md b/md/rfds/skill-evaluation/reference.md index ef6907b9e..c64bd85c4 100644 --- a/md/rfds/skill-evaluation/reference.md +++ b/md/rfds/skill-evaluation/reference.md @@ -240,7 +240,7 @@ A hook record contradicted by the transcript, including more records than the tr ## Outcomes and counting -A trial can succeed only if its client session completed. A timeout or client error is a failure even when the code it left would pass, and so is a session the runner lost hold of or whose ending its transcript cannot show, since it did not complete either; that code is still graded and reported, and a grading infrastructure error during it does not move the trial out of the count. For a completed trial: +A trial can succeed only if its client session completed. A timeout or client error is a failure even when the code it left would pass, and so is a session the runner lost hold of or whose ending its transcript cannot show, since it did not complete either. Its code is still graded and reported when process containment is certain, and a grading infrastructure error during it does not move the trial out of the count. When the runner cannot prove every writer stopped, it records an `uncontained_session` failure without inspecting configuration, collecting filesystem evidence, or grading the submission; `code` and `expectations` are then `null`. For a completed trial: | Kind | Succeeds when | Inconclusive when | | ---------- | ---------------------------------------------- | --------------------------------------------------------------- | diff --git a/symposium-eval/src/client/claude.rs b/symposium-eval/src/client/claude.rs index d5867c810..bf7a7e4e8 100644 --- a/symposium-eval/src/client/claude.rs +++ b/symposium-eval/src/client/claude.rs @@ -79,6 +79,14 @@ impl Claude { } impl ClientRunner for Claude { + fn client(&self) -> Client { + Client::ClaudeCode + } + + fn model(&self) -> &str { + &self.model + } + fn invocation( &self, trial: &Trial, @@ -182,6 +190,22 @@ mod tests { ); } + #[test] + fn the_model_launched_is_the_model_reported() { + let host = Host::new(); + let trial = trial(&host.batch(), 1, Configuration::Symposium); + let claude = Claude::enforced(model(), version::MINIMUM); + + let invocation = claude.invocation(&trial, "do the task", Vec::new()); + + let flag = invocation.args.iter().position(|arg| arg == "--model"); + let launched = flag.and_then(|flag| invocation.args.get(flag + 1)); + assert_eq!( + launched.map(|model| model.as_os_str()), + Some(claude.model().as_ref()) + ); + } + #[test] fn an_exploratory_session_uses_the_trials_own_sandbox() { let host = Host::new(); diff --git a/symposium-eval/src/client/claude/transcript.rs b/symposium-eval/src/client/claude/transcript.rs index b99683a7c..d2b431f15 100644 --- a/symposium-eval/src/client/claude/transcript.rs +++ b/symposium-eval/src/client/claude/transcript.rs @@ -24,8 +24,8 @@ use super::hooks::{HookFields, Hooks}; use super::loads::Skills; use super::version::Version; use crate::client::{ - Completion, InstalledSkill, Load, Malformation, ProvenanceError, Reading, Summary, Tokens, - TranscriptError, Unusable, Usage, + Completion, InstalledSkill, Load, Malformation, ProvenanceError, Reading, SessionIntroduction, + Summary, Tokens, TranscriptError, Unusable, Usage, }; use crate::jsonl::{self, Line, MAX_LINE}; @@ -102,6 +102,10 @@ struct Reader<'a> { /// Whether an `init` has been seen. One that cannot be read counts: the transcript /// is then malformed, not unattributed. introduced: bool, + /// A matching result appeared before an introduction. This is not itself a + /// provenance error: if no introduction follows, it proves only that the client + /// never reached the session boundary. + ended_before_introduction: bool, /// Why the transcript cannot say how the session ended, if it cannot. unusable: Option, calls: Calls, @@ -158,6 +162,7 @@ impl<'a> Reader<'a> { usage: Usage::Unknown, response: None, introduced: false, + ended_before_introduction: false, unusable: None, calls: Calls::new(project), hooks: Hooks::new(), @@ -241,12 +246,19 @@ impl<'a> Reader<'a> { Ok(()) => self.model = init.model, Err(error) => self.mismatched(error), } + if self.ended_before_introduction { + self.mismatched(ProvenanceError::Unstated); + } } fn ending(&mut self, ending: Ending<'_>) { + if !self.is_ours(&ending.session_id) { + return; + } if !self.introduced { - self.mismatched(ProvenanceError::Unstated); - } else if self.is_ours(&ending.session_id) { + self.ended_before_introduction = true; + self.damage(); + } else { self.usage = ending.usage(); (self.completion, self.response) = ending.completion(); self.calls.finish(); @@ -334,6 +346,11 @@ impl<'a> Reader<'a> { }, }; Reading { + introduction: if self.introduced { + SessionIntroduction::Observed + } else { + SessionIntroduction::NotObserved + }, summary, completion: self.unusable.map_or(Ok(self.completion), Err), loads: (self.calls.outcomes(&Skills::new(self.skills))) @@ -427,6 +444,10 @@ mod tests { session: Uuid, version: Version, ) -> Result { + reading_as(transcript, session, version).completion + } + + fn reading_as(transcript: &str, session: Uuid, version: Version) -> Reading { read_text( transcript, Path::new("/trial/project"), @@ -434,7 +455,6 @@ mod tests { version, &[], ) - .completion } /// As the recorded session, run by the version it was recorded with. @@ -872,23 +892,84 @@ mod tests { } #[test] - fn a_result_without_an_init_is_unstated() { + fn a_matching_result_without_an_init_only_shows_no_introduction() { let transcript = without(FINISHED, "system/init"); + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); - assert!(matches!( - completion(&transcript), - Err(Unusable::Provenance(ProvenanceError::Unstated)) - )); + assert_eq!(reading.completion.unwrap(), Completion::Missing); + assert_eq!(reading.introduction, SessionIntroduction::NotObserved); } #[test] fn a_result_before_the_init_is_unstated() { let transcript = format!("{}\n{FINISHED}", recorded_result()); + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); assert!(matches!( - completion(&transcript), + reading.completion, Err(Unusable::Provenance(ProvenanceError::Unstated)) )); + assert_eq!(reading.introduction, SessionIntroduction::Observed); + } + + #[test] + fn a_foreign_result_without_an_init_is_mismatched() { + let other = Uuid::new_v4(); + let transcript = format!("{}\n", as_session(recorded_result(), other)); + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); + + assert!(matches!( + reading.completion, + Err(Unusable::Provenance(ProvenanceError::Session { .. })) + )); + assert_eq!(reading.introduction, SessionIntroduction::NotObserved); + } + + #[test] + fn a_foreign_message_without_an_init_is_mismatched() { + let other = Uuid::new_v4(); + let assistant = first(FINISHED, "assistant"); + let transcript = format!("{}\n", as_session(assistant, other)); + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); + + assert!(matches!( + reading.completion, + Err(Unusable::Provenance(ProvenanceError::Session { .. })) + )); + assert_eq!(reading.introduction, SessionIntroduction::NotObserved); + } + + #[test] + fn transcript_damage_outranks_a_matching_result_without_an_init_in_either_order() { + for transcript in [ + format!("not json\n{}\n", recorded_result()), + format!("{}\nnot json\n", recorded_result()), + ] { + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); + + assert!(matches!( + reading.completion, + Err(Unusable::Transcript(TranscriptError::Malformed { .. })) + )); + assert_eq!(reading.introduction, SessionIntroduction::NotObserved); + } + } + + #[test] + fn foreign_evidence_outranks_a_matching_result_without_an_init_in_either_order() { + let foreign = as_session(first(FINISHED, "assistant"), Uuid::new_v4()); + for transcript in [ + format!("{}\n{foreign}\n", recorded_result()), + format!("{foreign}\n{}\n", recorded_result()), + ] { + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); + + assert!(matches!( + reading.completion, + Err(Unusable::Provenance(ProvenanceError::Session { .. })) + )); + assert_eq!(reading.introduction, SessionIntroduction::NotObserved); + } } #[test] @@ -963,8 +1044,26 @@ mod tests { fn an_init_without_its_version_is_malformed() { let init = without_field(recorded_init(), "claude_code_version"); let transcript = format!("{init}\n{}\n", recorded_result()); + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); - assert_eq!(malformed_line(completion(&transcript)), 1); + assert_eq!(malformed_line(reading.completion), 1); + assert_eq!(reading.introduction, SessionIntroduction::Observed); + } + + #[test] + fn damaged_output_without_an_init_does_not_prove_absence() { + for transcript in [ + "not json\n".to_owned(), + format!("{}\n", "x".repeat(MAX_LINE + 1)), + ] { + let reading = reading_as(&transcript, recorded_session(), version::MINIMUM); + + assert!(matches!( + reading.completion, + Err(Unusable::Transcript(TranscriptError::Malformed { .. })) + )); + assert_eq!(reading.introduction, SessionIntroduction::NotObserved); + } } #[test] diff --git a/symposium-eval/src/client/error.rs b/symposium-eval/src/client/error.rs index 5a2438625..a5b2c247e 100644 --- a/symposium-eval/src/client/error.rs +++ b/symposium-eval/src/client/error.rs @@ -6,6 +6,19 @@ use std::process::ExitStatus; use thiserror::Error; +use crate::grade::ProcessError; + +/// Why a client never reached the beginning of a session. +#[derive(Debug, Error)] +pub(crate) enum LaunchError { + /// The runner could not start the client process. + #[error(transparent)] + Process(#[from] ProcessError), + /// A wrapper process started, but the client exited before introducing a session. + #[error("the client exited with {status} before starting a session: {stderr}")] + Exited { status: ExitStatus, stderr: String }, +} + /// Why a session failed, by the client's own doing. #[derive(Debug, Error)] pub(crate) enum ClientError { diff --git a/symposium-eval/src/client/mod.rs b/symposium-eval/src/client/mod.rs index 6adce091f..9c562ff8f 100644 --- a/symposium-eval/src/client/mod.rs +++ b/symposium-eval/src/client/mod.rs @@ -14,17 +14,27 @@ pub(crate) mod paths; use std::ffi::OsString; use std::path::{Path, PathBuf}; use std::process::ExitStatus; -use std::time::Duration; +use std::time::{Duration, Instant}; -pub(crate) use error::{ClientError, Malformation, ProvenanceError, TranscriptError, Unusable}; +pub(crate) use error::{ + ClientError, LaunchError, Malformation, ProvenanceError, TranscriptError, Unusable, +}; use crate::grade::ProcessError; use crate::grade::process::{Backend, Finished, Invocation}; use crate::run::Trial; +use crate::run::client::Client; use crate::task::SkillName; /// One client, as the runner drives it. pub(crate) trait ClientRunner { + /// Which client this is, the only place a trial's client is named. + fn client(&self) -> Client; + + /// The model its sessions are asked to run, as requested: the only place a trial's + /// model is named. + fn model(&self) -> &str; + /// The command that starts a new session in `trial` and runs `prompt`, with `env` as /// its whole environment. fn invocation(&self, trial: &Trial, prompt: &str, env: Vec<(OsString, OsString)>) @@ -45,6 +55,9 @@ pub(crate) trait ClientRunner { /// What one transcript says. #[derive(Debug)] pub(crate) struct Reading { + /// Whether the adapter observed the event that introduces a session. This says + /// only what was observed; transcript errors decide whether absence is reliable. + pub(crate) introduction: SessionIntroduction, /// How the session ended, unless the transcript cannot be read, an event the adapter /// needs is malformed, or it is not this session's, from the client checked before /// launch. @@ -56,6 +69,13 @@ pub(crate) struct Reading { pub(crate) summary: Summary, } +/// Whether a client's transcript introduced its session. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum SessionIntroduction { + Observed, + NotObserved, +} + /// What a transcript says about the hooks its client ran, a record the agent cannot /// write (RFD reference, "What the evidence can and cannot show"). #[derive(Debug, Clone, PartialEq, Eq)] @@ -306,6 +326,8 @@ pub(crate) struct Execution { /// client never started, and not evidence when the backend lost hold of the output. pub(crate) transcript: PathBuf, pub(crate) ended: Ended, + /// How long the backend ran the client, transcript reading left out. + pub(crate) elapsed: Duration, pub(crate) loads: Vec, pub(crate) hooks: HookEvidence, pub(crate) activity: ToolActivity, @@ -323,7 +345,7 @@ pub(crate) enum Ended { }, Failed(ClientError), /// The client never started: a setup failure, not a trial. - NotLaunched(ProcessError), + NotLaunched(LaunchError), /// The runner could not supervise the client once it started. Unsupervised(ProcessError), /// The client exited, but its transcript cannot say how the session ended, whatever @@ -334,6 +356,19 @@ pub(crate) enum Ended { Mismatched(ProvenanceError), } +impl Ended { + /// Whether every process that could still change the trial was proved stopped. + pub(crate) fn containment_is_certain(&self) -> bool { + !matches!( + self, + Self::Unsupervised(_) + | Self::TimedOut { + after_deadline: Some(_) + } + ) + } +} + /// Runs `prompt` as a new session of `runner`'s client in `trial`, through `backend`, /// until it ends or `deadline` passes, looking for `skills` in its transcript. Never an /// error: whatever goes wrong is how the session ended. @@ -354,17 +389,23 @@ pub(crate) fn execute( let invocation = runner.invocation(trial, prompt, env); let logs = trial.session(); let transcript = invocation.log(&logs, "stdout"); + let started = Instant::now(); + let ran = backend.run(&invocation, deadline, &logs); + let elapsed = started.elapsed(); let unread = |ended| Execution { transcript: transcript.clone(), ended, + elapsed, loads: vec![Load::Unknown; skills.len()], hooks: runner.unread_hooks(), activity: ToolActivity::unread(), summary: Summary::unread(runner.unread_usage()), }; - let finished = match backend.run(&invocation, deadline, &logs) { + let finished = match ran { Ok(finished) => finished, - Err(error @ ProcessError::Spawn { .. }) => return unread(Ended::NotLaunched(error)), + Err(error @ ProcessError::Spawn { .. }) => { + return unread(Ended::NotLaunched(error.into())); + } // Every other error comes after the client started. Output the runner lost hold // of may still be written, so the transcript is not evidence. Err(error) => return unread(Ended::Unsupervised(error)), @@ -376,8 +417,8 @@ pub(crate) fn execute( }); } let reading = runner.read(trial, &transcript, skills); - let ended = match (finished, reading.completion) { - (_, Err(Unusable::Provenance(error))) => Ended::Mismatched(error), + let ended = match (finished, reading.introduction, reading.completion) { + (_, _, Err(Unusable::Provenance(error))) => Ended::Mismatched(error), ( Finished { status: None, @@ -385,20 +426,35 @@ pub(crate) fn execute( .. }, _, + _, ) => Ended::TimedOut { after_deadline }, - (_, Err(Unusable::Transcript(error))) => Ended::Unreadable(error), + (_, _, Err(Unusable::Transcript(error))) => Ended::Unreadable(error), ( Finished { status: Some(status), stderr, .. }, + SessionIntroduction::NotObserved, + _, + ) => Ended::NotLaunched(LaunchError::Exited { + status, + stderr: stderr.tail().trim().to_owned(), + }), + ( + Finished { + status: Some(status), + stderr, + .. + }, + SessionIntroduction::Observed, Ok(completion), ) => ending(status, completion, || stderr.tail().trim().to_owned()), }; Execution { transcript, ended, + elapsed, loads: reading.loads, hooks: reading.hooks, activity: reading.activity, @@ -434,6 +490,14 @@ mod tests { struct OneLine; impl ClientRunner for OneLine { + fn client(&self) -> Client { + Client::ClaudeCode + } + + fn model(&self) -> &str { + "a-model" + } + fn invocation( &self, trial: &Trial, @@ -464,6 +528,12 @@ mod tests { _ => Load::Unknown, }; Reading { + introduction: match text.as_deref().map(str::trim) { + Ok("") | Ok("foreign") | Ok("garbled") | Ok("result-without-introduction") => { + SessionIntroduction::NotObserved + } + _ => SessionIntroduction::Observed, + }, completion: text .map_err(|source| { TranscriptError::Read { @@ -494,7 +564,14 @@ mod tests { } .into()); } - "foreign" => return Err(ProvenanceError::Unstated.into()), + "foreign" => { + return Err(ProvenanceError::Session { + requested: "this-session".to_owned(), + found: "another-session".to_owned(), + } + .into()); + } + "result-without-introduction" => Completion::Missing, _ => Completion::Missing, }) } @@ -602,6 +679,87 @@ mod tests { .ended } + /// Runs for its duration, then writes a finished transcript. + struct Slow(Duration); + + impl Backend for Slow { + fn run( + &self, + invocation: &Invocation, + _: Duration, + logs: &Path, + ) -> Result { + std::thread::sleep(self.0); + Replay { + transcript: "finished\n", + status: Some(0), + output_stayed_open: false, + } + .run(invocation, Duration::ZERO, logs) + } + } + + /// [`OneLine`], taking its duration to read the transcript. + struct SlowReader(Duration); + + impl ClientRunner for SlowReader { + fn client(&self) -> Client { + OneLine.client() + } + + fn model(&self) -> &str { + "a-model" + } + + fn invocation( + &self, + trial: &Trial, + prompt: &str, + env: Vec<(OsString, OsString)>, + ) -> Invocation { + OneLine.invocation(trial, prompt, env) + } + + fn unread_hooks(&self) -> HookEvidence { + OneLine.unread_hooks() + } + + fn unread_usage(&self) -> Usage { + OneLine.unread_usage() + } + + fn read(&self, trial: &Trial, transcript: &Path, skills: &[InstalledSkill]) -> Reading { + std::thread::sleep(self.0); + OneLine.read(trial, transcript, skills) + } + } + + #[test] + fn a_sessions_time_is_the_clients_and_not_its_reading() { + let host = Host::new(); + let trial = trial(&host.batch(), 1, Configuration::Baseline); + let (running, reading) = (Duration::from_millis(100), Duration::from_millis(100)); + + let started = Instant::now(); + let execution = execute( + &SlowReader(reading), + &Slow(running), + &trial, + Vec::new(), + "do the task", + &[greeting()], + Duration::from_secs(60), + ); + let total = started.elapsed(); + + assert!(matches!(execution.ended, Ended::Completed), "{execution:?}"); + assert!(execution.elapsed >= running, "{:?}", execution.elapsed); + assert!( + total.saturating_sub(execution.elapsed) >= reading, + "{total:?}" + ); + } + #[test] fn a_finished_session_is_completed_and_its_transcript_kept() { let (_host, trial, execution) = run(&Replay { @@ -699,7 +857,7 @@ mod tests { assert!( matches!( execution.ended, - Ended::NotLaunched(ProcessError::Spawn { .. }) + Ended::NotLaunched(LaunchError::Process(ProcessError::Spawn { .. })) ), "{execution:?}" ); @@ -707,6 +865,40 @@ mod tests { assert_eq!(execution.hooks, HookEvidence::Unsupported); } + #[test] + fn a_clean_exit_before_the_session_introduction_is_not_launched() { + for status in [0, 126, 127] { + let ended = replay("", Some(status)); + + assert!( + matches!(ended, Ended::NotLaunched(LaunchError::Exited { .. })), + "{ended:?}" + ); + } + } + + #[test] + fn a_matching_result_without_an_introduction_is_not_launched() { + let ended = replay("result-without-introduction\n", Some(0)); + + assert!( + matches!(ended, Ended::NotLaunched(LaunchError::Exited { .. })), + "{ended:?}" + ); + } + + #[test] + fn an_exit_after_the_session_introduction_is_a_session_failure() { + for status in [126, 127] { + let ended = replay("finished\n", Some(status)); + + assert!( + matches!(ended, Ended::Failed(ClientError::Exited { .. })), + "{ended:?}" + ); + } + } + #[test] fn losing_hold_of_a_started_client_is_the_runners_problem() { let (_host, _, execution) = run(&Failing(|| ProcessError::OutputNotClosed)); @@ -735,6 +927,16 @@ mod tests { } } + #[test] + fn foreign_evidence_without_an_introduction_is_still_mismatched() { + let ended = replay("foreign\n", Some(126)); + + assert!( + matches!(ended, Ended::Mismatched(ProvenanceError::Session { .. })), + "{ended:?}" + ); + } + #[test] fn a_transcript_the_runner_cannot_read_is_unreadable_whatever_the_exit() { for status in [0, 1] { @@ -756,7 +958,7 @@ mod tests { let ended = replay("foreign\n", Some(status)); assert!( - matches!(ended, Ended::Mismatched(ProvenanceError::Unstated)), + matches!(ended, Ended::Mismatched(ProvenanceError::Session { .. })), "{ended:?}" ); } @@ -789,7 +991,7 @@ mod tests { let ended = replay("foreign\n", None); assert!( - matches!(ended, Ended::Mismatched(ProvenanceError::Unstated)), + matches!(ended, Ended::Mismatched(ProvenanceError::Session { .. })), "{ended:?}" ); } diff --git a/symposium-eval/src/expect/installed.rs b/symposium-eval/src/expect/installed.rs index 8be243cb2..49ed91675 100644 --- a/symposium-eval/src/expect/installed.rs +++ b/symposium-eval/src/expect/installed.rs @@ -118,7 +118,7 @@ fn installed( skill: &SkillName, source: &Path, ) -> (InstallEvidence, Option) { - let Ok(candidates) = candidates(skills, skill) else { + let Ok(candidates) = candidates(skills, skill.as_str()) else { return (InstallEvidence::Unreadable, None); }; let mut unreadable = candidates.unreadable; @@ -143,7 +143,7 @@ fn installed( /// Whether `skills` holds any copy of `skill`: one settles it, whatever else is there. fn presence(skills: &Path, skill: &SkillName) -> InstallEvidence { - match candidates(skills, skill) { + match candidates(skills, skill.as_str()) { Ok(candidates) if !candidates.copies.is_empty() => InstallEvidence::Present, Ok(candidates) if candidates.unreadable => InstallEvidence::Unreadable, Ok(candidates) if candidates.unidentified => InstallEvidence::Unidentified, @@ -165,7 +165,7 @@ struct Candidates { /// What `skills` holds for `skill`. A missing `skills` holds nothing. An entry that /// cannot be inspected is noted and passed over, so it cannot hide a copy that settles. -fn candidates(skills: &Path, skill: &SkillName) -> io::Result { +fn candidates(skills: &Path, skill: &str) -> io::Result { let entries = match fs::read_dir(skills) { Ok(entries) => entries, Err(error) if error.kind() == io::ErrorKind::NotFound => { @@ -186,11 +186,19 @@ fn candidates(skills: &Path, skill: &SkillName) -> io::Result { Ok(candidates) } +/// Marked copies of a provided skill, independently of install assertions. An +/// unreadable or unidentified entry cannot establish ownership and is left out. +pub(super) fn copy_candidates(skills: &Path, skill: &str) -> Vec { + candidates(skills, skill) + .map(|candidates| candidates.copies) + .unwrap_or_default() +} + /// The directory `entry` is and whose copy, if it is one of sync's copies that could be /// `skill`'s. fn candidate( entry: io::Result, - skill: &SkillName, + skill: &str, ) -> io::Result> { let entry = entry?; let Some(slot) = entry @@ -219,8 +227,8 @@ enum Slot { Hashed, } -fn slot(name: &str, skill: &SkillName) -> Option { - match name.strip_prefix(skill.as_str())? { +fn slot(name: &str, skill: &str) -> Option { + match name.strip_prefix(skill)? { "" => Some(Slot::Plain), rest => { let hash = rest.strip_prefix('-')?; @@ -252,14 +260,14 @@ enum Identity { /// Whose copy `dir` is, by the name in its `SKILL.md`, which sync names a copy after. /// The agent can write there, so the file is opened as evidence is, which a pipe cannot /// block. -fn identity(dir: &Path, skill: &SkillName) -> io::Result { +fn identity(dir: &Path, skill: &str) -> io::Result { let Some(bytes) = file::read(&dir.join("SKILL.md"), MAX_SKILL_MD)? else { return Ok(Identity::Unknown); }; let text = String::from_utf8(bytes) .map_err(|error| io::Error::new(io::ErrorKind::InvalidData, error))?; Ok(match skill_md::name(&text) { - Some(name) if name == skill.as_str() => Identity::Expected, + Some(name) if name == skill => Identity::Expected, Some(_) => Identity::Other, None => Identity::Unknown, }) diff --git a/symposium-eval/src/expect/ownership.rs b/symposium-eval/src/expect/ownership.rs index 09e84a152..b5dc987a9 100644 --- a/symposium-eval/src/expect/ownership.rs +++ b/symposium-eval/src/expect/ownership.rs @@ -10,29 +10,34 @@ use std::fs; use std::io; -use std::path::{Component, Path, PathBuf}; +use std::path::{Component, Path}; -use super::installed::{GITIGNORE, GITIGNORE_CONTENTS, MARKER, SkillInstall, same_bytes}; -use crate::task::SkillName; +use super::installed::{GITIGNORE, GITIGNORE_CONTENTS, MARKER, copy_candidates, same_bytes}; +use crate::task::SkillSource; use crate::tree::{self, EntryKind, TreeEntry}; -/// The copies in `installs` that are wholly Symposium's, as `/`-joined paths relative to -/// `project`, each compared with its skill's source from `source_of`. -pub(crate) fn symposium_copies<'a>( +/// Discovers copies independently of assertions, comparing every candidate against +/// the task's trusted providers. Returns `/`-joined paths relative to `project`. +pub(crate) fn symposium_copies( project: &Path, - installs: impl IntoIterator, - source_of: impl Fn(&SkillName) -> Option, + skills: &Path, + sources: &[SkillSource], ) -> Vec { - installs - .into_iter() - .filter_map(|install| { - let copy = install.matching_copy.as_deref()?; - let source = source_of(&install.skill)?; - is_symposium_copy(copy, &source) - .then(|| relative(project, copy)) - .flatten() + let mut owned = sources + .iter() + .flat_map(|source| { + copy_candidates(skills, &source.name) + .into_iter() + .filter_map(|copy| { + is_symposium_copy(©, &source.dir) + .then(|| relative(project, ©)) + .flatten() + }) }) - .collect() + .collect::>(); + owned.sort(); + owned.dedup(); + owned } /// Whether `copy` is exactly what sync writes from `source`. One that cannot be read is @@ -106,11 +111,12 @@ fn relative(project: &Path, path: &Path) -> Option { #[cfg(test)] mod tests { + use std::path::PathBuf; + + use crate::grade::copy_exact; use tempfile::TempDir; use super::*; - use crate::expect::Verdict; - use crate::expect::installed::{InstallAssertion, InstallEvidence}; /// A skill source, and a project whose skill directory holds sync's copy of it. struct Installed { @@ -143,14 +149,15 @@ mod tests { } fn owned(&self) -> Vec { - let install = SkillInstall { - skill: SkillName::new("first-skill").unwrap(), - assertion: InstallAssertion::Installed, - evidence: InstallEvidence::Matching, - verdict: Verdict::Pass, - matching_copy: Some(self.copy.clone()), + let source = SkillSource { + name: "first-skill".to_owned(), + dir: self.source.clone(), }; - symposium_copies(&self.project, [&install], |_| Some(self.source.clone())) + symposium_copies( + &self.project, + &self.project.join(".claude/skills"), + &[source], + ) } } @@ -161,6 +168,41 @@ mod tests { assert_eq!(installed.owned(), [".claude/skills/first-skill"]); } + #[test] + fn a_hashed_copy_is_checked_against_each_provided_source() { + let installed = Installed::new(); + let other_source = installed.project.join("other-source"); + copy_exact(&installed.source, &other_source, "test source").unwrap(); + fs::write(other_source.join("notes/more.md"), "different source").unwrap(); + let hashed = installed.copy.with_file_name("first-skill-0123abcd"); + copy_exact(&installed.copy, &hashed, "test copy").unwrap(); + fs::write(hashed.join("notes/more.md"), "different source").unwrap(); + let sources = [ + SkillSource { + name: "first-skill".to_owned(), + dir: installed.source.clone(), + }, + SkillSource { + name: "first-skill".to_owned(), + dir: other_source, + }, + ]; + + let owned = symposium_copies( + &installed.project, + &installed.project.join(".claude/skills"), + &sources, + ); + + assert_eq!( + owned, + [ + ".claude/skills/first-skill", + ".claude/skills/first-skill-0123abcd" + ] + ); + } + #[test] fn a_copy_of_a_source_with_its_own_gitignore_is_symposiums() { let installed = Installed::new(); @@ -204,19 +246,13 @@ mod tests { } #[test] - fn a_copy_that_did_not_match_is_not_considered() { + fn a_copy_without_a_provided_source_is_not_considered() { let installed = Installed::new(); - let install = SkillInstall { - skill: SkillName::new("first-skill").unwrap(), - assertion: InstallAssertion::Installed, - evidence: InstallEvidence::Differing, - verdict: Verdict::Fail, - matching_copy: None, - }; - - let owned = symposium_copies(&installed.project, [&install], |_| { - Some(installed.source.clone()) - }); + let owned = symposium_copies( + &installed.project, + &installed.project.join(".claude/skills"), + &[], + ); assert!(owned.is_empty()); } diff --git a/symposium-eval/src/grade/error.rs b/symposium-eval/src/grade/error.rs index 8aa9d631d..18e7d346f 100644 --- a/symposium-eval/src/grade/error.rs +++ b/symposium-eval/src/grade/error.rs @@ -133,6 +133,9 @@ pub(crate) enum GradeError { }, #[error("`cargo metadata` printed something other than its documented format")] Metadata(#[source] serde_json::Error), + /// Grading's sandbox could not be prepared, so nothing ran. + #[error("cannot prepare grading's sandbox")] + Sandbox(#[source] Box), } /// Why a hidden check's listing cannot stand for exactly one test. diff --git a/symposium-eval/src/grade/mod.rs b/symposium-eval/src/grade/mod.rs index 6227c74cf..eb39e4ef3 100644 --- a/symposium-eval/src/grade/mod.rs +++ b/symposium-eval/src/grade/mod.rs @@ -245,7 +245,6 @@ impl<'a> Submitted<'a> { /// A trial's project, which started as the starter and [`PROJECT_SCAFFOLD`], holding /// `symposium_copies`. - #[cfg_attr(not(test), expect(dead_code, reason = "the trial driver comes next"))] pub(crate) fn trial(project: &'a Path, symposium_copies: &'a [String]) -> Self { Self { project, diff --git a/symposium-eval/src/outcome.rs b/symposium-eval/src/outcome.rs index 3d30e7003..709515e68 100644 --- a/symposium-eval/src/outcome.rs +++ b/symposium-eval/src/outcome.rs @@ -51,8 +51,11 @@ pub(crate) enum Failure { /// How far a trial got. #[derive(Debug, Clone, Copy)] pub(crate) enum Session<'a> { - /// Setup or the pre-launch check failed, so the client was never started. + /// Setup or the pre-launch check failed, or the client could not be launched, so + /// it never started. NotStarted, + /// The client started, but its filesystem could not safely be inspected. + Uncontained, /// The session ended, and its submission was graded. Graded { ended: &'a Ended, @@ -72,6 +75,9 @@ enum Code { /// Where a trial of a `kind` task ends, from how far it got and the verdicts of its /// expectations, which a baseline trial has none of. pub(crate) fn decide(kind: TaskKind, session: Session<'_>, expectations: &[Verdict]) -> Outcome { + if matches!(session, Session::Uncontained) { + return Outcome::Failed(Failure::Session); + } let Session::Graded { ended, grading } = session else { return Outcome::SetupFailure; }; @@ -181,7 +187,7 @@ mod tests { #[test] fn a_trial_that_never_launched_is_a_setup_failure() { - let not_launched = Ended::NotLaunched(process_error()); + let not_launched = Ended::NotLaunched(process_error().into()); assert_eq!( decide(Regression, Session::NotStarted, &[]), @@ -193,6 +199,16 @@ mod tests { ); } + #[test] + fn uncertain_containment_fails_the_session_without_grading() { + for kind in [Regression, Capability] { + assert_eq!( + decide(kind, Session::Uncontained, &[]), + Outcome::Failed(Failure::Session) + ); + } + } + #[test] fn broken_isolation_excludes_the_trial_whatever_it_produced() { let mismatched = Ended::Mismatched(ProvenanceError::Unstated); diff --git a/symposium-eval/src/result/mod.rs b/symposium-eval/src/result/mod.rs index 565472364..2a7ac78c9 100644 --- a/symposium-eval/src/result/mod.rs +++ b/symposium-eval/src/result/mod.rs @@ -73,7 +73,7 @@ pub(crate) enum Arm { } impl Arm { - fn configuration(&self) -> Configuration { + pub(crate) fn configuration(&self) -> Configuration { match self { Self::Baseline => Configuration::Baseline, Self::Symposium(_) => Configuration::Symposium, @@ -128,14 +128,20 @@ pub(crate) enum SetupStep { /// How far a trial got, with what each phase it reached left. #[derive(Debug, Clone, Copy)] pub(crate) enum Reached<'a> { + /// The client started but its writers may survive; no filesystem evidence is read. + Uncontained { execution: &'a Execution }, Setup { step: SetupStep, error: &'a dyn std::error::Error, }, + /// The client could not be launched, so nothing was graded: a setup failure, with + /// the attempt kept. + Session { + execution: &'a Execution, + changes: &'a [ConfigChange], + }, Graded { execution: &'a Execution, - /// The agent's time, setup and grading left out. - elapsed: Duration, grading: &'a Grading, changes: &'a [ConfigChange], /// A Symposium trial's, and only a Symposium trial's. @@ -153,6 +159,10 @@ pub(crate) enum ResultError { }, #[error("the session looked for {named} skills but showed {observed} loads")] Loads { named: usize, observed: usize }, + #[error("a client that {} was {}", if *.launched { "launched" } else { "never launched" }, if *.launched { "left ungraded" } else { "graded" })] + SessionPhase { launched: bool }, + #[error("a session with {} containment was recorded in the wrong phase", if *.certain { "proven" } else { "uncertain" })] + SessionContainment { certain: bool }, } #[derive(Debug, Serialize)] @@ -190,6 +200,7 @@ impl TrialResult { let task = planned.task; let configuration = planned.arm.configuration(); reached.check_expectations(configuration)?; + reached.check_session_phase()?; Ok(Self { schema_version: SCHEMA_VERSION, task: TaskRecord::new(task, planned.content_hash), diff --git a/symposium-eval/src/result/reached.rs b/symposium-eval/src/result/reached.rs index 35502620e..3562add1c 100644 --- a/symposium-eval/src/result/reached.rs +++ b/symposium-eval/src/result/reached.rs @@ -4,6 +4,7 @@ use serde::Serialize; use super::session::SessionRecord; use super::{Reached, ResultError, SetupStep, sanitized}; +use crate::client::{Ended, Execution}; use crate::expect::config::ConfigChange; use crate::expect::decide::Expectations; use crate::grade::Grading; @@ -15,8 +16,16 @@ use crate::task::Task; #[derive(Debug, Serialize)] #[serde(tag = "phase", rename_all = "snake_case")] pub(super) enum ReachedRecord { - Setup { error: String }, - Prelaunch { error: String }, + Setup { + error: String, + }, + Prelaunch { + error: String, + }, + /// The client could not be launched; `session` says why. + Launch, + /// The client may still be writing; code and expectations were not inspected. + UncontainedSession, Graded, } @@ -43,7 +52,10 @@ pub(super) enum FailureRecord { impl<'a> Reached<'a> { pub(super) fn outcome(self, task: &Task) -> Outcome { match self { - Self::Setup { .. } => outcome::decide(task.kind(), Session::NotStarted, &[]), + Self::Uncontained { .. } => outcome::decide(task.kind(), Session::Uncontained, &[]), + Self::Setup { .. } | Self::Session { .. } => { + outcome::decide(task.kind(), Session::NotStarted, &[]) + } Self::Graded { execution, grading, @@ -62,13 +74,40 @@ impl<'a> Reached<'a> { pub(super) fn session(self, task: &Task) -> Result, ResultError> { match self { - Self::Graded { - execution, elapsed, .. - } => SessionRecord::new(task, execution, elapsed).map(Some), + Self::Uncontained { execution } + | Self::Session { execution, .. } + | Self::Graded { execution, .. } => SessionRecord::new(task, execution).map(Some), Self::Setup { .. } => Ok(None), } } + /// Only a contained, launched session may be graded. Uncertain containment has + /// its own phase, which carries no filesystem evidence. + pub(super) fn check_session_phase(self) -> Result<(), ResultError> { + let (launched, graded) = match self { + Self::Uncontained { execution } => { + return if execution.ended.containment_is_certain() { + Err(ResultError::SessionContainment { certain: true }) + } else { + Ok(()) + }; + } + Self::Session { execution, .. } => (launched(execution), false), + Self::Graded { execution, .. } => { + if !execution.ended.containment_is_certain() { + return Err(ResultError::SessionContainment { certain: false }); + } + (launched(execution), true) + } + Self::Setup { .. } => return Ok(()), + }; + if launched == graded { + Ok(()) + } else { + Err(ResultError::SessionPhase { launched }) + } + } + /// A graded trial has expectations exactly when it is a Symposium trial. One that /// stopped before launch has none, whatever its configuration. pub(super) fn check_expectations( @@ -91,32 +130,38 @@ impl<'a> Reached<'a> { pub(super) fn grading(self) -> Option<&'a Grading> { match self { Self::Graded { grading, .. } => Some(grading), - Self::Setup { .. } => None, + Self::Setup { .. } | Self::Session { .. } | Self::Uncontained { .. } => None, } } pub(super) fn expectations(self) -> Option<&'a Expectations> { match self { Self::Graded { expectations, .. } => expectations, - Self::Setup { .. } => None, + Self::Setup { .. } | Self::Session { .. } | Self::Uncontained { .. } => None, } } pub(super) fn changes(self) -> &'a [ConfigChange] { match self { - Self::Graded { changes, .. } => changes, - Self::Setup { .. } => &[], + Self::Session { changes, .. } | Self::Graded { changes, .. } => changes, + Self::Setup { .. } | Self::Uncontained { .. } => &[], } } pub(super) fn reported_model(self) -> Option { match self { - Self::Graded { execution, .. } => execution.summary.model.clone(), + Self::Uncontained { execution } + | Self::Session { execution, .. } + | Self::Graded { execution, .. } => execution.summary.model.clone(), Self::Setup { .. } => None, } } } +fn launched(execution: &Execution) -> bool { + !matches!(execution.ended, Ended::NotLaunched(_)) +} + impl From> for ReachedRecord { fn from(reached: Reached<'_>) -> Self { match reached { @@ -132,6 +177,8 @@ impl From> for ReachedRecord { } => Self::Prelaunch { error: sanitized(error), }, + Reached::Session { .. } => Self::Launch, + Reached::Uncontained { .. } => Self::UncontainedSession, Reached::Graded { .. } => Self::Graded, } } diff --git a/symposium-eval/src/result/session.rs b/symposium-eval/src/result/session.rs index 9a54035ca..03f28ed11 100644 --- a/symposium-eval/src/result/session.rs +++ b/symposium-eval/src/result/session.rs @@ -1,7 +1,6 @@ //! What a result records of the client's session. use std::collections::BTreeMap; -use std::time::Duration; use serde::Serialize; @@ -68,14 +67,10 @@ impl SessionRecord { /// /// When it shows a different number of loads than skills, so none can be matched /// to its skill. - pub(super) fn new( - task: &Task, - execution: &Execution, - elapsed: Duration, - ) -> Result { + pub(super) fn new(task: &Task, execution: &Execution) -> Result { Ok(Self { ended: EndedRecord::from(&execution.ended), - elapsed_secs: elapsed.as_secs_f64(), + elapsed_secs: execution.elapsed.as_secs_f64(), usage: UsageRecord::from(execution.summary.usage), observed_loads: observed_loads(task, &execution.loads)?, }) diff --git a/symposium-eval/src/result/tests/mod.rs b/symposium-eval/src/result/tests/mod.rs index 7f22878b6..4b1fec543 100644 --- a/symposium-eval/src/result/tests/mod.rs +++ b/symposium-eval/src/result/tests/mod.rs @@ -44,6 +44,7 @@ pub(super) fn execution(ended: Ended, usage: Usage) -> Execution { Execution { transcript: "transcript.jsonl".into(), ended, + elapsed: Duration::from_millis(1500), loads: vec![Load::Loaded], hooks: HookEvidence::Unsupported, activity: ToolActivity::unread(), @@ -160,7 +161,6 @@ pub(super) fn graded<'a>( ) -> Reached<'a> { Reached::Graded { execution, - elapsed: Duration::from_millis(1500), grading, changes: &[], expectations, @@ -206,6 +206,30 @@ fn a_trial_stopped_before_launch() { assert_result_snapshot!(result_of(Configuration::Symposium, reached)); } +/// A session whose client could not be started, as `client::execute` leaves it. +pub(super) fn not_launched() -> Execution { + let error = crate::grade::ProcessError::Spawn { + program: "claude".into(), + source: io::Error::from(io::ErrorKind::NotFound), + }; + let mut execution = execution(Ended::NotLaunched(error.into()), Usage::Unknown); + execution.elapsed = Duration::ZERO; + execution.loads = vec![Load::Unknown]; + execution.summary = Summary::unread(Usage::Unknown); + execution +} + +#[test] +fn a_trial_whose_client_never_launched() { + let execution = not_launched(); + let reached = Reached::Session { + execution: &execution, + changes: &[], + }; + + assert_result_snapshot!(result_of(Configuration::Symposium, reached)); +} + #[test] fn a_baseline_trial_whose_usage_is_unknown() { let execution = execution(Ended::Completed, Usage::Unknown); diff --git a/symposium-eval/src/result/tests/refused.rs b/symposium-eval/src/result/tests/refused.rs index 2db9d48c6..0f4055ae4 100644 --- a/symposium-eval/src/result/tests/refused.rs +++ b/symposium-eval/src/result/tests/refused.rs @@ -2,7 +2,7 @@ use std::io; -use super::{execution, expectations, graded, passing, try_result_of}; +use super::{execution, expectations, graded, not_launched, passing, try_result_of}; use crate::client::{Ended, Load, Usage}; use crate::result::{Reached, ResultError, SetupStep}; use crate::run::Configuration; @@ -63,3 +63,59 @@ fn loads_must_match_the_skills_looked_for() { ); } } + +#[test] +fn only_a_client_that_never_launched_goes_ungraded() { + let (launched, unlaunched) = (execution(Ended::Completed, Usage::Unknown), not_launched()); + let grading = passing(); + let session = |execution| Reached::Session { + execution, + changes: &[], + }; + let result = |reached| try_result_of(Configuration::Baseline, reached); + + assert!(result(session(&unlaunched)).is_ok()); + assert!(result(graded(&launched, &grading, None)).is_ok()); + assert_eq!( + result(session(&launched)).unwrap_err(), + ResultError::SessionPhase { launched: true } + ); + assert_eq!( + result(graded(&unlaunched, &grading, None)).unwrap_err(), + ResultError::SessionPhase { launched: false } + ); +} + +#[test] +fn uncertain_containment_has_its_own_ungraded_phase() { + let grading = passing(); + for ended in [ + Ended::Unsupervised(crate::grade::ProcessError::OutputNotClosed), + Ended::TimedOut { + after_deadline: Some(crate::grade::ProcessError::OutputNotClosed), + }, + ] { + let execution = execution(ended, Usage::Unknown); + let reached = Reached::Uncontained { + execution: &execution, + }; + for configuration in [Configuration::Baseline, Configuration::Symposium] { + assert!(try_result_of(configuration, reached).is_ok()); + } + assert_eq!( + try_result_of(Configuration::Baseline, graded(&execution, &grading, None)).unwrap_err(), + ResultError::SessionContainment { certain: false } + ); + } + let contained = execution(Ended::Completed, Usage::Unknown); + assert_eq!( + try_result_of( + Configuration::Baseline, + Reached::Uncontained { + execution: &contained + } + ) + .unwrap_err(), + ResultError::SessionContainment { certain: true } + ); +} diff --git a/symposium-eval/src/result/tests/snapshots/symposium_eval__result__tests__a_trial_whose_client_never_launched.snap b/symposium-eval/src/result/tests/snapshots/symposium_eval__result__tests__a_trial_whose_client_never_launched.snap new file mode 100644 index 000000000..7fb62c717 --- /dev/null +++ b/symposium-eval/src/result/tests/snapshots/symposium_eval__result__tests__a_trial_whose_client_never_launched.snap @@ -0,0 +1,61 @@ +--- +source: symposium-eval/src/result/tests/mod.rs +expression: "result_of(Configuration::Symposium, reached)" +--- +{ + "schema_version": 1, + "task": { + "name": "formatting", + "kind": "regression", + "content_hash": "[hash]" + }, + "client": { + "name": "claude", + "version": "2.1.316" + }, + "model": { + "name": "haiku", + "settings": {}, + "reported": null + }, + "mode": "with-symposium", + "trial": 3, + "symposium": { + "version": "cargo-agents 1.0.0", + "sha256": "[hash]", + "revision": null + }, + "os": "linux x86_64", + "toolchain": "rustc 1.99.1", + "deadlines_secs": { + "setup": 120, + "agent": 600, + "grading": 300 + }, + "isolation": { + "mode": "enforced", + "srt_version": "0.0.81", + "config_changes": [] + }, + "reached": { + "phase": "launch" + }, + "outcome": { + "result": "setup_failure" + }, + "session": { + "ended": { + "status": "not_launched", + "error": "cannot start `claude`: entity not found" + }, + "elapsed_secs": 0.0, + "usage": { + "status": "unknown" + }, + "observed_loads": { + "rust-formatting": "unknown" + } + }, + "code": null, + "expectations": null +} diff --git a/symposium-eval/src/run/allocate.rs b/symposium-eval/src/run/allocate.rs new file mode 100644 index 000000000..a34602049 --- /dev/null +++ b/symposium-eval/src/run/allocate.rs @@ -0,0 +1,232 @@ +//! Making a trial in two steps: its root first, so a result always has a place to go, +//! then its contents. +//! +//! Only a [`Trial`] can be set up, launched or graded, and only [`NewTrial::prepare`] +//! makes one, so a trial whose preparation failed can never run. It keeps its root, so +//! the runner can still record why it stopped. + +use std::path::{Path, PathBuf}; + +use super::{Batch, Configuration, Inputs, RunError, Trial, create_dirs, create_fresh}; +use crate::grade::{PROJECT_SCAFFOLD, copy_tree}; + +/// A trial whose root exists and holds nothing yet, with the inputs it will be filled +/// from. +#[derive(Debug)] +#[must_use = "an allocated trial's root stays on disk; prepare it or record why it stopped"] +pub(crate) struct NewTrial<'a> { + trial: Trial, + inputs: &'a Inputs, +} + +/// A trial whose preparation failed, with the root it keeps for its result. +#[derive(Debug)] +pub(crate) struct NotPrepared { + root: PathBuf, + error: RunError, +} + +impl Batch { + /// Creates trial `number`'s root, empty, to be filled from `inputs`, this batch's + /// copy of the task. + /// + /// # Errors + /// + /// When `inputs` are another batch's, which never gets a root, or the trial already + /// exists, so a restarted run never inherits an earlier attempt, or its root cannot + /// be created. + pub(crate) fn allocate_trial<'a>( + &self, + number: usize, + configuration: Configuration, + inputs: &'a Inputs, + ) -> Result, RunError> { + if inputs.batch() != self.root { + return Err(RunError::ForeignInputs(inputs.batch().to_owned())); + } + let trial = Trial { + batch: self.clone(), + root: self.root.join(format!("trial-{number}-{configuration}")), + configuration, + }; + create_fresh(&trial.root)?; + Ok(NewTrial { trial, inputs }) + } + + /// Allocates trial `number` and prepares it from `inputs`. + /// + /// # Errors + /// + /// As [`Self::allocate_trial`] and [`NewTrial::prepare`]. + pub(crate) fn create_trial( + &self, + number: usize, + configuration: Configuration, + inputs: &Inputs, + ) -> Result { + self.allocate_trial(number, configuration, inputs)? + .prepare() + .map_err(NotPrepared::into_error) + } +} + +impl NewTrial<'_> { + pub(crate) fn root(&self) -> &Path { + self.trial.root() + } + + /// Fills the trial: empty working directories, the starter in `project/`, the + /// fixtures beside it, and, for Symposium, the task's plugins as the trial's + /// registry. + /// + /// # Errors + /// + /// When a directory cannot be created or the task cannot be copied. The root is + /// kept. + pub(crate) fn prepare(self) -> Result { + match self.fill() { + Ok(()) => Ok(self.trial), + Err(error) => Err(NotPrepared { + root: self.trial.root, + error, + }), + } + } + + fn fill(&self) -> Result<(), RunError> { + let trial = &self.trial; + let (evals, task) = (self.inputs.evals(), self.inputs.task()); + create_dirs([ + trial.home(), + trial.bin(), + trial.cargo_home(), + trial.tmp(), + trial.evidence(), + trial.session(), + ])?; + trial.batch.link_tools(trial)?; + copy_tree(&evals.starter_dir(task.name()), &trial.project(), "starter") + .map_err(RunError::Copy)?; + create_dirs(PROJECT_SCAFFOLD.map(|dir| trial.project().join(dir)))?; + // Fixtures are not in the policy's writable paths, so `srt` keeps them read-only. + for fixture in task.fixtures() { + let destination = trial.fixtures().join(fixture.as_str()); + copy_tree(&evals.fixture_dir(fixture), &destination, "fixture") + .map_err(RunError::Copy)?; + } + if trial.configuration == Configuration::Symposium { + create_dirs([trial.registry()])?; + for plugin in task.plugins() { + let destination = trial.registry().join(plugin.as_str()); + copy_tree(&evals.plugin_dir(plugin), &destination, "plugin") + .map_err(RunError::Copy)?; + } + } + Ok(()) + } +} + +impl NotPrepared { + /// The root the trial was allocated, where its result goes. + pub(crate) fn root(&self) -> &Path { + &self.root + } + + pub(crate) fn error(&self) -> &RunError { + &self.error + } + + pub(crate) fn into_error(self) -> RunError { + self.error + } +} + +#[cfg(test)] +mod tests { + use std::fs; + use std::io; + + use tempfile::TempDir; + + use super::*; + use crate::run::test_support::store_durability; + use crate::task::BatchName; + + fn batch(root: &TempDir) -> Batch { + Batch::create(root.path(), &BatchName::new("batch").unwrap()).unwrap() + } + + #[test] + fn allocating_a_trial_creates_only_its_empty_root() { + let root = TempDir::new().unwrap(); + let batch = batch(&root); + let inputs = store_durability(&batch); + + let new = batch + .allocate_trial(1, Configuration::Symposium, &inputs) + .unwrap(); + + assert_eq!( + new.root(), + batch.root.join("trial-1-with-symposium").as_path() + ); + assert_eq!(fs::read_dir(new.root()).unwrap().count(), 0); + } + + #[test] + fn an_allocated_root_is_never_allocated_again() { + let root = TempDir::new().unwrap(); + let batch = batch(&root); + let inputs = store_durability(&batch); + let _first = batch + .allocate_trial(1, Configuration::Baseline, &inputs) + .unwrap(); + + let again = batch.allocate_trial(1, Configuration::Baseline, &inputs); + + assert!( + matches!(&again, Err(RunError::Create { source, .. }) if source.kind() == io::ErrorKind::AlreadyExists), + "{again:?}" + ); + } + + #[test] + fn a_failed_preparation_keeps_its_root() { + let root = TempDir::new().unwrap(); + let batch = batch(&root); + let inputs = store_durability(&batch); + // The batch's own copy of the task, broken after it was taken. + fs::remove_dir_all(inputs.evals().starter_dir(inputs.task().name())).unwrap(); + let new = batch + .allocate_trial(1, Configuration::Symposium, &inputs) + .unwrap(); + let allocated = new.root().to_owned(); + + let failed = new.prepare().unwrap_err(); + + assert_eq!(failed.root(), allocated); + assert!( + matches!(failed.error(), RunError::Copy(_)), + "{:?}", + failed.error() + ); + assert!(allocated.is_dir()); + } + + #[test] + fn preparing_fills_the_allocated_root() { + let root = TempDir::new().unwrap(); + let batch = batch(&root); + let inputs = store_durability(&batch); + let new = batch + .allocate_trial(1, Configuration::Baseline, &inputs) + .unwrap(); + let allocated = new.root().to_owned(); + + let trial = new.prepare().unwrap(); + + assert_eq!(trial.root(), allocated); + assert!(trial.project().join("Cargo.lock").is_file()); + assert!(trial.project().join(".claude").is_dir()); + } +} diff --git a/symposium-eval/src/run/mod.rs b/symposium-eval/src/run/mod.rs index 29ad62b4f..6cdcfc480 100644 --- a/symposium-eval/src/run/mod.rs +++ b/symposium-eval/src/run/mod.rs @@ -22,6 +22,7 @@ #[cfg(all(test, target_os = "linux"))] mod adversarial; +mod allocate; pub(crate) mod client; mod env; mod error; @@ -37,6 +38,7 @@ mod srt; pub(crate) mod stage; #[cfg(test)] pub(crate) mod test_support; +pub(crate) mod trial; use std::fmt; use std::fs; @@ -49,9 +51,7 @@ pub(crate) use policy::{Machine, Policy, Reach}; pub(crate) use srt::SRT_VERSION; use stage::StagedTool; -use crate::grade::PROJECT_SCAFFOLD; use crate::grade::ProcessError; -use crate::grade::copy_tree; use crate::grade::process::Finished; use crate::task::BatchName; @@ -152,59 +152,6 @@ impl Batch { pub(crate) fn symposium_tools(&self) -> PathBuf { self.root.join("tools").join("symposium") } - - /// Creates trial `number` of the task in `inputs`, this batch's copy: empty working - /// directories, the starter in `project/`, the fixtures beside it, and, for - /// Symposium, the task's plugins as the trial's registry. - /// - /// # Errors - /// - /// When `inputs` are another batch's, the trial already exists, a directory cannot - /// be created, or the task cannot be copied. - pub(crate) fn create_trial( - &self, - number: usize, - configuration: Configuration, - inputs: &Inputs, - ) -> Result { - if inputs.batch() != self.root { - return Err(RunError::ForeignInputs(inputs.batch().to_owned())); - } - let (evals, task) = (inputs.evals(), inputs.task()); - let trial = Trial { - batch: self.clone(), - root: self.root.join(format!("trial-{number}-{configuration}")), - configuration, - }; - create_fresh(&trial.root)?; - create_dirs([ - trial.home(), - trial.bin(), - trial.cargo_home(), - trial.tmp(), - trial.evidence(), - trial.session(), - ])?; - self.link_tools(&trial)?; - copy_tree(&evals.starter_dir(task.name()), &trial.project(), "starter") - .map_err(RunError::Copy)?; - create_dirs(PROJECT_SCAFFOLD.map(|dir| trial.project().join(dir)))?; - // Fixtures are not in the policy's writable paths, so `srt` keeps them read-only. - for fixture in task.fixtures() { - let destination = trial.fixtures().join(fixture.as_str()); - copy_tree(&evals.fixture_dir(fixture), &destination, "fixture") - .map_err(RunError::Copy)?; - } - if configuration == Configuration::Symposium { - create_dirs([trial.registry()])?; - for plugin in task.plugins() { - let destination = trial.registry().join(plugin.as_str()); - copy_tree(&evals.plugin_dir(plugin), &destination, "plugin") - .map_err(RunError::Copy)?; - } - } - Ok(trial) - } } impl Trial { diff --git a/symposium-eval/src/run/test_support.rs b/symposium-eval/src/run/test_support.rs index f52c64d7e..7fd6bb59f 100644 --- a/symposium-eval/src/run/test_support.rs +++ b/symposium-eval/src/run/test_support.rs @@ -158,9 +158,30 @@ impl RealSrt { tool: &super::stage::Tool, audience: super::stage::Audience, ) -> PathBuf { + self.stage_tool(name, tool, audience).command().to_owned() + } + + /// As [`Self::stage`], returning what was staged. + pub(crate) fn stage_tool( + &mut self, + name: &str, + tool: &super::stage::Tool, + audience: super::stage::Audience, + ) -> super::stage::StagedTool { let name = crate::task::ToolName::new(name).unwrap(); - let staged = self.batch.stage(name, tool, audience).unwrap(); - staged.command().to_owned() + self.batch.stage(name, tool, audience).unwrap().clone() + } + + pub(crate) fn srt(&self) -> &super::srt::Srt { + &self.srt + } + + pub(crate) fn machine(&self) -> &Machine { + &self.machine + } + + pub(crate) fn api_key(&self) -> &ApiKey { + &self.api_key } pub(crate) fn decoy_home(&self) -> &std::path::Path { diff --git a/symposium-eval/src/run/trial/claude-stand-in.js b/symposium-eval/src/run/trial/claude-stand-in.js new file mode 100644 index 000000000..9a0573cb3 --- /dev/null +++ b/symposium-eval/src/run/trial/claude-stand-in.js @@ -0,0 +1,96 @@ +#!/usr/bin/env node +// A stand-in for Claude Code in print mode, for tests that run a whole trial without a +// model. It speaks the `stream-json` events the adapter reads, runs the hooks that +// `~/.claude/settings.json` registers as Claude Code would, reads each installed skill, +// and leaves `double` in `src/lib.rs` unformatted, so only a formatter makes the +// formatting task pass. +'use strict'; +const childProcess = require('child_process'); +const crypto = require('crypto'); +const fs = require('fs'); +const path = require('path'); + +const VERSION = '2.1.316'; +const args = process.argv.slice(2); +if (args[0] === '--version') { + console.log(`${VERSION} (Claude Code)`); + process.exit(0); +} +const option = (name) => args[args.indexOf(name) + 1]; +const session = option('--session-id'); +const model = option('--model'); + +const emit = (event) => process.stdout.write(`${JSON.stringify({ ...event, session_id: session })}\n`); + +function registeredHooks(event) { + const settings = path.join(process.env.HOME, '.claude', 'settings.json'); + if (!fs.existsSync(settings)) { + return []; + } + const groups = JSON.parse(fs.readFileSync(settings, 'utf8')).hooks?.[event] ?? []; + return groups.flatMap((group) => group.hooks.map((hook) => hook.command)); +} + +function runHooks(event, name, fields) { + for (const command of registeredHooks(event)) { + const hook = { hook_id: crypto.randomUUID(), hook_name: name, hook_event: event }; + emit({ type: 'system', subtype: 'hook_started', ...hook }); + const input = JSON.stringify({ + session_id: session, + transcript_path: path.join(process.env.HOME, 'transcript.jsonl'), + cwd: process.cwd(), + hook_event_name: event, + ...fields, + }); + const ran = childProcess.spawnSync('sh', ['-c', command], { input, encoding: 'utf8' }); + emit({ + type: 'system', + subtype: 'hook_response', + ...hook, + output: ran.stdout, + stdout: ran.stdout, + stderr: ran.stderr, + exit_code: ran.status, + outcome: ran.status === 0 ? 'success' : 'error', + }); + } +} + +runHooks('SessionStart', 'SessionStart:startup', { source: 'startup' }); +emit({ + type: 'system', + subtype: 'init', + cwd: process.cwd(), + tools: [], + model, + permissionMode: 'bypassPermissions', + claude_code_version: VERSION, + skills: [], +}); +// Read each installed skill, as the agent does when the prompt names one. +const skills = path.join('.claude', 'skills'); +for (const skill of fs.existsSync(skills) ? fs.readdirSync(skills) : []) { + const id = `toolu_${crypto.randomUUID()}`; + const file_path = path.join(skills, skill, 'SKILL.md'); + emit({ + type: 'assistant', + message: { role: 'assistant', content: [{ type: 'tool_use', id, name: 'Read', input: { file_path } }] }, + parent_tool_use_id: null, + }); + const content = fs.readFileSync(file_path, 'utf8'); + emit({ + type: 'user', + message: { role: 'user', content: [{ tool_use_id: id, type: 'tool_result', content }] }, + parent_tool_use_id: null, + }); +} +fs.writeFileSync('src/lib.rs', '//! Small arithmetic helpers.\n\npub fn double(x:i32)->i32{x*2}\n'); +runHooks('Stop', 'Stop', { stop_hook_active: false }); +const tokens = { inputTokens: 1, outputTokens: 1, cacheReadInputTokens: 0, cacheCreationInputTokens: 0 }; +emit({ + type: 'result', + subtype: 'success', + is_error: false, + result: 'done', + modelUsage: { [model]: tokens }, +}); diff --git a/symposium-eval/src/run/trial/mod.rs b/symposium-eval/src/run/trial/mod.rs new file mode 100644 index 000000000..4a05be3e7 --- /dev/null +++ b/symposium-eval/src/run/trial/mod.rs @@ -0,0 +1,153 @@ +//! Running one trial end to end (RFD reference, "Running a trial"). +//! +//! The trial's root is allocated first, so whatever happens next has a place for its +//! result. Then each phase runs in turn: preparing the trial, setting up Symposium, the +//! checks before launch, the session, gathering what Symposium left, grading, and +//! deciding expectations. A phase that fails stops the trial, and its result says where. +//! Only a root that cannot be allocated, parts that cannot make one result, or a result +//! that cannot be written are the runner's own errors. +//! +//! Symposium's evidence is gathered before grading, which needs it to tell the skill +//! copies sync installed from the agent's own edits. + +mod phases; +#[cfg(all(test, target_os = "linux"))] +mod real; +mod sandbox; +#[cfg(test)] +mod tests; + +use std::io; +use std::path::PathBuf; + +use thiserror::Error; + +pub(crate) use sandbox::Sandbox; + +use super::{ApiKey, Batch, Inputs, Machine, RunError}; +use crate::client::{ClientRunner, Execution}; +use crate::expect::config::ConfigChange; +use crate::expect::decide::Expectations; +use crate::grade::Grading; +use crate::result::{Arm, Planned, Provenance, Reached, ResultError, SetupStep, TrialResult}; + +/// Where a trial's result is written, in its root. +pub(crate) const RESULT_FILE: &str = "result.json"; + +/// What every trial of a batch shares. +#[derive(Debug)] +pub(crate) struct Driver<'a, S> { + pub(crate) batch: &'a Batch, + pub(crate) inputs: &'a Inputs, + pub(crate) machine: &'a Machine, + pub(crate) sandbox: &'a S, + /// The only host a trial's network reaches. + pub(crate) api_host: &'a str, + pub(crate) api_key: &'a ApiKey, + pub(crate) provenance: &'a Provenance, +} + +/// Why a trial has no result. +#[derive(Debug, Error)] +pub(crate) enum TrialError { + #[error("cannot allocate the trial")] + Allocate(#[source] RunError), + #[error("the trial's parts cannot make one result")] + Result(#[source] ResultError), + #[error("cannot write `{}`", .path.display())] + Write { + path: PathBuf, + #[source] + source: io::Error, + }, +} + +/// How far a trial got, owning what each phase left. +#[derive(Debug)] +#[expect(clippy::large_enum_variant, reason = "one per trial, never collected")] +enum Ran { + Setup(Stopped), + /// The client may still be writing; no trial paths have been inspected. + Uncontained { + execution: Execution, + }, + Session { + execution: Execution, + changes: Vec, + }, + Graded { + execution: Execution, + grading: Grading, + changes: Vec, + expectations: Option, + }, +} + +/// A trial stopped before its client launched. +#[derive(Debug)] +struct Stopped { + step: SetupStep, + error: RunError, +} + +impl Driver<'_, S> { + /// Runs trial `number` of `arm` with `runner`'s client, and writes its result. + /// Returns where the result is. + /// + /// # Errors + /// + /// When the trial cannot be allocated, its parts cannot make one result, or the + /// result cannot be written. Anything else that goes wrong is in the result. + pub(crate) fn run( + &self, + number: usize, + arm: Arm, + runner: &impl ClientRunner, + ) -> Result { + let new = self + .batch + .allocate_trial(number, arm.configuration(), self.inputs) + .map_err(TrialError::Allocate)?; + let path = new.root().join(RESULT_FILE); + let planned = Planned { + task: self.inputs.task(), + content_hash: self.inputs.digest(), + client: runner.client(), + model: runner.model(), + arm, + trial: number, + }; + let ran = self.ran(new, runner); + let result = TrialResult::new(planned, self.provenance.clone(), ran.reached()) + .map_err(TrialError::Result)?; + result.write(&path).map_err(|source| TrialError::Write { + path: path.clone(), + source, + })?; + Ok(path) + } +} + +impl Ran { + fn reached(&self) -> Reached<'_> { + match self { + Self::Setup(stopped) => Reached::Setup { + step: stopped.step, + error: &stopped.error, + }, + Self::Uncontained { execution } => Reached::Uncontained { execution }, + Self::Session { execution, changes } => Reached::Session { execution, changes }, + Self::Graded { + execution, + grading, + changes, + expectations, + } => Reached::Graded { + execution, + grading, + changes, + expectations: expectations.as_ref(), + }, + } + } +} diff --git a/symposium-eval/src/run/trial/phases.rs b/symposium-eval/src/run/trial/phases.rs new file mode 100644 index 000000000..3a9c6743d --- /dev/null +++ b/symposium-eval/src/run/trial/phases.rs @@ -0,0 +1,183 @@ +//! Each phase of a trial, from a new root to graded code and decided expectations. + +use std::ffi::OsString; +use std::path::{Path, PathBuf}; + +use super::{Driver, Ran, Sandbox, Stopped}; +use crate::client::{ClientRunner, Ended, Execution, Load, execute}; +use crate::expect::config::{ConfigChange, Snapshot}; +use crate::expect::decide::{self, Outside, SymposiumTrial}; +use crate::expect::{SkillUse, ownership, skill_use}; +use crate::grade::{self, GradeError, GradeLimits, Grading, Submitted}; +use crate::result::SetupStep; +use crate::run::allocate::NewTrial; +use crate::run::client::Client; +use crate::run::{Configuration, Policy, RunError, Trial}; +use crate::task::SkillName; + +/// A trial ready to launch. +struct Launch { + backend: B, + env: Vec<(OsString, OsString)>, + /// The client's and Symposium's config as setup left it. + snapshot: Snapshot, +} + +impl Stopped { + fn at(step: SetupStep) -> impl Fn(RunError) -> Self + Copy { + move |error| Self { step, error } + } +} + +impl Driver<'_, S> { + /// Runs each phase in turn, until one stops the trial or its code is graded. + pub(super) fn ran(&self, new: NewTrial<'_>, runner: &impl ClientRunner) -> Ran { + let client = runner.client(); + let trial = match new.prepare() { + Ok(trial) => trial, + Err(failed) => { + return Ran::Setup(Stopped { + step: SetupStep::Setup, + error: failed.into_error(), + }); + } + }; + let launch = match self.ready(&trial, client) { + Ok(launch) => launch, + Err(stopped) => return Ran::Setup(stopped), + }; + let task = self.inputs.task(); + let execution = execute( + runner, + &launch.backend, + &trial, + launch.env, + task.prompt(), + &decide::installed_skills(&trial, client, task), + self.provenance.deadlines.agent, + ); + // Path-based readers require all writers to have stopped. In particular, even + // comparing configuration must wait for proven containment. + if !execution.ended.containment_is_certain() { + return Ran::Uncontained { execution }; + } + let changes = launch.snapshot.changes(); + if matches!(execution.ended, Ended::NotLaunched(_)) { + return Ran::Session { execution, changes }; + } + self.graded(&trial, client, execution, changes) + } + + /// Writes the trial's policy, sets up Symposium for a Symposium trial, and runs the + /// checks before launch. + fn ready(&self, trial: &Trial, client: Client) -> Result>, Stopped> { + let setup = Stopped::at(SetupStep::Setup); + self.write_policy(trial).map_err(setup)?; + let backend = self.sandbox.for_trial(trial).map_err(setup)?; + let env = trial + .environment(self.machine, self.api_key) + .map_err(setup)?; + if trial.configuration() == Configuration::Symposium { + let deadline = self.provenance.deadlines.setup; + trial + .set_up_symposium(client, &backend, env.clone(), deadline) + .map_err(setup)?; + } + let snapshot = Snapshot::take(&trial.home(), &trial.project(), client); + trial + .check_before_launch(client, &env) + .map_err(Stopped::at(SetupStep::Prelaunch))?; + Ok(Launch { + backend, + env, + snapshot, + }) + } + + fn write_policy(&self, trial: &Trial) -> Result<(), RunError> { + let policy = Policy::for_trial(trial, self.machine, self.api_host, self.api_key)?; + trial.write_policy(&policy) + } + + /// Grades what the session left, after reading what Symposium did. + fn graded( + &self, + trial: &Trial, + client: Client, + execution: Execution, + changes: Vec, + ) -> Ran { + let task = self.inputs.task(); + let symposium = SymposiumTrial::new(trial); + let facts = symposium.map(|symposium| { + decide::gather(symposium, client, task, |skill| self.source_of(skill)) + }); + let copies = symposium.map_or_else(Vec::new, |_| { + ownership::symposium_copies( + &trial.project(), + &trial.project().join(client.skill_dir()), + task.provided_skills(), + ) + }); + let grading = self.grade(trial, &copies); + let expectations = symposium.zip(facts).map(|(symposium, facts)| { + let outside = Outside { + hooks: &execution.hooks, + activity: &execution.activity, + changes: &changes, + }; + decide::decide(symposium, client, facts, outside, self.uses(&execution)) + }); + Ran::Graded { + execution, + grading, + changes, + expectations, + } + } + + /// Grades `trial`'s project, leaving out `copies`, Symposium's own skill copies. + fn grade(&self, trial: &Trial, copies: &[String]) -> Grading { + let prepared = trial + .prepare_grading(self.machine) + .and_then(|dirs| Ok((dirs, self.sandbox.for_grading(trial)?))); + let (dirs, backend) = match prepared { + Ok(prepared) => prepared, + Err(error) => { + return Grading::Infrastructure { + error: GradeError::Sandbox(Box::new(error)), + partial: None, + }; + } + }; + let limits = GradeLimits { + commands: self.provenance.deadlines.grading, + ..GradeLimits::default() + }; + grade::grade( + self.inputs.evals(), + self.inputs.task(), + Submitted::trial(&trial.project(), copies), + &backend, + &limits, + &dirs, + ) + } + + /// The `used` and `not_used` verdicts, from the loads the session showed for each + /// skill, in the order it looked for them. + fn uses(&self, execution: &Execution) -> Vec { + let expected = self.inputs.task().expected_skills(); + skill_use(expected, |skill| { + expected + .names() + .position(|name| name == skill) + .and_then(|index| execution.loads.get(index).copied()) + .unwrap_or(Load::Unknown) + }) + } + + fn source_of(&self, skill: &SkillName) -> Option { + self.inputs.task().skill_source(skill).map(Path::to_owned) + } +} diff --git a/symposium-eval/src/run/trial/real.rs b/symposium-eval/src/run/trial/real.rs new file mode 100644 index 000000000..0d40478df --- /dev/null +++ b/symposium-eval/src/run/trial/real.rs @@ -0,0 +1,134 @@ +//! Whole trials under real `srt`, with the `cargo-agents` built from this checkout and a +//! stand-in for Claude Code, run through the real adapter: run on Linux with +//! `cargo test -p symposium-eval run::trial::real -- --ignored`. +//! +//! The stand-in leaves `double` unformatted, so a trial passes only if Symposium's +//! format-on-stop hook ran: the Symposium trial succeeds, and the baseline fails on +//! formatting. + +use std::fs; +use std::time::Duration; + +use serde_json::{Value, json}; +use tempfile::TempDir; + +use super::Driver; +use crate::client::claude::Claude; +use crate::client::claude::version; +use crate::result::{Arm, Deadlines, IsolationMode, Provenance, SymposiumBuild}; +use crate::run::stage::{Audience, Tool}; +use crate::run::test_support::{RealSrt, api_key, built_cargo_agents, executable, inputs, on_path}; +use crate::run::{Configuration, SRT_VERSION}; + +const STAND_IN: &str = include_str!("claude-stand-in.js"); + +#[test] +#[ignore = "needs node, srt, and bubblewrap installed, and builds cargo-agents"] +fn symposium_stop_hook_formats_the_submission_and_the_trial_passes() { + let scenario = FormattingScenario::new(); + + let result = scenario.run(Configuration::Symposium); + + assert_eq!(result["code"]["cargo_fmt_check"], "pass", "{result:#}"); + assert_eq!(result["outcome"]["result"], "succeeded", "{result:#}"); + assert_eq!( + result["code"]["edits_outside_editable"], + json!([]), + "{result:#}" + ); +} + +#[test] +#[ignore = "needs node, srt, and bubblewrap installed, and builds cargo-agents"] +fn baseline_trial_completes_but_fails_the_formatting_check() { + let scenario = FormattingScenario::new(); + + let result = scenario.run(Configuration::Baseline); + + assert_eq!( + result["session"]["ended"]["status"], "completed", + "{result:#}" + ); + assert_eq!(result["code"]["cargo_fmt_check"], "fail", "{result:#}"); + assert_eq!(result["outcome"]["result"], "failed", "{result:#}"); + assert_eq!(result["outcome"]["failure"], "code", "{result:#}"); +} + +/// A fresh batch with a real sandbox and a deterministic Claude stand-in. The stand-in +/// always leaves the submission unformatted, so only Symposium's hook can format it. +struct FormattingScenario { + real: RealSrt, + client_version: version::Version, + symposium_build: SymposiumBuild, +} + +impl FormattingScenario { + fn new() -> Self { + let sources = TempDir::new().expect("create tool staging directory"); + let mut real = RealSrt::new(api_key()); + real.stage("node", &Tool::Executable(on_path("node")), Audience::Common); + let package = sources.path().join("claude"); + fs::create_dir(&package).expect("create Claude stand-in package"); + executable(&package.join("cli.js"), STAND_IN); + let tool = Tool::NodePackage { + package, + entry: "cli.js".into(), + }; + let claude = real.stage_tool("claude", &tool, Audience::Common); + let cargo_agents = built_cargo_agents(); + real.stage( + "cargo-agents", + &Tool::Executable(cargo_agents.clone()), + Audience::Symposium, + ); + let client_version = version::preflight(real.batch(), &claude, sources.path()) + .expect("probe the staged Claude stand-in version"); + let symposium_build = SymposiumBuild { + version: "cargo-agents (this checkout)".to_owned(), + sha256: crate::digest::file(&cargo_agents).expect("hash cargo-agents"), + revision: None, + }; + Self { + real, + client_version, + symposium_build, + } + } + + fn run(self, configuration: Configuration) -> Value { + let inputs = inputs(self.real.batch(), "formatting"); + let provenance = provenance(self.client_version); + let driver = Driver { + batch: self.real.batch(), + inputs: &inputs, + machine: self.real.machine(), + sandbox: self.real.srt(), + api_host: "api.example.com", + api_key: self.real.api_key(), + provenance: &provenance, + }; + let arm = match configuration { + Configuration::Symposium => Arm::Symposium(self.symposium_build), + Configuration::Baseline => Arm::Baseline, + }; + let runner = Claude::enforced("a-model".to_owned(), self.client_version); + let path = driver.run(1, arm, &runner).expect("run formatting trial"); + let bytes = fs::read(path).expect("read the trial's result.json"); + serde_json::from_slice(&bytes).expect("parse the trial's result.json") + } +} + +fn provenance(client_version: version::Version) -> Provenance { + Provenance { + client_version: Some(client_version.to_string()), + os: std::env::consts::OS.to_owned(), + toolchain: None, + isolation: IsolationMode::Enforced, + srt_version: Some(SRT_VERSION.to_owned()), + deadlines: Deadlines { + setup: Duration::from_secs(120), + agent: Duration::from_secs(300), + grading: Duration::from_secs(600), + }, + } +} diff --git a/symposium-eval/src/run/trial/sandbox.rs b/symposium-eval/src/run/trial/sandbox.rs new file mode 100644 index 000000000..3f69299e9 --- /dev/null +++ b/symposium-eval/src/run/trial/sandbox.rs @@ -0,0 +1,30 @@ +//! Where a trial's commands run. Every run confines them with `srt`; only tests supply +//! another sandbox, so nothing outside a test can run a trial unconfined. + +use crate::grade::process::Backend; +use crate::run::srt::{Sandboxed, Srt}; +use crate::run::{RunError, Trial}; + +pub(crate) trait Sandbox { + type Backend<'a>: Backend + where + Self: 'a; + + /// Runs `trial`'s own commands, its client's included, under its policy. + fn for_trial(&self, trial: &Trial) -> Result, RunError>; + + /// Runs grading of `trial`'s submission under its grading policy. + fn for_grading(&self, trial: &Trial) -> Result, RunError>; +} + +impl Sandbox for Srt { + type Backend<'a> = Sandboxed<'a>; + + fn for_trial(&self, trial: &Trial) -> Result, RunError> { + Srt::for_trial(self, trial) + } + + fn for_grading(&self, trial: &Trial) -> Result, RunError> { + Srt::for_grading(self, trial) + } +} diff --git a/symposium-eval/src/run/trial/tests.rs b/symposium-eval/src/run/trial/tests.rs new file mode 100644 index 000000000..b28489bab --- /dev/null +++ b/symposium-eval/src/run/trial/tests.rs @@ -0,0 +1,631 @@ +//! Each way a trial can end, through the driver, with a scripted sandbox: `init` and the +//! client are played, and grading runs for real, unconfined. + +mod ownership; + +use std::cell::Cell; +use std::ffi::OsString; +use std::fs; +use std::io; +use std::path::{Path, PathBuf}; +use std::process::ExitStatus; +use std::time::Duration; + +use serde_json::Value; + +use super::*; +use crate::client::{ + Completion, HookEvidence, InstalledSkill, Load, Reading, SessionIntroduction, Summary, + ToolActivity, Usage, +}; +use crate::grade::ProcessError; +use crate::grade::process::{Backend, Finished, Invocation, Local}; +use crate::result::{Deadlines, IsolationMode, SymposiumBuild}; +use crate::run::client::Client; +use crate::run::test_support::{Host, api_key, inputs}; +use crate::run::{Configuration, Trial}; + +/// How the played `cargo-agents init` ends. +#[derive(Clone, Copy)] +enum Init { + Succeeds, + Fails, + /// Succeeds, but leaves a sync record, as a sync would. + Syncs, +} + +/// What the played client does. +#[derive(Clone, Copy)] +enum Session { + /// Writes the solution and a finished transcript. + Solves, + /// Solves after SessionStart installs a skill from the trial's trusted registry. + InstallsSkill, + /// Changes a skill's payload after sync installed it. + ChangesSkillContents, + /// Changes sync's metadata after installing a skill. + ChangesSkillMarker, + /// Writes the solution, then runs past its deadline. + TimesOut, + CannotStart, + /// The sandbox starts, but the client exits before introducing a session. + WrapperExits(i32), + /// Starts, but the runner loses supervision of its process tree. + Unsupervised, + /// Times out, but a descendant keeps an output pipe open. + UncleanTimeout, + /// Solves, and writes a `result.json` in the trial's root. + PlantsResult, +} + +struct Scripted { + init: Init, + session: Session, + /// Whether the trial's sandbox can be prepared. + trial_sandbox: bool, + /// Whether grading's sandbox can be prepared. + grading_sandbox: bool, + grading_calls: Cell, +} + +impl Scripted { + fn new(session: Session) -> Self { + Self { + init: Init::Succeeds, + session, + trial_sandbox: true, + grading_sandbox: true, + grading_calls: Cell::new(0), + } + } +} + +enum Played<'a> { + Trial { script: &'a Scripted, trial: Trial }, + Grading, +} + +impl Sandbox for Scripted { + type Backend<'a> = Played<'a>; + + fn for_trial(&self, trial: &Trial) -> Result, RunError> { + if !self.trial_sandbox { + return Err(RunError::NoPolicy(trial.policy_file())); + } + Ok(Played::Trial { + script: self, + trial: trial.clone(), + }) + } + + fn for_grading(&self, trial: &Trial) -> Result, RunError> { + self.grading_calls.set(self.grading_calls.get() + 1); + if self.grading_sandbox { + Ok(Played::Grading) + } else { + Err(RunError::NoPolicy(trial.grading_policy_file())) + } + } +} + +impl Backend for Played<'_> { + fn run( + &self, + invocation: &Invocation, + deadline: Duration, + logs: &Path, + ) -> Result { + let Self::Trial { script, trial } = self else { + return Local.run(invocation, deadline, logs); + }; + if invocation.program == "cargo-agents" { + return Ok(init(script.init, trial)); + } + fs::write(launched(logs), launched_model(invocation)).unwrap(); + play(script.session, trial, &invocation.log(logs, "stdout")) + } +} + +/// Where the played client keeps the model it was launched with. +fn launched(logs: &Path) -> PathBuf { + logs.join("launched-model") +} + +fn launched_model(invocation: &Invocation) -> String { + let flag = invocation.args.iter().position(|arg| arg == "--model"); + let model = flag.and_then(|flag| invocation.args.get(flag + 1)).unwrap(); + model.to_str().unwrap().to_owned() +} + +fn init(init: Init, trial: &Trial) -> Finished { + match init { + Init::Succeeds => exited(0), + Init::Fails => exited(1), + Init::Syncs => { + fs::create_dir_all(trial.sync_records()).unwrap(); + exited(0) + } + } +} + +fn play(session: Session, trial: &Trial, transcript: &Path) -> Result { + if matches!(session, Session::Unsupervised | Session::UncleanTimeout) { + // This would be reported as a created config if the driver inspected paths. + fs::write(trial.project().join(".claude/settings.local.json"), "{}").unwrap(); + } + if let Session::CannotStart = session { + return Err(ProcessError::Spawn { + program: "client".into(), + source: io::Error::from(io::ErrorKind::NotFound), + }); + } + match session { + Session::WrapperExits(code) => { + fs::write(transcript, "").unwrap(); + return Ok(exited(code)); + } + Session::TimesOut => return Ok(Finished::never_started()), + Session::Unsupervised => return Err(ProcessError::OutputNotClosed), + Session::UncleanTimeout => { + let mut finished = Finished::never_started(); + finished.after_deadline = Some(ProcessError::OutputNotClosed); + return Ok(finished); + } + Session::Solves + | Session::InstallsSkill + | Session::ChangesSkillContents + | Session::ChangesSkillMarker + | Session::CannotStart + | Session::PlantsResult => {} + } + if matches!( + session, + Session::InstallsSkill | Session::ChangesSkillContents | Session::ChangesSkillMarker + ) { + ownership::install_skill(trial, session); + } + fs::copy(solution(), trial.project().join("src/lib.rs")).unwrap(); + fs::write(transcript, "finished\n").unwrap(); + if let Session::PlantsResult = session { + fs::write(trial.root().join(RESULT_FILE), "planted").unwrap(); + } + Ok(exited(0)) +} + +fn solution() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")).join("../evals/tasks/formatting/solution/src/lib.rs") +} + +fn exited(code: i32) -> Finished { + let mut finished = Finished::never_started(); + finished.status = Some(exit_status(code)); + finished +} + +#[cfg(unix)] +fn exit_status(code: i32) -> ExitStatus { + std::os::unix::process::ExitStatusExt::from_raw(code << 8) +} + +#[cfg(windows)] +fn exit_status(code: i32) -> ExitStatus { + std::os::windows::process::ExitStatusExt::from_raw(code.cast_unsigned()) +} + +/// A client run as `client`, whose transcript finishes when it says `finished`. It +/// passes its model as `--model`, so a test can see which model was launched. +struct Stub { + client: Client, + model: &'static str, + /// Shows no loads, whatever skills it was asked about. + loses_loads: bool, +} + +impl Stub { + fn new() -> Self { + Self { + client: Client::ClaudeCode, + model: "a-model", + loses_loads: false, + } + } +} + +impl ClientRunner for Stub { + fn client(&self) -> Client { + self.client + } + + fn model(&self) -> &str { + self.model + } + + fn invocation( + &self, + trial: &Trial, + prompt: &str, + env: Vec<(OsString, OsString)>, + ) -> Invocation { + Invocation { + program: "client".into(), + args: vec!["--model".into(), self.model.into(), prompt.into()], + dir: trial.project(), + env, + log_name: "client".to_owned(), + } + } + + fn read(&self, _: &Trial, transcript: &Path, skills: &[InstalledSkill]) -> Reading { + let finished = fs::read_to_string(transcript).is_ok_and(|text| text == "finished\n"); + let loads = if self.loses_loads { 0 } else { skills.len() }; + Reading { + introduction: if finished { + SessionIntroduction::Observed + } else { + SessionIntroduction::NotObserved + }, + completion: Ok(if finished { + Completion::Finished + } else { + Completion::Missing + }), + loads: vec![Load::Unknown; loads], + hooks: HookEvidence::Unsupported, + activity: ToolActivity::unread(), + summary: Summary::unread(Usage::Unsupported), + } + } + + fn unread_hooks(&self) -> HookEvidence { + HookEvidence::Unsupported + } + + fn unread_usage(&self) -> Usage { + Usage::Unsupported + } +} + +fn provenance() -> Provenance { + Provenance { + client_version: Some("1.0.0".to_owned()), + os: "test".to_owned(), + toolchain: None, + isolation: IsolationMode::Enforced, + srt_version: None, + deadlines: Deadlines { + setup: Duration::from_secs(60), + agent: Duration::from_secs(60), + grading: Duration::from_secs(300), + }, + } +} + +fn arm(configuration: Configuration) -> Arm { + match configuration { + Configuration::Baseline => Arm::Baseline, + Configuration::Symposium => Arm::Symposium(SymposiumBuild { + version: "cargo-agents 1.0.0".to_owned(), + sha256: crate::digest::file(&solution()).unwrap(), + revision: None, + }), + } +} + +/// A batch of `formatting` on a scratch machine, to run trials in. +struct Bench { + _host: Host, + batch: Batch, + inputs: Inputs, + machine: Machine, + provenance: Provenance, +} + +impl Bench { + fn new() -> Self { + let host = Host::new(); + let batch = host.batch(); + let inputs = inputs(&batch, "formatting"); + let machine = host.machine(); + Self { + _host: host, + batch, + inputs, + machine, + provenance: provenance(), + } + } + + fn run( + &self, + configuration: Configuration, + sandbox: &Scripted, + runner: &Stub, + ) -> Result { + let key = api_key(); + let driver = Driver { + batch: &self.batch, + inputs: &self.inputs, + machine: &self.machine, + sandbox, + api_host: "api.example.com", + api_key: &key, + provenance: &self.provenance, + }; + driver.run(1, arm(configuration), runner) + } + + /// The result of trial 1, which must have one. + fn result(&self, configuration: Configuration, sandbox: &Scripted) -> Value { + let path = self.run(configuration, sandbox, &Stub::new()).unwrap(); + serde_json::from_slice(&fs::read(path).unwrap()).unwrap() + } + + fn trial_root(&self, configuration: Configuration) -> PathBuf { + self.batch + .eval_root() + .join(format!("batch/trial-1-{configuration}")) + } +} + +#[test] +fn a_solved_trial_is_graded_and_recorded() { + let bench = Bench::new(); + + let result = bench.result(Configuration::Baseline, &Scripted::new(Session::Solves)); + + assert_eq!(result["reached"]["phase"], "graded", "{result:#}"); + assert_eq!(result["outcome"]["result"], "succeeded", "{result:#}"); + assert_eq!(result["session"]["ended"]["status"], "completed"); + assert!(result["expectations"].is_null()); +} + +#[test] +fn uncertain_client_containment_stops_before_evidence_or_grading() { + for configuration in [Configuration::Baseline, Configuration::Symposium] { + for session in [Session::Unsupervised, Session::UncleanTimeout] { + let bench = Bench::new(); + let sandbox = Scripted::new(session); + let result = bench.result(configuration, &sandbox); + assert_eq!(sandbox.grading_calls.get(), 0); + assert!(!bench.trial_root(configuration).join("grading").exists()); + assert_eq!( + result["reached"]["phase"], "uncontained_session", + "{result:#}" + ); + assert_eq!(result["outcome"]["result"], "failed", "{result:#}"); + assert_eq!(result["outcome"]["failure"], "session", "{result:#}"); + assert!(result["code"].is_null()); + assert!(result["expectations"].is_null()); + assert_eq!(result["isolation"]["config_changes"], serde_json::json!([])); + } + } +} + +#[test] +fn a_symposium_trial_is_graded_with_its_expectations() { + let bench = Bench::new(); + + let result = bench.result(Configuration::Symposium, &Scripted::new(Session::Solves)); + + assert_eq!(result["reached"]["phase"], "graded", "{result:#}"); + assert!(result["expectations"].is_object(), "{result:#}"); + // Nothing played Symposium's part, and the stub's transcript cannot show the + // missing evidence was left alone, so no expectation is decided. + assert_eq!(result["outcome"]["result"], "inconclusive", "{result:#}"); +} + +#[test] +fn a_failed_preparation_stops_at_setup() { + let bench = Bench::new(); + let starter = bench.inputs.evals().starter_dir(bench.inputs.task().name()); + fs::remove_dir_all(starter).unwrap(); + + let result = bench.result(Configuration::Baseline, &Scripted::new(Session::Solves)); + + assert_eq!(result["reached"]["phase"], "setup", "{result:#}"); + assert_eq!(result["outcome"]["result"], "setup_failure"); + assert!(result["session"].is_null()); +} + +#[test] +fn a_failed_init_stops_at_setup() { + let bench = Bench::new(); + let sandbox = Scripted { + init: Init::Fails, + ..Scripted::new(Session::Solves) + }; + + let result = bench.result(Configuration::Symposium, &sandbox); + + assert_eq!(result["reached"]["phase"], "setup", "{result:#}"); + assert!( + result["reached"]["error"] + .as_str() + .unwrap() + .contains("cargo agents init"), + "{result:#}" + ); + assert!(result["session"].is_null()); +} + +#[test] +fn a_failed_check_before_launch_stops_at_prelaunch() { + let bench = Bench::new(); + let sandbox = Scripted { + init: Init::Syncs, + ..Scripted::new(Session::Solves) + }; + + let result = bench.result(Configuration::Symposium, &sandbox); + + assert_eq!(result["reached"]["phase"], "prelaunch", "{result:#}"); + assert_eq!(result["outcome"]["result"], "setup_failure"); + assert!(result["session"].is_null()); +} + +#[test] +fn a_client_that_cannot_start_is_kept_and_not_graded() { + let bench = Bench::new(); + + let result = bench.result( + Configuration::Baseline, + &Scripted::new(Session::CannotStart), + ); + + assert_eq!(result["reached"]["phase"], "launch", "{result:#}"); + assert_eq!(result["session"]["ended"]["status"], "not_launched"); + assert!(result["code"].is_null()); + assert!( + !bench + .trial_root(Configuration::Baseline) + .join("grading") + .exists() + ); +} + +#[test] +fn a_wrapper_exit_before_the_session_is_setup_failure_in_both_arms() { + for code in [126, 127] { + for configuration in [Configuration::Baseline, Configuration::Symposium] { + let bench = Bench::new(); + let sandbox = Scripted::new(Session::WrapperExits(code)); + + let result = bench.result(configuration, &sandbox); + + assert_eq!(result["reached"]["phase"], "launch", "{result:#}"); + assert_eq!(result["outcome"]["result"], "setup_failure"); + assert_eq!(result["session"]["ended"]["status"], "not_launched"); + assert!( + result["session"]["ended"]["error"] + .as_str() + .is_some_and(|error| error.contains(&code.to_string())), + "{result:#}" + ); + assert!(result["code"].is_null(), "{result:#}"); + assert_eq!(sandbox.grading_calls.get(), 0); + assert!(!bench.trial_root(configuration).join("grading").exists()); + } + } +} + +#[test] +fn a_timed_out_session_is_still_graded() { + let bench = Bench::new(); + + let result = bench.result(Configuration::Baseline, &Scripted::new(Session::TimesOut)); + + assert_eq!( + result["session"]["ended"]["status"], "timed_out", + "{result:#}" + ); + assert_eq!(result["reached"]["phase"], "graded"); + assert!(result["code"].is_object(), "{result:#}"); +} + +#[test] +fn a_trial_sandbox_that_cannot_be_prepared_stops_at_setup() { + let bench = Bench::new(); + let sandbox = Scripted { + trial_sandbox: false, + ..Scripted::new(Session::Solves) + }; + + let result = bench.result(Configuration::Symposium, &sandbox); + + assert_eq!(result["reached"]["phase"], "setup", "{result:#}"); + assert_eq!(result["outcome"]["result"], "setup_failure"); + for phase in ["session", "code", "expectations"] { + assert!(result[phase].is_null(), "{phase}: {result:#}"); + } +} + +#[test] +fn a_grading_sandbox_that_cannot_be_prepared_excludes_the_trial() { + let bench = Bench::new(); + let sandbox = Scripted { + grading_sandbox: false, + ..Scripted::new(Session::Solves) + }; + + let result = bench.result(Configuration::Baseline, &sandbox); + + assert_eq!( + result["outcome"]["result"], "grading_exclusion", + "{result:#}" + ); +} + +#[test] +fn the_result_names_the_client_and_model_the_runner_launched() { + let bench = Bench::new(); + let runner = Stub { + client: Client::Codex, + model: "another-model", + ..Stub::new() + }; + + let path = bench + .run( + Configuration::Baseline, + &Scripted::new(Session::CannotStart), + &runner, + ) + .unwrap(); + + let result: Value = serde_json::from_slice(&fs::read(path).unwrap()).unwrap(); + assert_eq!(result["client"]["name"], "codex", "{result:#}"); + let session = bench.trial_root(Configuration::Baseline).join("session"); + let launched = fs::read_to_string(launched(&session)).unwrap(); + assert_eq!(launched, "another-model"); + assert_eq!(result["model"]["name"], launched, "{result:#}"); +} + +#[test] +fn a_planted_result_is_never_replaced() { + let bench = Bench::new(); + + let ran = bench.run( + Configuration::Baseline, + &Scripted::new(Session::PlantsResult), + &Stub::new(), + ); + + assert!(matches!(ran, Err(TrialError::Write { .. })), "{ran:?}"); + let planted = bench.trial_root(Configuration::Baseline).join(RESULT_FILE); + assert_eq!(fs::read_to_string(planted).unwrap(), "planted"); +} + +#[test] +fn parts_that_cannot_make_one_result_are_the_runners_error() { + let bench = Bench::new(); + let runner = Stub { + loses_loads: true, + ..Stub::new() + }; + + let ran = bench.run( + Configuration::Baseline, + &Scripted::new(Session::Solves), + &runner, + ); + + assert!( + matches!(ran, Err(TrialError::Result(ResultError::Loads { .. }))), + "{ran:?}" + ); + let root = bench.trial_root(Configuration::Baseline); + assert!(!root.join(RESULT_FILE).exists()); +} + +#[test] +fn a_trial_is_never_run_twice() { + let bench = Bench::new(); + let sandbox = Scripted::new(Session::CannotStart); + bench + .run(Configuration::Baseline, &sandbox, &Stub::new()) + .unwrap(); + + let again = bench.run(Configuration::Baseline, &sandbox, &Stub::new()); + + assert!(matches!(again, Err(TrialError::Allocate(_))), "{again:?}"); +} diff --git a/symposium-eval/src/run/trial/tests/ownership.rs b/symposium-eval/src/run/trial/tests/ownership.rs new file mode 100644 index 000000000..99726da81 --- /dev/null +++ b/symposium-eval/src/run/trial/tests/ownership.rs @@ -0,0 +1,150 @@ +//! Runner-installed files must not become part of the code being graded. + +use super::*; +use crate::grade::copy_exact; + +#[test] +fn an_installed_skill_is_not_imported_into_grading() { + assert_clean_grading(&skill_trial_result(&skill_bench("installed"))); +} + +#[test] +fn a_used_only_skill_is_not_imported_into_grading() { + assert_clean_grading(&skill_trial_result(&skill_bench("used"))); +} + +#[test] +fn an_unnamed_sibling_skill_is_not_imported_into_grading() { + let mut bench = skill_bench("installed"); + add_sibling_skill(&mut bench); + + let result = skill_trial_result(&bench); + + assert!( + bench + .trial_root(Configuration::Symposium) + .join("project/.claude/skills/rust-testing/SKILL.md") + .is_file(), + "the session must have installed the unnamed sibling" + ); + assert_clean_grading(&result); +} + +#[test] +fn altered_skill_contents_remain_in_the_grading_submission() { + assert_modified_skill_is_graded(Session::ChangesSkillContents); +} + +#[test] +fn altered_skill_metadata_remains_in_the_grading_submission() { + assert_modified_skill_is_graded(Session::ChangesSkillMarker); +} + +fn assert_modified_skill_is_graded(session: Session) { + let bench = skill_bench("used"); + + let result = bench.result(Configuration::Symposium, &Scripted::new(session)); + + assert_eq!(result["code"]["build"], "pass", "{result:#}"); + assert_eq!( + result["code"]["hidden"]["double"], "fail", + "an altered skill must remain an agent edit in the grading copy: {result:#}" + ); +} + +fn skill_trial_result(bench: &Bench) -> Value { + let sandbox = Scripted::new(Session::InstallsSkill); + + let result = bench.result(Configuration::Symposium, &sandbox); + + let installed = bench + .trial_root(Configuration::Symposium) + .join("project/.claude/skills/rust-formatting/SKILL.md"); + assert!( + installed.is_file(), + "the session must have installed the skill" + ); + assert_eq!(result["session"]["ended"]["status"], "completed"); + assert_eq!(result["code"]["build"], "pass", "{result:#}"); + result +} + +fn assert_clean_grading(result: &Value) { + assert_eq!( + result["code"]["hidden"]["double"], "pass", + "a runner-installed skill must not reach the grading copy: {result:#}" + ); + assert_eq!( + result["code"]["edits_outside_editable"], + serde_json::json!([]) + ); +} + +fn skill_bench(assertion: &str) -> Bench { + let mut bench = Bench::new(); + let name = bench.inputs.task().name().clone(); + let task = bench.inputs.evals().task_dir(&name); + fs::write( + task.join("task.toml"), + format!( + "kind = \"regression\"\nplugins = [\"format-on-stop\"]\nprompt = \"Double a number.\"\neditable = [\"src/**\", \".claude/**\"]\n\n[expected.skills]\n{assertion} = [\"rust-formatting\"]\n" + ), + ) + .unwrap(); + fs::write( + task.join("checks/double.rs"), + r#"#[test] +fn doubles_without_runner_skills() { + assert_eq!(formatting::double(21), 42); + assert!( + !std::path::Path::new(".claude/skills").exists(), + "runner-installed skills leaked into the grading copy" + ); +} +"#, + ) + .unwrap(); + bench.inputs = Inputs::reopen(&bench.batch, &name).unwrap(); + bench +} + +fn add_sibling_skill(bench: &mut Bench) { + let skill = bench + .batch + .inputs() + .join("plugins/format-on-stop/skills/rust-testing"); + fs::create_dir_all(&skill).unwrap(); + fs::write( + skill.join("SKILL.md"), + "---\nname: rust-testing\ndescription: Write focused Rust tests.\n---\nUse assertions.\n", + ) + .unwrap(); + bench.inputs = Inputs::reopen(&bench.batch, bench.inputs.task().name()).unwrap(); +} + +/// Plays the SessionStart sync with the exact loose-skill layout this PR supports. +pub(super) fn install_skill(trial: &Trial, session: Session) { + let skills = trial.registry().join("format-on-stop/skills"); + for source in fs::read_dir(skills).unwrap() { + let source = source.unwrap(); + install_copy(trial, &source.path(), &source.file_name()); + } + let copy = trial.project().join(".claude/skills/rust-formatting"); + match session { + Session::ChangesSkillContents => { + fs::write(copy.join("SKILL.md"), "changed by the agent").unwrap(); + } + Session::ChangesSkillMarker => { + fs::write(copy.join(".symposium"), "changed by the agent").unwrap(); + } + _ => {} + } +} + +fn install_copy(trial: &Trial, source: &Path, name: &std::ffi::OsStr) { + let copy = trial.project().join(".claude/skills").join(name); + fs::create_dir_all(copy.parent().unwrap()).unwrap(); + copy_exact(source, ©, "test skill").unwrap(); + fs::write(copy.join(".symposium"), "").unwrap(); + fs::write(copy.join(".gitignore"), "*\n").unwrap(); +} diff --git a/symposium-eval/src/task/mod.rs b/symposium-eval/src/task/mod.rs index 75ed20864..e90d49809 100644 --- a/symposium-eval/src/task/mod.rs +++ b/symposium-eval/src/task/mod.rs @@ -174,6 +174,11 @@ impl Task { self.kind } + /// What the agent is asked to do. + pub(crate) fn prompt(&self) -> &str { + &self.prompt + } + pub(crate) fn fixtures(&self) -> &[FixtureName] { &self.fixtures } @@ -205,10 +210,17 @@ impl Task { expect(dead_code, reason = "the runner decides these later in RFD step 4") )] pub(crate) fn skill_source(&self, skill: &SkillName) -> Option<&Path> { - self.skill_sources + let mut sources = self + .skill_sources .iter() - .find(|source| source.name == *skill) - .map(|source| source.dir.as_path()) + .filter(|source| source.name == skill.as_str()); + let source = sources.next()?; + sources.next().is_none().then_some(source.dir.as_path()) + } + + /// Every trusted provider, whether or not an expectation names its skill. + pub(crate) fn provided_skills(&self) -> &[SkillSource] { + &self.skill_sources } /// The plugin hooks that must leave a successful record, each with its event. @@ -772,6 +784,30 @@ mod tests { ); } + #[test] + fn the_provider_catalogue_keeps_unnamed_skills_from_every_origin() { + let files = [ + ("first/Symposium.toml", SKILLS.to_owned()), + ("first/skills/named/SKILL.md", skill_md("named")), + ("first/skills/unnamed/SKILL.md", skill_md("unnamed")), + ("second/SKILL.md", skill_md("unnamed")), + ]; + let (_root, task) = + expecting_skills(&["first", "second"], r#"installed = ["named"]"#, &files); + let task = task.unwrap(); + + assert_eq!(task.provided_skills().len(), 3); + assert_eq!( + task.provided_skills() + .iter() + .filter(|source| source.name == "unnamed") + .count(), + 2 + ); + assert!(task.skill_source(&skill_name("named")).is_some()); + assert!(task.skill_source(&skill_name("unnamed")).is_none()); + } + #[test] fn a_skill_two_sources_provide_is_ambiguous_wherever_it_is_listed() { let files = [ diff --git a/symposium-eval/src/task/skills.rs b/symposium-eval/src/task/skills.rs index e09b51d6c..7fe6477bd 100644 --- a/symposium-eval/src/task/skills.rs +++ b/symposium-eval/src/task/skills.rs @@ -1,4 +1,4 @@ -//! The `SKILL.md` that provides each expected skill, from the task's plugins (RFD +//! The `SKILL.md` for every skill provided by the task's plugins (RFD //! reference, "Skills"). //! //! Skills are found as Symposium finds a registry plugin's: a plugin with a @@ -21,29 +21,22 @@ use crate::tree::EntryKind; /// holding an entry of this name, whatever kind it is. const MARKER: &str = ".symposium"; -/// An expected skill, and the directory in `evals/` that provides it. +/// A provided skill, and its trusted source directory in `evals/`. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) struct SkillSource { - pub(crate) name: SkillName, + pub(crate) name: String, pub(crate) dir: PathBuf, } -/// A skill a plugin provides. -struct Provided { - name: String, - dir: PathBuf, -} - -/// Finds the one provider of each skill `expected` names. A skill that must be installed -/// or used needs one; any named skill with two is refused, since sync would then install -/// neither under its own name. One `SKILL.md` reached through two groups is one -/// provider, as sync counts it. +/// Catalogues every provider and validates the skills `expected` names. A skill that +/// must be installed or used needs one provider; any named skill with two is refused. +/// One `SKILL.md` reached through two groups is one provider, as sync counts it. pub(super) fn resolve( plugins: &[Plugin<'_>], expected: &SkillExpectations, errors: &mut Vec, ) -> Vec { - let mut provided: Vec = Vec::new(); + let mut provided: Vec = Vec::new(); for plugin in plugins { for skill in plugin_skills(plugin, errors) { if !provided.iter().any(|seen| seen.dir == skill.dir) { @@ -53,7 +46,6 @@ pub(super) fn resolve( } let needed = |skill: &SkillName| expected.installed().contains(skill) || expected.used().contains(skill); - let mut sources = Vec::new(); for skill in expected.names() { let mut providers = provided .iter() @@ -64,18 +56,15 @@ pub(super) fn resolve( first: first.dir.clone(), second: second.dir.clone(), }), - (Some(only), None) => sources.push(SkillSource { - name: skill.clone(), - dir: only.dir.clone(), - }), + (Some(_), None) => {} (None, _) if needed(skill) => errors.push(TaskError::UnprovidedSkill(skill.clone())), (None, _) => {} } } - sources + provided } -fn plugin_skills(plugin: &Plugin<'_>, errors: &mut Vec) -> Vec { +fn plugin_skills(plugin: &Plugin<'_>, errors: &mut Vec) -> Vec { let Some(manifest) = &plugin.manifest else { // A plugin without a manifest is a registry's bare skill only at its root. let bare = plugin @@ -190,7 +179,7 @@ fn is_within(path: &str, dir: &str) -> bool { } /// Each skill in `dirs` with its name, if Symposium would load it. -fn named(plugin: &Plugin<'_>, dirs: Vec, errors: &mut Vec) -> Vec { +fn named(plugin: &Plugin<'_>, dirs: Vec, errors: &mut Vec) -> Vec { dirs.into_iter() .filter_map(|dir| { let path = dir.join("SKILL.md"); @@ -202,7 +191,7 @@ fn named(plugin: &Plugin<'_>, dirs: Vec, errors: &mut Vec) - } }; match skill_md::loadable_name(&text) { - Ok(name) => Some(Provided { name, dir }), + Ok(name) => Some(SkillSource { name, dir }), Err(problem) => { errors.push(TaskError::UnloadableSkill { plugin: plugin.name.clone(),