Skip to content

test(eval): write a versioned trial result - #437

Merged
jackh726 merged 2 commits into
mainfrom
feat/agent-eval-results
Nov 16, 2026
Merged

jackh726 merged 2 commits into
mainfrom
feat/agent-eval-results

Conversation

@gmemuriuki

@gmemuriuki gmemuriuki commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

The evaluator can now collect the pieces of a trial, but it does not yet turn them into a durable result.

This adds the outcome decision and writes a versioned result.json. Setup failures, failed submissions, inconclusive evidence, and successful trials remain distinct rather than being collapsed into a single failure. Results are written atomically and never replace an existing
result.

Snapshot tests cover the result shape and the precedence between the different outcomes.

Stack: #436 → #437 → #438 → #439 → #441 → #442 → #443

Disclosure questions

AI disclosure.

  • The AI tool authored large parts of the code

Questions for reviewers.

Does the outcome precedence preserve the distinctions we will need when reading evaluation results?

@gmemuriuki
gmemuriuki force-pushed the feat/agent-eval-results branch from 84fdda6 to 2c66a67 Compare November 16, 2026 11:31
Base automatically changed from feat/agent-eval-evidence to main November 16, 2026 14:34
Place each trial in the RFD's count from how far it got, its grading,
and its expectations: a setup failure, an isolation or grading
exclusion, or succeeded, failed, or inconclusive.

Count a session the runner lost hold of, or whose ending it could not
read, as failed, since excluding it would reward a configuration that
breaks supervision.
Record each trial as result.json, from schema types of its own built
by small conversions, as cargo and nextest do. What was planned is
always there; a phase's output is null only when the trial stopped
before it, and reached says where and why.

Write it atomically and never over an earlier result, and pin each
shape with an insta snapshot.
@jackh726
jackh726 force-pushed the feat/agent-eval-results branch from 2c66a67 to b92f2f3 Compare November 16, 2026 14:34
@jackh726
jackh726 added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 7f62cc5 Nov 16, 2026
8 checks passed
@jackh726
jackh726 deleted the feat/agent-eval-results branch November 16, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants