Skip to content

report schema: state the evaluate verdicts and counts in the report - #316

Merged
jar-ben merged 1 commit into
masterfrom
jaroslav/evaluation-summary
Nov 18, 2026
Merged

jar-ben merged 1 commit into
masterfrom
jaroslav/evaluation-summary

Conversation

@jar-ben

@jar-ben jar-ben commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Problem

An evaluate report holds the raw facts (gap lists, run records, rule outcomes, property coverage)
but not the verdicts drawn from them. Every renderer derives them itself, and the derivations
differ.

Fix

The report states the verdicts, so a renderer only displays them:

  • ConfRunRecord.status: ran, build_failed or run_failed.
  • ConfCoverage.status: complete, incomplete, orphaned or unknown. It is about what the
    confs select, not how their runs went; unknown is a conf that did not build, so what it selects
    is not known.
  • AutoProverReport.evaluation_summary: how many specs have each status, how many confs in scope
    ran or failed, how many rules have each outcome, and the two property axes (attempted, and the
    faithfulness of the attempted ones).

All three are optional, so older reports load. schema_version 3.6.

Testing

The evaluate-report round trip carries the new fields, and a report without them loads. The
non-expensive suite passes.

@jar-ben
jar-ben force-pushed the jaroslav/evaluation-summary branch from 0b7b5f0 to 442f3f4 Compare November 18, 2026 12:51
@jar-ben
jar-ben merged commit ba5053c into master Nov 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant