Skip to content

Preserve WildBench saved-answer metadata during judging - #171

Merged
kargibora merged 1 commit into
mainfrom
fix/wildbench-output-metadata
Nov 5, 2026
Merged

kargibora merged 1 commit into
mainfrom
fix/wildbench-output-metadata

Conversation

@kargibora

Copy link
Copy Markdown
Member

Summary

Preserve WildBench's instruction and saved-answer tags when preparing answers for judging.

Problem

Both tables contain primary_tag. Combining them raises a metadata collision error, so scoring saved WildBench answers stops before judging.

Change

Rename the saved-answer field to output_primary_tag in the WildBench adapter. Keep the instruction's primary_tag unchanged. The shared preparation code needs no changes.

Before / after

Before:

Instruction:  primary_tag = Math
Saved answer: primary_tag = Algebra
Preparation:  overlapping metadata error

After:

Scored annotation:
  primary_tag: Math
  output_primary_tag: Algebra

Both values survive through preparation and judging.

Validation

  • All 20 WildBench tests pass, including a regression that preserves distinct tag values in scored annotations.
  • With the task-download work applied, two real saved answers were scored with network access blocked and a dummy judge. Both tag columns were retained.

Lines changed

Compared with main:

Area Added Removed Net
Adapter 5 1 +4
Test 46 0 +46
Total 51 1 +50

@kargibora

Copy link
Copy Markdown
Member Author

This is a hot fix so merging this

@kargibora
kargibora merged commit fc6d82f into main Nov 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant