Repository navigation
Prepare task data and scoring assets for offline evaluation - #130
Merged
Merged
Conversation
kargibora
marked this pull request as ready for review
October 22, 2026 12:35
geoalgo
approved these changes
Nov 2, 2026
kargibora
force-pushed
the
feat/task-download
branch
from
November 4, 2026 16:36
45e3b80 to
9e9508d
Compare
Member
Author
|
This is a PR that fixes a potenital problem in the framework. It is adapted and updated to the recent main (also required for oellm-eval). Merging this |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prepare a task's data and scoring assets before running evaluation on a machine without network access. The download command uses the same adapters and effective configuration as the evaluation runner.
Problem
Downloading a task's main dataset does not prepare everything an evaluation may need. Saved candidate and baseline answers, metric calibration data, and tokenizer assets can still trigger downloads during a run. Config overrides also need to affect preparation, including the dataset revision and selected metrics.
Changes
judgearena tasks downloadfor packaged tasks or a run config.This prepares data and scoring assets; it does not generate answers or download model weights.
Before / after
Before, preparing a dataset alone could leave the evaluation needing more downloads:
Now, prepare the packaged task directly:
Or prepare the exact inputs selected by a run config:
The config path uses the resolved dataset, filters, candidate, baseline and metrics, including overrides of the task defaults.
Validation
Lines changed
Compared with
main: