Skip to content

Prepare task data and scoring assets for offline evaluation - #130

Merged
kargibora merged 1 commit into
mainfrom
feat/task-download
Nov 4, 2026
Merged

kargibora merged 1 commit into
mainfrom
feat/task-download

Conversation

@kargibora

@kargibora kargibora commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

Prepare a task's data and scoring assets before running evaluation on a machine without network access. The download command uses the same adapters and effective configuration as the evaluation runner.

Problem

Downloading a task's main dataset does not prepare everything an evaluation may need. Saved candidate and baseline answers, metric calibration data, and tokenizer assets can still trigger downloads during a run. Config overrides also need to affect preparation, including the dataset revision and selected metrics.

Changes

  • Add judgearena tasks download for packaged tasks or a run config.
  • Reuse dataset adapters, baseline resolution and saved-answer loading.
  • Let metrics prefetch their scoring assets through a small optional hook.
  • Keep downloaded data separate for each dataset declaration, so different revisions do not share a fallback cache.
  • Cache WildBench's saved-answer model list for offline reuse.

This prepares data and scoring assets; it does not generate answers or download model weights.

Before / after

Before, preparing a dataset alone could leave the evaluation needing more downloads:

# Conceptual: this only prepares the main dataset.
download_dataset()
# Saved answers or scoring assets may still be fetched during evaluation.
run_evaluation()

Now, prepare the packaged task directly:

judgearena tasks download mt-bench

Or prepare the exact inputs selected by a run config:

# run.yaml
task: mt-bench
model:
  name: gpt-4
  baseline: gpt-3.5-turbo
judge:
  model: Dummy/[[A]]
generation:
  n_instructions: 2
# With network access:
judgearena tasks download --config_path run.yaml

# Later, using the same cache:
HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 judgearena --config_path run.yaml

The config path uses the resolved dataset, filters, candidate, baseline and metrics, including overrides of the task defaults.

Validation

  • 589 tests pass, including config overrides and revision-isolated reuse.
  • A two-question MT-Bench run completed with saved answers and network connections blocked.
  • WildBench data and saved answers loaded offline. Its separate metadata collision fix is outside this PR.
  • Lint passes.

Lines changed

Compared with main:

Area Added Removed Net
Code 153 37 +116
Tests 85 11 +74
Documentation 23 0 +23
Total 261 48 +213

@kargibora
kargibora marked this pull request as ready for review October 22, 2026 12:35
@kargibora kargibora changed the title Make the Tasks downloadable from the cli Prepare task data and scoring assets for offline evaluation Nov 4, 2026
@kargibora

Copy link
Copy Markdown
Member Author

This is a PR that fixes a potenital problem in the framework. It is adapted and updated to the recent main (also required for oellm-eval). Merging this

@kargibora
kargibora merged commit 7139499 into main Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants