Skip to content

About

Evaluating LLM with swappable judges: local, remote, openrouter on multiple benchmarks.

Resources

Stars

21 stars

Watchers

2 watching

Forks

Repository files navigation

🏛️ JudgeArena: LLM Evaluation with Swappable Judges

JudgeArena lets you compare language models and study the judges that evaluate them. Choose a benchmark, a model and a judge, then inspect the answers, judgments and scores from the run.

You can run a packaged benchmark or bring your own dataset, prompt, parser and metrics. Models and judges can run locally or through an API. Multilingual tasks let you evaluate across languages using the same setup.

Benchmarks · Configuration · Models · Cache and results · Custom tasks

✨ Key Features

  • 🎯 Run established benchmarks including AlpacaEval, Arena-Hard, MT-Bench and WildBench.
  • 🔄 Choose your judge from local models or hosted APIs, independently of the model you evaluate.
  • 🌍 Evaluate across languages with m-ArenaHard, multilingual fluency and arena datasets.
  • 🛠️ Bring your own components: datasets, prompt bundles, parsers and metrics.
  • 📁 Inspect and reuse results with saved annotations, resolved configs and inference caching.

🚀 Quick start

Install from source with Python 3.12 or later and uv:

git clone https://github.com/OpenEuroLLM/JudgeArena
cd JudgeArena
uv sync

For an API run, set your provider's credentials. This example uses OpenRouter:

export OPENROUTER_API_KEY="your-api-key"

uv run judgearena \
  --task arena-hard-v0.1 \
  --model.name OpenRouter/google/gemma-4-31b-it \
  --judge.model OpenRouter/google/gemma-4-31b-it \
  --generation.n_instructions 10

This generates ten candidate answers and compares them with the task's saved GPT-4 baseline answers. The judge evaluates both answer orders. The run saves its annotations, scores and resolved configuration under results/.

The example uses the same model for generation and judging; choose a different --judge.model to evaluate with another judge. Ten questions are useful for checking your setup. Omit the limit to evaluate the full task.

For local inference, install uv sync --extra vllm or uv sync --extra llamacpp. See model setup for backend options and chat templates.

⚙️ Configure an experiment

Save your settings in run.yaml:

task: arena-hard-v0.1
model:
  name: OpenRouter/google/gemma-4-31b-it
judge:
  model: OpenRouter/google/gemma-4-31b-it
generation:
  n_instructions: 10

Run the file, or override individual fields:

uv run judgearena --config_path run.yaml
uv run judgearena --config_path run.yaml --generation.n_instructions 100

Tasks define benchmark defaults. Experiment configs override selected settings, and CLI flags override the config. Omitted fields inherit task settings when the implementation stays the same; switching the runner or adapter replaces that section. See configuration for prompt, metric and dataset overrides.

📊 Supported benchmarks

Benchmark What it evaluates Evaluation mode
AlpacaEval Instruction-following answers, with length-controlled scoring Pairwise
Arena-Hard v0.1 and v2.0 Challenging questions, including category-specific judging in v2.0 Pairwise
m-ArenaHard v0.1 and v2.0 Translated Arena-Hard questions across 23 languages Pairwise
MT-Bench Multi-turn conversations Pairwise
MT-Bench-101 Answers with supplied conversation histories Independent scoring, then score comparison
WildBench Conversations with question-specific checklists Pairwise reward or independent score
Multilingual fluency Sentence continuations across 43 languages Pairwise
LMArena and ComparIA Arena battles with human preference labels Model Elo estimation or judge meta-evaluation

List the installed tasks and inspect their defaults:

uv run judgearena tasks list
uv run judgearena tasks show arena-hard-v0.1 --resolved

The benchmark guide lists task names, language selection, baselines and examples. Changing a benchmark's judge or rubric changes the evaluation protocol; record those choices when comparing results.

🛠️ Use your own data and judgments

A task can keep its dataset adapter, prompt bundles, parsers and metrics in one folder. Reference a Python component directly from YAML, or register a name in the task family's register.py and use it from other configs.

Prompt bundles keep the template, parser selection and judging mode together. Parsing logic stays in Python. You can also pass data directly through the Python API without creating a task file.

🤝 Contributing

Bug reports, new benchmarks and documentation fixes are welcome. Open an issue to discuss a change, or send a pull request.

Install the checks before making code changes:

uv run pre-commit install

📚 Citation

If you use JudgeArena in your research, please cite:

@article{lushtaku2026judgearena,
  title = {JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation},
  author = {Lushtaku, Erlis and Kargi, Bora and Elganzory, Ali and Ferreira, Fabio and Salamanca, Alejandro R. and Kreutzer, Julia and Salinas, David},
  journal = {arXiv preprint arXiv:2608.02620},
  year = {2026},
}

See additional citations for Soft-Elo and the tuned judge configurations.

About

Evaluating LLM with swappable judges: local, remote, openrouter on multiple benchmarks.

Resources

Stars

21 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages