JudgeBench

350 challenging response pairs with objective ground truth across Knowledge, Reasoning, Math, and Coding testing LLM-as-a-Judge models.26 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
o3-mini →OpenAI	76% 3 reports · range 70.57–80.86%	JudgeBench · Overall pairwise accuracy[1] View 3 reports 80.86% · JudgeBench · Overall pairwise accuracy · Arena-Hard judge scaffold · JudgeBench · Arena-Hard judge scaffold [1] 76.57% · JudgeBench · Overall pairwise accuracy · Arena-Hard judge scaffold · JudgeBench · Arena-Hard judge scaffold [1] 70.57% · JudgeBench · Overall pairwise accuracy · Arena-Hard judge scaffold · JudgeBench · Arena-Hard judge scaffold [1]	Published observations span 10.29 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. Leads this displayed comparison.
o1-preview →OpenAI	75.43% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#2 of 32 in this displayed comparison; 0.57 pp below the leader. Source notes Source cell: 75.43
Deepseek-R1 →DeepSeek	73.14% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#3 of 32 in this displayed comparison; 2.86 pp below the leader. Source notes Source cell: 73.14
o1-mini →OpenAI	65.71% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#4 of 32 in this displayed comparison; 10.29 pp below the leader. Source notes Source cell: 65.71
Claude-3.5-Sonnet →Anthropic	64.29% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#5 of 32 in this displayed comparison; 11.71 pp below the leader. Source notes Source cell: 64.29
Skywork-Reward-Gemma-2-27B →Google	64.29% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Reward models	#5 of 32 in this displayed comparison; 11.71 pp below the leader. Source notes Source cell: 64.29
InternLM2-20B-Reward →	63.43% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Reward models	#7 of 32 in this displayed comparison; 12.57 pp below the leader. Source notes Source cell: 63.43
Skywork-Reward-Llama-3.1-8B →Meta	62.29% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Reward models	#8 of 32 in this displayed comparison; 13.71 pp below the leader. Source notes Source cell: 62.29
GRM-Gemma-2B →Google	59.43% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Reward models	#9 of 32 in this displayed comparison; 16.57 pp below the leader. Source notes Source cell: 59.43
InternLM2-7B-Reward →	59.43% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Reward models	#9 of 32 in this displayed comparison; 16.57 pp below the leader. Source notes Source cell: 59.43
Skywork-LLaMA-3.1B-70B →Meta	57.43% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#11 of 32 in this displayed comparison; 18.57 pp below the leader. Source notes Source cell: 57.43
Llama-3.1-405B-Instruct →Meta	56.86% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#12 of 32 in this displayed comparison; 19.14 pp below the leader. Source notes Source cell: 56.86
Arena-Hard Judge (GPT-4o) →OpenAI	56.57% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Prompted Judges	#13 of 32 in this displayed comparison; 19.43 pp below the leader. Source notes Source cell: 56.57
GPT-4o →OpenAI	56.57% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#13 of 32 in this displayed comparison; 19.43 pp below the leader. Source notes Source cell: 56.57
Skywork-LLaMA-3.1B-8B →Meta	53.43% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#15 of 32 in this displayed comparison; 22.57 pp below the leader. Source notes Source cell: 53.43
Llama-3.1-70B-Instruct →Meta	52.29% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#16 of 32 in this displayed comparison; 23.71 pp below the leader. Source notes Source cell: 52.29
Vanilla (GPT-4o) →OpenAI	50.86% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Prompted Judges	#17 of 32 in this displayed comparison; 25.14 pp below the leader. Source notes Source cell: 50.86
GPT-4o-mini →OpenAI	50% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#18 of 32 in this displayed comparison; 26 pp below the leader. Source notes Source cell: 50.00
Gemini-1.5-pro →Google	47.14% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#19 of 32 in this displayed comparison; 28.86 pp below the leader. Source notes Source cell: 47.14
VertexAI Evaluation (Gemini-1.5-pro) →Google	44.57% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Prompted Judges	#20 of 32 in this displayed comparison; 31.43 pp below the leader. Source notes Source cell: 44.57
Llama-3.1-8B-Instruct →Meta	40.86% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#21 of 32 in this displayed comparison; 35.14 pp below the leader. Source notes Source cell: 40.86
Prometheus2-8x7b →	40.29% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#22 of 32 in this displayed comparison; 35.71 pp below the leader. Source notes Source cell: 40.29
Gemini-1.5-flash →Google	39.71% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#23 of 32 in this displayed comparison; 36.29 pp below the leader. Source notes Source cell: 39.71
Prometheus2-bgb-8x7b →	39.43% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#24 of 32 in this displayed comparison; 36.57 pp below the leader. Source notes Source cell: 39.43
AutoJ →	36.57% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#25 of 32 in this displayed comparison; 39.43 pp below the leader. Source notes Source cell: 36.57
JudgeLM-33B →	35.71% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#26 of 32 in this displayed comparison; 40.29 pp below the leader. Source notes Source cell: 35.71
Prometheus2-7b →	34.86% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#27 of 32 in this displayed comparison; 41.14 pp below the leader. Source notes Source cell: 34.86
ChatEval →	34% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Multi-Agent Judges	#28 of 32 in this displayed comparison; 42 pp below the leader. Source notes Source cell: 34.00
Claude-3-Haiku →Anthropic	33.14% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Arena-Hard judge scaffold	#29 of 32 in this displayed comparison; 42.86 pp below the leader. Source notes Source cell: 33.14
JudgeLM-13B →	26.86% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#30 of 32 in this displayed comparison; 49.14 pp below the leader. Source notes Source cell: 26.86
JudgeLM-7B →	25.14% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#31 of 32 in this displayed comparison; 50.86 pp below the leader. Source notes Source cell: 25.14
PandaLM →	13.14% 1 report	JudgeBench · Overall pairwise accuracy[1]JudgeBench · Prompted and trained judges · Fine-tuned Judges	#32 of 32 in this displayed comparison; 62.86 pp below the leader. Source notes Source cell: 13.14

Sources

JudgeBench: A Benchmark for Evaluating LLM-based Judges — published result table ↗
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang et al. · 2024
All 8 authors
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica
Dates & evaluation details
JudgeBench · Overall pairwise accuracy · Prompted and trained judges · 2026-10-26
Published model comparison, 2024. JudgeBench · Prompted and trained judges. Pairwise evaluation accuracy over knowledge, reasoning, math and coding. The source judge or reward-model configuration is retained. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
JudgeBench · Overall pairwise accuracy · Arena-Hard judge scaffold · 2026-10-26
Published model comparison, 2024. JudgeBench · Arena-Hard judge scaffold. Pairwise evaluation accuracy over knowledge, reasoning, math and coding. The source judge or reward-model configuration is retained. Table 2; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
JudgeBench · Overall pairwise accuracy · Reward models · 2026-10-26
Published model comparison, 2024. JudgeBench · Reward models. Pairwise evaluation accuracy over knowledge, reasoning, math and coding. The source judge or reward-model configuration is retained. Table 3; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

350 challenging response pairs with objective ground truth across Knowledge, Reasoning, Math, and Coding testing LLM-as-a-Judge models.

Reading the measurement

Separate satisfying instructions from factual accuracy, human preference and safe behavior. Inspect strict all-constraint success as well as average criterion credit. Judge choice and adversarial prompt selection can materially change the result.

Published evaluation

Pairwise evaluation accuracy over knowledge, reasoning, math and coding. The source judge or reward-model configuration is retained.

Where this fits in an evaluation suite

Format compliance, preference, truthfulness, and safety require separate measurements. Keep desirable writing style from masking factual errors.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Pairwise evaluation accuracy over knowledge, reasoning, math and coding. The source judge or reward-model configuration is retained.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

JudgeBench: A Benchmark for Evaluating LLM-based Judges — published result table ↗
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang et al. · 2024
All 8 authors
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All instructions & behavior benchmarks →

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-PersonalityBench

IFBench

IFEval

WildBench