Legal work is assessed against task criteria and the supplied source documents, with a gate for material hallucinations.
24 model results
One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
Published observations span 0.04 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
Leads this displayed comparison.
Published breakdowns
95.5% criterion credit; all-pass 25.6% before hallucination checks → 13.1% after. 0.62 material hallucinations per task.
92.8% criterion credit; all-pass 16.7% before hallucination checks → 9.4% after. 0.74 material hallucinations per task. #3 of 24 in this displayed comparison; 3.63 pp below the leader.
96.0% of criteria met; all-pass credit falls to 8.9% after gating. 1.68 material hallucinations per task. #4 of 24 in this displayed comparison; 4.19 pp below the leader.
Published observations span 0.01 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#5 of 24 in this displayed comparison; 4.47 pp below the leader.
Published breakdowns
90.6% criterion credit; all-pass 8.9% before hallucination checks → 8.6% after. 0.03 material hallucinations per task.
90.6% criterion credit; all-pass 7.5% before hallucination checks → 6.9% after. 0.09 material hallucinations per task. #6 of 24 in this displayed comparison; 6.13 pp below the leader.
Published observations span 0.01 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#7 of 24 in this displayed comparison; 6.68 pp below the leader.
Published breakdowns
92.4% criterion credit; all-pass 14.4% before hallucination checks → 6.4% after. 1.32 material hallucinations per task.
93.0% of criteria met; all-pass credit falls to 5.3% after gating. 2.09 material hallucinations per task. #8 of 24 in this displayed comparison; 7.8 pp below the leader.
Published observations span 0.03 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#9 of 24 in this displayed comparison; 8.89 pp below the leader.
Published breakdowns
90.0% criterion credit; all-pass 8.9% before hallucination checks → 4.2% after. 0.59 material hallucinations per task.
94.8% criterion credit; all-pass 20.8% before hallucination checks → 3.9% after. 3.82 material hallucinations per task. #10 of 24 in this displayed comparison; 9.19 pp below the leader.
86.9% of criteria met; all-pass credit falls to 3.6% after gating. 0.07 material hallucinations per task. #11 of 24 in this displayed comparison; 9.47 pp below the leader.
86.0% criterion credit; all-pass 3.9% before hallucination checks → 3.3% after. 0.38 material hallucinations per task. #12 of 24 in this displayed comparison; 9.74 pp below the leader.
93.0% criterion credit; all-pass 16.4% before hallucination checks → 3.1% after. 2.11 material hallucinations per task. #13 of 24 in this displayed comparison; 10.02 pp below the leader.
92.1% criterion credit; all-pass 11.7% before hallucination checks → 2.8% after. 1.03 material hallucinations per task. #14 of 24 in this displayed comparison; 10.3 pp below the leader.
92.5% of criteria met; all-pass credit falls to 1.7% after gating. 2.58 material hallucinations per task. #15 of 24 in this displayed comparison; 11.41 pp below the leader.
91.1% criterion credit; all-pass 10.0% before hallucination checks → 0.8% after. 4.04 material hallucinations per task. #16 of 24 in this displayed comparison; 12.24 pp below the leader.
87.7% criterion credit; all-pass 4.4% before hallucination checks → 0.6% after. 1.61 material hallucinations per task. #17 of 24 in this displayed comparison; 12.52 pp below the leader.
93.0% criterion credit; all-pass 13.9% before hallucination checks → 0.3% after. 3.94 material hallucinations per task. #18 of 24 in this displayed comparison; 12.8 pp below the leader.
54.7% of criteria met; all-pass credit falls to 0.0% after gating. 1.07 material hallucinations per task. #19 of 24 in this displayed comparison; 13.08 pp below the leader.
64.5% of criteria met; all-pass credit falls to 0.0% after gating. 9.06 material hallucinations per task. #19 of 24 in this displayed comparison; 13.08 pp below the leader.
90.5% criterion credit; all-pass 6.7% before hallucination checks → 0.0% after. 13.96 material hallucinations per task. #19 of 24 in this displayed comparison; 13.08 pp below the leader.
81.3% criterion credit; all-pass 1.4% before hallucination checks → 0.0% after. 10.44 material hallucinations per task. #19 of 24 in this displayed comparison; 13.08 pp below the leader.
77.6% of criteria met; all-pass credit falls to 0.0% after gating. 7.24 material hallucinations per task. #19 of 24 in this displayed comparison; 13.08 pp below the leader.
85.6% of criteria met; all-pass credit falls to 0.0% after gating. 7.06 material hallucinations per task. #19 of 24 in this displayed comparison; 13.08 pp below the leader.
Publisher-selected recent-model cohort from the public evaluation page. Model labels preserve effort and fallback configurations. Values are a dated snapshot, not a live feed. Consult the publisher’s methodology for the harness, task split and judging; do not merge this cohort with vendor-run results.
Source-reported model settings are max unless a cell says otherwise. The original page reports numbers without a uniform percent symbol; original formatting is retained. OSWorld includes two scoring rules in each cell: leaderboard ordering uses partial credit only. The internal composite and AA index use separate scales; their scores cannot be averaged.
TRACE THE SCORE · Per-model notes use published breakdowns where available.
Reported in recent model launches
These model reports include this benchmark. Their harnesses, task subsets, scoring policies, and resource budgets can differ; follow each report’s evaluation notes before joining the numbers.
Legal work is assessed against task criteria and the supplied source documents, with a gate for material hallucinations.
How to read the result
The all-pass metric asks a stricter question than average rubric credit: did every required part survive review?
Limits and comparison traps
Version 1.1 adds a hallucination gate. Keep it separate from earlier results and from claims about deployment readiness.
Benchmark reference notes
AA's implementation of Harvey LAB where any material hallucination against source documents zeroes out the deliverable score.
Where this fits in an evaluation suite
Read accuracy together with error and abstention rates. Domain averages can hide language gaps, specialist failures, and poorly calibrated confidence.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
Use the exact implementation, version, task subset, tool access, attempt budget, and judge described by the source.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.