What it measures
STEMBench120 uses 120 tasks, with 20 in each domain. Integer progress levels are supplemented by an experimental fractional score; the public STEMBench-small release contains a separate 60-task set.
How to read the result
Read complete-run means together with the progress distribution. A fractional progress score is not a solved percentage, and the ten published runs report no full-target successes.
Limits and comparison traps
Tool-assisted Codex runs and closed-book API runs have different resources. The fractional audit’s review-sensitivity allowances are not statistical confidence intervals.
Benchmark reference notes
120 open/research-level tasks across 6 domains (Biology, Chemistry, Physics, Astronomy, Earth Science, Engineering; 20 each) auditing scientific progress, reproducibility, and scope honesty.
Where this fits in an evaluation suite
A correct multiple-choice answer and a supported scientific contribution demand different evidence. Check whether the score rewards a proposal, an executed experiment, or a completed research target.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
Preserve the named model configuration, scoring rubric, task set, review coverage, and complete-run denominator.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.
PRIMARY EVIDENCE & FURTHER READING
Reviewed 2026-10-26. Explanation and interpretation by RLTasks. This evaluation is from Ulam, which powers RLTasks.