What it measures
A full evaluation uses 226 candidate problems. External grading distinguishes mathematically sound work, useful incomplete progress, material gaps, and incorrect claims.
How to read the result
A strong-claim audit answers a different question from an average grade: how often does an asserted resolution survive scrutiny? Review both before choosing a mathematical research assistant.
Limits and comparison traps
Published runs use different workflows, including Codex and API setups. Small grade differences can depend on a few audit decisions. This is mathematical review, not formal theorem certification.
Benchmark reference notes
226 open/research math candidate problems evaluating scoped progress, theorem use, counterexamples, finite checks, and proof-gap audits.
Where this fits in an evaluation suite
Final-answer accuracy, proof credit, and expert-reviewed research progress are distinct signals. A proof-aware evaluation should retain partial work and audit strong claims.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
Preserve the named model configuration, scoring rubric, task set, review coverage, and complete-run denominator.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.
PRIMARY EVIDENCE & FURTHER READING
Reviewed 2026-10-26. Explanation and interpretation by RLTasks. This evaluation is from Ulam, which powers RLTasks.