What it measures
Open mathematical problems test literature search, proof construction, proof criticism, and formalization. The published comparison reports separate research rubric ratings rather than a solved percentage.
Reading the measurement
Partial credit can distinguish a useful approach from an empty attempt, but it is not the same as a completed task. Read the rubric, full-success count and score distribution together. For mathematical work, preserve the difference between expert review and acceptance by a formal proof checker.
Published evaluation
Author-assigned rubric ratings across research behaviors on open mathematical problems. Literature search, theorem proving and proof checking use 0–5 scales; formalization uses a binary 0/1 scale. The report describes testing dozens of problems without a fixed complete-run denominator or confidence intervals, so these are research assessments rather than percentages solved. GPT-5.2 uses extended thinking; the Grok version is not specified. Ulam powers RLTasks; affiliation disclosed.
Where this fits in an evaluation suite
Final-answer accuracy, proof credit, and expert-reviewed research progress are distinct signals. A proof-aware evaluation should retain partial work and audit strong claims.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
Author-assigned rubric ratings across research behaviors on open mathematical problems. Literature search, theorem proving and proof checking use 0–5 scales; formalization uses a binary 0/1 scale. The report describes testing dozens of problems without a fixed complete-run denominator or confidence intervals, so these are research assessments rather than percentages solved. GPT-5.2 uses extended thinking; the Grok version is not specified. Ulam powers RLTasks; affiliation disclosed.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.
PRIMARY EVIDENCE & FURTHER READING
Reviewed 2026-10-27. Explanation and interpretation by RLTasks. This evaluation is from Ulam, which powers RLTasks.