SimoBench

What it measures

The release contains 126 selected problems. Each submitted proof receives an olympiad-style grade from zero to seven, for a maximum full-run total of 882.

How to read the result

A model can get the right idea without completing a proof. Inspect the grade distribution and count of full-credit solutions rather than treating partial-credit totals as binary pass rates.

Limits and comparison traps

Synthetic variants reduce direct statement reuse but cannot establish absence of contamination. Reference solutions support mathematical judging, not machine-checked formal proof.

Benchmark reference notes

126 olympiad and research-adjacent proof problems scored across 7 criteria (882 total points) including proof validity, scope control, and literature triage.

Where this fits in an evaluation suite

Final-answer accuracy, proof credit, and expert-reviewed research progress are distinct signals. A proof-aware evaluation should retain partial work and audit strong claims.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Preserve the named model configuration, scoring rubric, task set, review coverage, and complete-run denominator.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.