GPQA Main

What it measures

The broader GPQA science evaluation complements the more commonly reported Diamond subset.

How to read the result

Keep subset identity visible. A broader question distribution can reveal whether a gain on Diamond extends to the rest of the suite.

Limits and comparison traps

Main and Diamond have different question selections and denominators; their percentages are not interchangeable.

Published evaluation

Accuracy on the 448-question main subset. Prompting and answer-selection methods are retained in the source model labels.

Where this fits in an evaluation suite

A correct multiple-choice answer and a supported scientific contribution demand different evidence. Check whether the score rewards a proposal, an executed experiment, or a completed research target.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Accuracy on the 448-question main subset. Prompting and answer-selection methods are retained in the source model labels.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.