What it measures

102 data-driven scientific discovery coding tasks extracted from peer-reviewed papers across bioinformatics, computational chemistry, GIS, and psychology.

Reading the measurement

Partial credit can distinguish a useful approach from an empty attempt, but it is not the same as a completed task. Read the rubric, full-success count and score distribution together. For mathematical work, preserve the difference between expert review and acceptance by a formal proof checker.

Published evaluation

Task success · Without expert knowledge; ScienceAgentBench · Without expert knowledge. The source’s task split, model configuration and scoring units are retained.

Where this fits in an evaluation suite

A correct multiple-choice answer and a supported scientific contribution demand different evidence. Check whether the score rewards a proposal, an executed experiment, or a completed research target.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Task success · Without expert knowledge; ScienceAgentBench · Without expert knowledge. The source’s task split, model configuration and scoring units are retained.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

  1. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery — published result table ↗

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu et al. · 2024

    All 20 authors

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, Huan Sun

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.