o1-preview

Sources

Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.

1. Aider refactoring benchmark ↗

   Paul Gauthier, Aider contributors

2. ARC Prize - Leaderboard ↗

   ARC Prize Foundation

3. CFBench: A Comprehensive Constraints-Following Benchmark for LLMs — published result table ↗

   Tao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang et al. · 2024

   All 14 authors

   Tao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou

4. JudgeBench: A Benchmark for Evaluating LLM-based Judges — published result table ↗

   Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang et al. · 2024

   All 8 authors

   Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica

5. HARP: A challenging human-annotated math reasoning benchmark — published result table ↗

   Yue, Albert S., Madaan, Lovish, Moskovitz, Ted, Strouse, DJ, Singh, Aaditya K. · 2024

6. ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning — published result table ↗

   Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark et al. · 2025

   All 7 authors

   Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, Yejin Choi

7. Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs — published result table ↗

   Gulati, Aryan, Miranda, Brando, Chen, Eric, Xia, Emily, Fronsdal, Kai, Dumont, Bruno et al. · 2025

   All 8 authors

   Gulati, Aryan, Miranda, Brando, Chen, Eric, Xia, Emily, Fronsdal, Kai, Dumont, Bruno, Obbad, Elyas, Koyejo, Sanmi

8. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models ↗

   Zhang, Andy K., Perry, Neil, Dulepet, Riya, Ji, Joey, Menders, Celeste, Lin, Justin W. et al. · 2024

   All 27 authors

   Zhang, Andy K., Perry, Neil, Dulepet, Riya, Ji, Joey, Menders, Celeste, Lin, Justin W., Jones, Eliot, Hussein, Gashon, Liu, Samantha, Jasper, Donovan, Peetathawatchai, Pura, Glenn, Ari, Sivashankar, Vikram, Zamoshchin, Daniel, Glikbarg, Leo, Askaryar, Derek, Yang, Mike, Zhang, Teddy, Alluri, Rishi, Tran, Nathan, Sangpisit, Rinnara, Yiorkadjis, Polycarpos, Osele, Kenny, Raghupathi, Gautham, Boneh, Dan, Ho, Daniel E., Liang, Percy

9. GPQA Diamond ↗

   Epoch AI

10. MATH Level 5 ↗

    Epoch AI

11. OTIS Mock AIME 2024-2025 ↗

    Epoch AI

12. SimpleBench ↗

    Epoch AI

13. AI benchmarks & capabilities ↗

    Epoch AI

14. WeirdML ↗

    Håvard Tveit Ihle

15. Measuring AI Ability to Complete Long Software Tasks ↗

    METR

16. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models — published result table ↗

    Gao, Bofei, Song, Feifan, Yang, Zhe, Cai, Zefan, Miao, Yibo, Dong, Qingxiu et al. · 2024

    All 20 authors

    Gao, Bofei, Song, Feifan, Yang, Zhe, Cai, Zefan, Miao, Yibo, Dong, Qingxiu, Li, Lei, Ma, Chenghao, Chen, Liang, Xu, Runxin, Tang, Zhengyang, Wang, Benyou, Zan, Daoguang, Quan, Shanghaoran, Zhang, Ge, Sha, Lei, Zhang, Yichang, Ren, Xuancheng, Liu, Tianyu, Chang, Baobao

17. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change — published result table ↗

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, Subbarao Kambhampati · 2022

18. Simple Evals — model benchmark results ↗

    OpenAI · 2025

19. BioLP-bench: Measuring understanding of biological lab protocols by large language models ↗

    Igor Ivanov · 2024