RLTasksby ulam.ai
Models / OpenAI

OpenAI / MODEL EVIDENCE

o1

44 benchmarks · 71 observations · 44 sources

Scores at a glance

44 benchmarks · large dot: o1; small dots: closest models where reported

LMCA ↗22.27%
2.7868.26

o1 (high) · LMCA · Score · % ↗

SimpleQA ↗42.6%
7.662.5

o1 · SimpleQA · Correct answers · OpenAI Simple Evals · archived July 2025 ↗

SWE-Lancer ↗9.3%
9.39.3

o1 · SWE-Lancer · IC SWE (Diamond) · pass@1 · original evaluation ↗

8 more reported results

15.6% · o1 · SWE-Lancer · IC SWE (Diamond) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

16.5% · o1 · SWE-Lancer · IC SWE (Diamond) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

13.1% · o1 · SWE-Lancer · IC SWE (Diamond) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

20.3% · o1 · SWE-Lancer · IC SWE (Full) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

41.5% · o1 · SWE-Lancer · SWE Manager (Diamond) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

46.3% · o1 · SWE-Lancer · SWE Manager (Full) · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

29.7% · o1 · SWE-Lancer · SWE-Lancer Diamond · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

32.9% · o1 · SWE-Lancer · SWE-Lancer Full · pass@1 · original evaluation · SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? ↗

Closest models by score

Similar positions on shared benchmark leaderboards. Smaller gaps are closer.

How similarity is measured

Mean absolute gap in percentile positions on shared benchmark settings; lower is closer. At least 60% overlap is required, with a minimum of three when available. Independent evaluations are used when there are at least three; otherwise published report comparisons are used. Repeated model observations within one setting are averaged before ranking. Writing-style traits and composite indices are excluded. This describes similarity, not an overall quality ranking.

Filter comparisons to this model →

Sources

Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.

  1. Aider LLM Leaderboards — Polyglot coding benchmark ↗

    Aider

  2. Aider refactoring benchmark ↗

    Paul Gauthier, Aider contributors

  3. ARC Prize - Leaderboard ↗

    ARC Prize Foundation

  4. MATH-500 Benchmark | Artificial Analysis ↗

    Artificial Analysis

  5. SpatialViz-Bench: A Cognitively-Grounded Benchmark for Diagnosing Spatial Visualization in MLLMs ↗

    Wang, Siting, Pei, Minnan, Sun, Luoyang, Deng, Cheng, Li, Yuchen, Shao, Kun et al. · 2025

    All 9 authors

    Wang, Siting, Pei, Minnan, Sun, Luoyang, Deng, Cheng, Li, Yuchen, Shao, Kun, Tian, Zheng, Zhang, Haifeng, Wang, Jun

  6. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding — published result table ↗

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua et al. · 2025

    All 9 authors

    Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, Bowen Zhou

  7. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — published result table ↗

    Miserendino, Samuel, Wang, Michele, Patwardhan, Tejal, Heidecke, Johannes · 2025

  8. HealthBench: Evaluating Large Language Models Towards Improved Human Health — published result table ↗

    Arora, Rahul K., Wei, Jason, Hicks, Rebecca Soskin, Bowman, Preston, Quiñonero-Candela, Joaquin, Tsimpourlas, Foivos et al. · 2025

    All 12 authors

    Arora, Rahul K., Wei, Jason, Hicks, Rebecca Soskin, Bowman, Preston, Quiñonero-Candela, Joaquin, Tsimpourlas, Foivos, Sharman, Michael, Shah, Meghan, Vallone, Andrea, Beutel, Alex, Heidecke, Johannes, Singhal, Karan

  9. TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts — published result table ↗

    Jaffe, Oliver, Sherburn, Dane · 2026

  10. Are large language models superhuman chemists? — published result table ↗

    Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan et al. · 2024

    All 35 authors

    Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, Amir Mohammad Elahi, Mehrdad Asgari, Juliane Eberhardt, Hani M. Elbeheiry, María Victoria Gil, Maximilian Greiner, Caroline T. Holick, Christina Glaubitz, Tim Hoffmann, Abdelrahman Ibrahim, Lea C. Klepsch, Yannik Köster, Fabian Alexander Kreth, Jakob Meyer, Santiago Miret, Jan Matthias Peschel, Michael Ringleb, Nicole Roesner, Johanna Schreiber, Ulrich S. Schubert, Leanne M. Stafast, Dinga Wonanke, Michael Pieler, Philippe Schwaller, Kevin Maik Jablonka

  11. DTBench — Conceptual Reasoning Index ↗

    Conceptual Reasoning Index

  12. LMCA — Conceptual Reasoning Index ↗

    Conceptual Reasoning Index

  13. Chess Puzzles ↗

    Epoch AI

  14. FrontierMath Tiers 1-3 (v1, old) ↗

    Epoch AI

  15. FrontierMath Tiers 1-3 (v2) ↗

    Epoch AI

  16. GPQA Diamond ↗

    Epoch AI

  17. Humanity's Last Exam ↗

    Epoch AI

  18. MATH Level 5 ↗

    Epoch AI

  19. OTIS Mock AIME 2024-2025 ↗

    Epoch AI

  20. SimpleQA Verified ↗

    Epoch AI

  21. SimpleBench ↗

    Epoch AI

  22. Capabilities & benchmarking — retained external evaluation records ↗

    Epoch AI · 2026

  23. AI benchmarks & capabilities ↗

    Epoch AI

  24. Fiction.liveBench — March 14, 2025 benchmark release ↗

    Fiction.live · 2025

  25. GeoBench — GeoGuessr model benchmark ↗

    GeoBench

  26. Writing benchmark — incorporating mandatory story elements ↗

    lechmazur

  27. WeirdML ↗

    Håvard Tveit Ihle

  28. EnigmaEval ↗

    Scale Labs / SEAL

  29. Scale Labs — evaluation methodology and results ↗

    Scale Labs / SEAL · 2026

  30. LiveBench — model evaluation leaderboard ↗

    LiveBench Team

  31. Measuring AI Ability to Complete Long Software Tasks ↗

    METR

  32. BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese — published result table ↗

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye et al. · 2025

    All 16 authors

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, Yining Hua

  33. Simple Evals — model benchmark results ↗

    OpenAI · 2025

  34. CadEval Dashboard — run 20250422_095709 ↗

    willpatrick · 2025

  35. AIME — evaluation methodology and results ↗

    Vals AI · 2026

  36. GPQA Diamond — evaluation methodology and results ↗

    Vals AI · 2026

  37. LiveCodeBench — evaluation methodology and results ↗

    Vals AI · 2026

  38. LegalBench — evaluation methodology and results ↗

    Vals AI · 2026

  39. MATH 500 — evaluation methodology and results ↗

    Vals AI · 2026

  40. MedQA — evaluation methodology and results ↗

    Vals AI · 2026

  41. MGSM — evaluation methodology and results ↗

    Vals AI · 2026

  42. MMLU Pro — evaluation methodology and results ↗

    Vals AI · 2026

  43. MMMU Pro — evaluation methodology and results ↗

    Vals AI · 2026

  44. TaxEval — evaluation methodology and results ↗

    Vals AI · 2026

Release dates reported by sources

2024-12-05 · 2024-12-17

Names used in source tables

O1 · o1 · o1 (high) · o1 (low) · o1 (medium) · o1 (unknown thinking) · o1‡