RLTasksby ulam.ai
Models / Anthropic

Anthropic / MODEL EVIDENCE

Claude 3.7 Sonnet

45 benchmarks · 130 observations · 48 sources · released 2025-02-24

Scores at a glance

45 benchmarks · large dot: Claude 3.7 Sonnet; small dots: closest models where reported

GPQA Diamond ↗66.04%
66.0466.04

Claude 3.7 Sonnet · GPQA diamond · Best score (across scorers) · % ↗

5 more reported results

79.73% · Claude 3.7 Sonnet (64k thinking) · GPQA diamond · Best score (across scorers) · % · Epoch AI · GPQA diamond ↗

76.77% · Claude 3.7 Sonnet (32k thinking) · GPQA diamond · Best score (across scorers) · % · Epoch AI · GPQA diamond ↗

76.77% · Claude 3.7 Sonnet (16k thinking) · GPQA diamond · Best score (across scorers) · % · Epoch AI · GPQA diamond ↗

75.25 % · Claude 3.7 Sonnet (Thinking) · GPQA Diamond · Accuracy · Vals AI · GPQA Diamond ↗

67.42 % · Claude 3.7 Sonnet (Nonthinking) · GPQA Diamond · Accuracy · Vals AI · GPQA Diamond ↗

FrontierMath Tiers 1-3 (v1, old) ↗3.1%
3.13.1

Claude 3.7 Sonnet · FrontierMath-2025-02-28-Private · Best score (across scorers) · % ↗

3 more reported results

4.14% · Claude 3.7 Sonnet (16k thinking) · FrontierMath-2025-02-28-Private · Best score (across scorers) · % · Epoch AI · FrontierMath-2025-02-28-Private ↗

3.45% · Claude 3.7 Sonnet (32k thinking) · FrontierMath-2025-02-28-Private · Best score (across scorers) · % · Epoch AI · FrontierMath-2025-02-28-Private ↗

3.1% · Claude 3.7 Sonnet (64k thinking) · FrontierMath-2025-02-28-Private · Best score (across scorers) · % · Epoch AI · FrontierMath-2025-02-28-Private ↗

MATH Level 5 ↗68.18%
68.1868.18

Claude 3.7 Sonnet · MATH level 5 · Best score (across scorers) · % ↗

3 more reported results

91.16% · Claude 3.7 Sonnet (64k thinking) · MATH level 5 · Best score (across scorers) · % · Epoch AI · MATH level 5 ↗

90.03% · Claude 3.7 Sonnet (32k thinking) · MATH level 5 · Best score (across scorers) · % · Epoch AI · MATH level 5 ↗

86.25% · Claude 3.7 Sonnet (16k thinking) · MATH level 5 · Best score (across scorers) · % · Epoch AI · MATH level 5 ↗

OTIS Mock AIME 2024-2025 ↗21.94%
21.9421.94

Claude 3.7 Sonnet · OTIS Mock AIME 2024-2025 · Best score (across scorers) · % ↗

3 more reported results

57.78% · Claude 3.7 Sonnet (64k thinking) · OTIS Mock AIME 2024-2025 · Best score (across scorers) · % · Epoch AI · OTIS Mock AIME 2024-2025 ↗

53.33% · Claude 3.7 Sonnet (32k thinking) · OTIS Mock AIME 2024-2025 · Best score (across scorers) · % · Epoch AI · OTIS Mock AIME 2024-2025 ↗

46.67% · Claude 3.7 Sonnet (16k thinking) · OTIS Mock AIME 2024-2025 · Best score (across scorers) · % · Epoch AI · OTIS Mock AIME 2024-2025 ↗

CadEval ↗54%
1274

Claude 3.7 Sonnet · CadEval · Overall pass (%) · % ↗

Cybench ↗20%
555

Claude 3.7 Sonnet · Cybench · Unguided % Solved · % ↗

GeoBench ↗68%
6868

Claude 3.7 Sonnet · GeoBench · ACW Country % · % ↗

1 more reported result

65% · Claude 3.7 Sonnet (15k thinking) · GeoBench · ACW Country % · % · Epoch AI · GeoBench ↗

GSO ↗3.8%
026.5

Claude 3.7 Sonnet · GSO-Bench · Score OPT@1 · % ↗

1 more reported result

3.8% · Claude 3.7 Sonnet (unknown thinking) · GSO-Bench · Score OPT@1 · % · Epoch AI · GSO-Bench ↗

VPCT ↗39%
3939

Claude 3.7 Sonnet · VPCT · Correct · % ↗

1 more reported result

35% · Claude 3.7 Sonnet (64k thinking) · VPCT · Correct · % · Epoch AI · VPCT ↗

METR task performance & time horizons ↗55.82%
27.1878.86

Claude 3.7 Sonnet · METR task performance & time horizons · Average task accuracy ↗

3 more reported results

60.38894 minutes · Claude 3.7 Sonnet · METR task performance & time horizons · 50% success task horizon · METR-Horizon-v1.1 · Epoch AI · METR Autonomy / Task-Horizon Suite · public result data ↗

59.97% · Claude 3.7 Sonnet (16k thinking) · METR task performance & time horizons · Average task accuracy · Epoch AI · METR Time Horizons ↗

56.09162 minutes · Claude 3.7 Sonnet (16k thinking) · METR task performance & time horizons · 50% success task horizon · Epoch AI · METR Autonomy / Task-Horizon Suite · public result data ↗

SpatialViz-Bench ↗33.9%
31.7844.66

Claude 3.7 Sonnet (unknown thinking) · SpatialViz-Bench · Overall score · % ↗

CorpFin v2 ↗60.41 %
28.6373.19

Claude 3.7 Sonnet (Thinking) · CorpFin · Accuracy ↗

MATH-500 ↗91.6 %
91.691.6

Claude 3.7 Sonnet (Thinking) · MATH 500 · Accuracy ↗

1 more reported result

76.8 % · Claude 3.7 Sonnet (Nonthinking) · MATH 500 · Accuracy · Vals AI · MATH 500 ↗

MMLU-Pro ↗82.73 %
82.7382.73

Claude 3.7 Sonnet (Thinking) · MMLU Pro · Accuracy ↗

1 more reported result

80.66 % · Claude 3.7 Sonnet (Nonthinking) · MMLU Pro · Accuracy · Vals AI · MMLU Pro ↗

MMMU-Pro ↗75.1 %
75.175.1

Claude 3.7 Sonnet (Thinking) · MMMU Pro · Accuracy ↗

1 more reported result

71.52 % · Claude 3.7 Sonnet (Nonthinking) · MMMU Pro · Accuracy · Vals AI · MMMU Pro ↗

Fortress ↗38.01%
5.7874.39

Claude 3.7 Sonnet · Scale Labs · Risk elicitation score ↗

PolyMath ↗11%
5.740.3

Claude-3.7-sonnet† · PolyMath · PolyMath-Top · Mean accuracy across the published languages ↗

3 more reported results

15% · Claude-3.7-sonnet† · PolyMath · PolyMath-High · Mean accuracy across the published languages · PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts ↗

28.3% · Claude-3.7-sonnet† · PolyMath · PolyMath-Medium · Mean accuracy across the published languages · PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts ↗

90.9% · Claude-3.7-sonnet† · PolyMath · PolyMath-Low · Mean accuracy across the published languages · PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts ↗

SWE-bench Live ↗17.67%
717.67

Claude 3.7 Sonnet · SWE-bench Live · Resolved tasks · Lite comparison ↗

1 more reported result

11.33% · Claude 3.7 Sonnet · SWE-bench Live · Resolved tasks · Lite comparison · SWE-bench Goes Live! ↗

Multi-SWE-bench ↗44.6%
26.248.2

Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Python · language-specific results ↗

22 more reported results

45.8% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Python · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

52.2% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Python · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

14.06% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Java · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

23.44% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Java · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

21.88% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Java · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

3.57% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · TypeScript · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

11.16% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · TypeScript · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

2.23% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · TypeScript · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

1.97% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · JavaScript · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

4.78% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · JavaScript · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

5.06% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · JavaScript · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

5.84% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Go · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

5.37% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Go · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

7.48% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Go · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

5.44% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Rust · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

6.69% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Rust · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

15.9% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · Rust · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

2.34% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · C · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

8.59% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · C · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

3.1% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · C++ · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

11.63% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · C++ · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

14.73% · Claude-3.7-Sonnet · Multi-SWE-bench · Resolved · C++ · language-specific results · Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving ↗

BrowseComp-ZH ↗17.7%
4.842.9

Claude-3.7-Sonnet · BrowseComp-ZH · Answer accuracy · original comparison ↗

HealthBench Consensus ↗88.14%
75.0993.98

Claude 3.7 Sonnet (extended thinking) · HealthBench Consensus · Overall criterion score · original comparison ↗

HealthBench Hard ↗2%
032

Claude 3.7 Sonnet (extended thinking) · HealthBench Hard · Weighted rubric score · original model comparison ↗

LiveCodeBench Pro ↗992 Elo
1999

Claude 3.7 Sonnet (Max Reasoning) · LiveCodeBench Pro · Elo rating · human contest calibration ↗

3 more reported results

0% · Claude 3.7 Sonnet (Max Reasoning) · LiveCodeBench Pro · Hard · pass@1 · difficulty breakdown · LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? ↗

1.4% · Claude 3.7 Sonnet (Max Reasoning) · LiveCodeBench Pro · Medium · pass@1 · difficulty breakdown · LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? ↗

36.6% · Claude 3.7 Sonnet (Max Reasoning) · LiveCodeBench Pro · Easy · pass@1 · difficulty breakdown · LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? ↗

CyberSecEval ↗33.51%
22.9933.51

Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · malware analysis · Overall accuracy · CyberSecEval 4 · CyberSOCEval defensive-analysis comparison ↗

8 more reported results

37% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · malware analysis · Easy accuracy · CyberSecEval 4 · CyberSOCEval defensive-analysis comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

23% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · malware analysis · Medium accuracy · CyberSecEval 4 · CyberSOCEval defensive-analysis comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

24% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · malware analysis · Hard accuracy · CyberSecEval 4 · CyberSOCEval defensive-analysis comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

49.25% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · threat intelligence · image input · Overall accuracy · CyberSecEval 4 · CyberSOCEval image-input comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

68% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · threat intelligence · image input · NSA reports · CyberSecEval 4 · CyberSOCEval image-input comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

53% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · threat intelligence · image input · CrowdStrike reports · CyberSecEval 4 · CyberSOCEval image-input comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

42% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · threat intelligence · image input · IC3 reports · CyberSecEval 4 · CyberSOCEval image-input comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

44% · Claude 3.7 Sonnet · CyberSecEval · CyberSOCEval · threat intelligence · image input · CISA reports · CyberSecEval 4 · CyberSOCEval image-input comparison · CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning ↗

Closest models by score

Similar positions on shared benchmark leaderboards. Smaller gaps are closer.

11.4 percentile-point gap

4 of 4 shared benchmark comparisons

14.0 percentile-point gap

3 of 4 shared benchmark comparisons

How similarity is measured

Mean absolute gap in percentile positions on shared benchmark settings; lower is closer. At least 60% overlap is required, with a minimum of three when available. Independent evaluations are used when there are at least three; otherwise published report comparisons are used. Repeated model observations within one setting are averaged before ranking. Writing-style traits and composite indices are excluded. This describes similarity, not an overall quality ranking.

Filter comparisons to this model →

Sources

Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.

  1. Aider LLM Leaderboards — Polyglot coding benchmark ↗

    Aider

  2. ARC Prize - Leaderboard ↗

    ARC Prize Foundation

  3. SpatialViz-Bench: A Cognitively-Grounded Benchmark for Diagnosing Spatial Visualization in MLLMs ↗

    Wang, Siting, Pei, Minnan, Sun, Luoyang, Deng, Cheng, Li, Yuchen, Shao, Kun et al. · 2025

    All 9 authors

    Wang, Siting, Pei, Minnan, Sun, Luoyang, Deng, Cheng, Li, Yuchen, Shao, Kun, Tian, Zheng, Zhang, Haifeng, Wang, Jun

  4. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving — published result table ↗

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin et al. · 2025

    All 19 authors

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, Liang Xiang

  5. PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts — published result table ↗

    Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang et al. · 2025

    All 16 authors

    Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, Jingren Zhou

  6. HealthBench: Evaluating Large Language Models Towards Improved Human Health — published result table ↗

    Arora, Rahul K., Wei, Jason, Hicks, Rebecca Soskin, Bowman, Preston, Quiñonero-Candela, Joaquin, Tsimpourlas, Foivos et al. · 2025

    All 12 authors

    Arora, Rahul K., Wei, Jason, Hicks, Rebecca Soskin, Bowman, Preston, Quiñonero-Candela, Joaquin, Tsimpourlas, Foivos, Sharman, Michael, Shah, Meghan, Vallone, Andrea, Beutel, Alex, Heidecke, Johannes, Singhal, Karan

  7. SWE-bench Goes Live! — published result table ↗

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie et al. · 2025

    All 15 authors

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang

  8. LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? — published result table ↗

    Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He et al. · 2025

    All 19 authors

    Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, Saining Xie

  9. CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning — published result table ↗

    Deason, Lauren, Bali, Adam, Bejean, Ciprian, Bolocan, Diana, Crnkovich, James, Croitoru, Ioana et al. · 2025

    All 23 authors

    Deason, Lauren, Bali, Adam, Bejean, Ciprian, Bolocan, Diana, Crnkovich, James, Croitoru, Ioana, Durai, Krishna, Midler, Chase, Miron, Calin, Molnar, David, Moon, Brad, Ostarcevic, Bruno, Peltea, Alberto, Rosenberg, Matt, Sandu, Catalin, Saputkin, Arthur, Shah, Sagar, Stan, Daniel, Szocs, Ernest, Wan, Shengye, Whitman, Spencer, Krasser, Sven, Saxe, Joshua

  10. CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers — published result table ↗

    Pan, Haining, Roggeveen, James V., Berg, Erez, Carrasquilla, Juan, Chowdhury, Debanjan, Ganguli, Surya et al. · 2025

    All 19 authors

    Pan, Haining, Roggeveen, James V., Berg, Erez, Carrasquilla, Juan, Chowdhury, Debanjan, Ganguli, Surya, Ghimenti, Federico, Hasik, Juraj, Hunt, Henry, Jiang, Hong-Chen, Kamb, Mason, Kao, Ying-Jer, Khatami, Ehsan, Lawler, Michael J., Luo, Di, Neupert, Titus, Qi, Xiaoliang, Brenner, Michael P., Kim, Eun-Ah

  11. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models ↗

    Zhang, Andy K., Perry, Neil, Dulepet, Riya, Ji, Joey, Menders, Celeste, Lin, Justin W. et al. · 2024

    All 27 authors

    Zhang, Andy K., Perry, Neil, Dulepet, Riya, Ji, Joey, Menders, Celeste, Lin, Justin W., Jones, Eliot, Hussein, Gashon, Liu, Samantha, Jasper, Donovan, Peetathawatchai, Pura, Glenn, Ari, Sivashankar, Vikram, Zamoshchin, Daniel, Glikbarg, Leo, Askaryar, Derek, Yang, Mike, Zhang, Teddy, Alluri, Rishi, Tran, Nathan, Sangpisit, Rinnara, Yiorkadjis, Polycarpos, Osele, Kenny, Raghupathi, Gautham, Boneh, Dan, Ho, Daniel E., Liang, Percy

  12. FutureSearch Evals ↗

    FutureSearch

  13. ARC-AGI-2 ↗

    Epoch AI

  14. FrontierMath Tiers 1-3 (v1, old) ↗

    Epoch AI

  15. GPQA Diamond ↗

    Epoch AI

  16. Humanity's Last Exam ↗

    Epoch AI

  17. MATH Level 5 ↗

    Epoch AI

  18. METR Time Horizons ↗

    Epoch AI

  19. OTIS Mock AIME 2024-2025 ↗

    Epoch AI

  20. SimpleBench ↗

    Epoch AI

  21. SWE-bench Verified ↗

    Epoch AI

  22. The Agent Company ↗

    Epoch AI

  23. Capabilities & benchmarking — retained external evaluation records ↗

    Epoch AI · 2026

  24. AI benchmarks & capabilities ↗

    Epoch AI

  25. Fiction.liveBench — March 14, 2025 benchmark release ↗

    Fiction.live · 2025

  26. GeoBench — GeoGuessr model benchmark ↗

    GeoBench

  27. Writing benchmark — incorporating mandatory story elements ↗

    lechmazur

  28. GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents ↗

    Shetty, Manish, Jain, Naman, Liu, Jinjian, Kethanaboyina, Vijay, Sen, Koushik, Stoica, Ion · 2025

  29. EnigmaEval ↗

    Scale Labs / SEAL

  30. Scale Labs — evaluation methodology and results ↗

    Scale Labs / SEAL · 2026

  31. LiveBench — model evaluation leaderboard ↗

    LiveBench Team

  32. Measuring AI Ability to Complete Long Software Tasks ↗

    METR

  33. Metr Autonomy Task Horizon Suite — methodology and published results ↗

    METR · 2026

  34. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments ↗

    Xie, Tianbao, Zhang, Danyang, Chen, Jixuan, Li, Xiaochuan, Zhao, Siheng, Cao, Ruisheng et al. · 2024

    All 17 authors

    Xie, Tianbao, Zhang, Danyang, Chen, Jixuan, Li, Xiaochuan, Zhao, Siheng, Cao, Ruisheng, Hua, Toh Jing, Cheng, Zhoujun, Shin, Dongchan, Lei, Fangyu, Liu, Yitao, Xu, Yiheng, Zhou, Shuyan, Savarese, Silvio, Xiong, Caiming, Zhong, Victor, Yu, Tao

  35. BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese — published result table ↗

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye et al. · 2025

    All 16 authors

    Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, Yining Hua

  36. CadEval Dashboard — run 20250422_095709 ↗

    willpatrick · 2025

  37. AIME — evaluation methodology and results ↗

    Vals AI · 2026

  38. CorpFin — evaluation methodology and results ↗

    Vals AI · 2026

  39. GPQA Diamond — evaluation methodology and results ↗

    Vals AI · 2026

  40. LiveCodeBench — evaluation methodology and results ↗

    Vals AI · 2026

  41. LegalBench — evaluation methodology and results ↗

    Vals AI · 2026

  42. MATH 500 — evaluation methodology and results ↗

    Vals AI · 2026

  43. MedQA — evaluation methodology and results ↗

    Vals AI · 2026

  44. MGSM — evaluation methodology and results ↗

    Vals AI · 2026

  45. MMLU Pro — evaluation methodology and results ↗

    Vals AI · 2026

  46. MMMU Pro — evaluation methodology and results ↗

    Vals AI · 2026

  47. MortgageTax — evaluation methodology and results ↗

    Vals AI · 2026

  48. TaxEval — evaluation methodology and results ↗

    Vals AI · 2026

Names used in source tables

Claude 3.7 Sonnet · Claude 3.7 Sonnet (12k thinking) · Claude 3.7 Sonnet (15k thinking) · Claude 3.7 Sonnet (16k thinking) · Claude 3.7 Sonnet (1k thinking) · Claude 3.7 Sonnet (2k thinking) · Claude 3.7 Sonnet (32k thinking) · Claude 3.7 Sonnet (64k thinking) · Claude 3.7 Sonnet (8k thinking) · Claude 3.7 Sonnet (Max Reasoning) · Claude 3.7 Sonnet (Nonthinking) · Claude 3.7 Sonnet (Thinking) · Claude 3.7 Sonnet (extended thinking) · Claude 3.7 Sonnet (unknown thinking) · Claude-3.7-Sonnet · Claude-3.7-sonnet†