SuperGPQA ↗48.16%
Claude-3-5-sonnet-20241022 · SuperGPQA · Sample-weighted accuracy · reasoning, chat and base-model cohorts ↗
Anthropic / MODEL EVIDENCE
Claude-3-5-sonnet-20241022
7 benchmarks · 47 observations · 8 sources
Scores at a glance
7 benchmarks · large dot: Claude-3-5-sonnet-20241022; small dots: closest models where reported
claude-3-5-sonnet-20241022 · Aider coding benchmarks · Polyglot · percent correct ↗
1 more reported result
84.2% · claude-3-5-sonnet-20241022 · Aider coding benchmarks · Refactoring · percent correct · Aider refactoring benchmark ↗
Chatbot Arena ↗1184 Elo
Claude 3.5 Sonnet (20241022) · Chatbot Arena · Chatbot Arena Vision · English Elo · 2024-11-13 · November 13, 2024 snapshot ↗
Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Disabled oversight · Goal nudging · 300 rollouts · Apollo scheming evaluations ↗
19 more reported results
0.3% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Deleted successor model · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0.7% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Attempted to delete successor model · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0.3% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Copied itself onto other server · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0.7% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Attempted to copy to other server · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0.3% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Replaced successor model · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Removed successor prompt · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Replaced successor prompt · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Covertly produced false ranking · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Manipulated input data · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Overwrote output file · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Disabled oversight · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Deleted successor model · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Attempted to delete successor model · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Copied itself onto other server · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Attempted to copy to other server · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Replaced successor model · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Removed successor prompt · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Replaced successor prompt · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3.5 Sonnet 20241022 · Apollo Scheming & Sandbagging Evals · Covertly produced false ranking · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
claude-3-5-sonnet-20241022 · FrontierMath-2025-02-28-Public · FrontierMath public · 10 questions · source run score · Epoch AI · public February 2025 sample · historical release ↗
claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · 200-challenge full set ↗
16 more reported results
22% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · 200-challenge full set · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
19% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · 200-challenge full set · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
11.5385% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · crypto subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
15.3846% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · crypto subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
20% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · forensics subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
26.6667% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · forensics subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
25% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · misc subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
29.1667% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · misc subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
17.9487% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · pwn subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
20.5128% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · pwn subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
12.8205% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · pwn subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
33.3333% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · rev subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
27.451% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · rev subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
29.4118% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · rev subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
15.7895% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · web subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
5.2632% · claude-3.5-sonnet-20241022 · NYU CTF Bench · pass@1 · web subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
ForecastBench ↗63.2 Brier index
claude-3-5-sonnet-20241022† · Overall difficulty-adjusted Brier index ↗
Closest models by score
Similar positions on shared benchmark leaderboards. Smaller gaps are closer.
There are not enough shared scored settings to identify close comparisons reliably.
Sources
Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.
- Aider code editing benchmark ↗
- Aider refactoring benchmark ↗
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗
All 50 authors
Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha
- Frontier Models are Capable of In-context Scheming — published result table ↗
- Forecastbench — methodology and published results ↗
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
All 13 authors
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines — published result table ↗
All 97 authors
M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, David Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tyshawn Hsing, Ming Xu, Zhenzhu Yang, Zekun Moore Wang, Junting Zhou, Yuelin Bai, Xingyuan Bu, Chenglin Cai, Liang Chen, Yifan Chen, Chengtuo Cheng, Tianhao Cheng, Keyi Ding, Siming Huang, Yun Huang, Yaoru Li, Yizhe Li, Zhaoqun Li, Tianhao Liang, Chengdong Lin, Hongquan Lin, Yinghao Ma, Tianyang Pang, Zhongyuan Peng, Zifan Peng, Qige Qi, Shi Qiu, Xingwei Qu, Shanghaoran Quan, Yizhou Tan, Zili Wang, Chenqing Wang, Hao Wang, Yiya Wang, Yubo Wang, Jiajun Xu, Kexin Yang, Ruibin Yuan, Yuanhao Yue, Tianyang Zhan, Chun Zhang, Jinyang Zhang, Xiyue Zhang, Xingjian Zhang, Yue Zhang, Yongchi Zhao, Xiangyu Zheng, Chenghua Zhong, Yang Gao, Zhoujun Li, Dayiheng Liu, Qian Liu, Tianyu Liu, Shiwen Ni, Junran Peng, Yujia Qin, Wenbo Su, Guoyin Wang, Shi Wang, Jian Yang, Min Yang, Meng Cao, Xiang Yue, Zhaoxiang Zhang, Wangchunshu Zhou, Jiaheng Liu, Qunshu Lin, Wenhao Huang, Ge Zhang
- A Rosetta Stone for AI Benchmarks — released Epoch evaluation runs ↗
Names used in source tables
Claude 3.5 Sonnet (20241022) · Claude 3.5 Sonnet 20241022 · Claude-3-5-sonnet-20241022 · claude-3-5-sonnet-20241022 · claude-3-5-sonnet-20241022† · claude-3.5-sonnet-20241022