claude-3.5-sonnet-20240620 · Aider coding benchmarks · Refactoring · percent correct ↗
Anthropic / MODEL EVIDENCE
claude-3-5-sonnet-20240620
7 benchmarks · 47 observations · 7 sources
Scores at a glance
7 benchmarks · large dot: claude-3-5-sonnet-20240620; small dots: closest models where reported
Chatbot Arena ↗1187 Elo
Claude 3.5 Sonnet (20240620) · Chatbot Arena · Chatbot Arena Vision · English Elo · 2024-11-13 · November 13, 2024 snapshot ↗
Claude 3.5 Sonnet (20240620) · MMLU-Redux (2.0) · Virology · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison ↗
4 more reported results
96% · Claude 3.5 Sonnet (20240620) · MMLU-Redux (2.0) · Logical Fallacies · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
73% · Claude 3.5 Sonnet (20240620) · MMLU-Redux (2.0) · College Chemistry · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
77% · Claude 3.5 Sonnet (20240620) · MMLU-Redux (2.0) · Professional Law · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
98% · Claude 3.5 Sonnet (20240620) · MMLU-Redux (2.0) · Human Sexuality · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
claude-3-5-sonnet-20240620 · FrontierMath-2025-02-28-Public · FrontierMath public · 10 questions · source run score · Epoch AI · public February 2025 sample · historical release ↗
NYU CTF Bench ↗13.5%
claude-3.5-sonnet-20240620 · NYU CTF Bench · pass@1 · 200-challenge full set ↗
6 more reported results
7.6923% · claude-3.5-sonnet-20240620 · NYU CTF Bench · pass@1 · crypto subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
20% · claude-3.5-sonnet-20240620 · NYU CTF Bench · pass@1 · forensics subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
16.6667% · claude-3.5-sonnet-20240620 · NYU CTF Bench · pass@1 · misc subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
17.9487% · claude-3.5-sonnet-20240620 · NYU CTF Bench · pass@1 · pwn subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
17.6471% · claude-3.5-sonnet-20240620 · NYU CTF Bench · pass@1 · rev subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
0% · claude-3.5-sonnet-20240620 · NYU CTF Bench · pass@1 · web subset · NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
ForecastBench ↗63.1 Brier index
claude-3-5-sonnet-20240620† · Overall difficulty-adjusted Brier index ↗
claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · LitQA2 ↗
32 more reported results
2% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SuppQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
46% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · FigQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
83% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · TableQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
10.1% · claude-3-5-sonnet-20240620 · LAB-Bench · DbQA · subtask macro average · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
48% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · ProtocolQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
51.5333% · claude-3-5-sonnet-20240620 · LAB-Bench · SeqQA · subtask macro average · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
28% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · CloningScenarios · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_dga_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_gene_location_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_mirna_targets_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
55% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_mouse_tumor_gene_sets-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
14% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_oncogenic_signatures_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_tfbs_GTRD_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
2% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_variant_from_sequence_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
9% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_variant_multi_sequence_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
21% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_vax_response_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · DbQA_viral_ppi_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
7% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-AAid-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
87% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-AAseq-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-numlen-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
87% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-transeff-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
57% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-enzprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
61% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-gibshindprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
65% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-gibssmaprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
73% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-geneprimers-enz-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
29% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-len-primers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
35% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-primers-len-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
94% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-seq-enzprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
97% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-seq-primers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
59% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_Prop-seq-gcpercent-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
12% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_RE-seq-lenfrags-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
10% · claude-3-5-sonnet-20240620 · LAB-Bench (v1) · Accuracy · SeqQA_RE-seq-numfrags-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
Closest models by score
Similar positions on shared benchmark leaderboards. Smaller gaps are closer.
There are not enough shared scored settings to identify close comparisons reliably.
Sources
Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.
- Aider refactoring benchmark ↗
- Are We Done with MMLU? — published result table ↗
All 16 authors
Gema, Aryo Pradipta, Leang, Joshua Ong Jun, Hong, Giwon, Devoto, Alessio, Mancino, Alberto Carlo Maria, Saxena, Rohit, He, Xuanli, Zhao, Yu, Du, Xiaotang, Madani, Mohammad Reza Ghasemi, Barale, Claire, McHardy, Robert, Harris, Joshua, Kaddour, Jean, van Krieken, Emile, Minervini, Pasquale
- LAB-Bench: Measuring Capabilities of Language Models for Biology Research — published result table ↗
All 9 authors
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, Samuel G. Rodriques
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗
All 50 authors
Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha
- Forecastbench — methodology and published results ↗
- NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
All 13 authors
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique
- A Rosetta Stone for AI Benchmarks — released Epoch evaluation runs ↗
Names used in source tables
Claude 3.5 Sonnet (20240620) · claude-3-5-sonnet-20240620 · claude-3-5-sonnet-20240620† · claude-3.5-sonnet-20240620