claude-3-opus-20240229 · Aider coding benchmarks · Refactoring · percent correct ↗
Anthropic / MODEL EVIDENCE
claude-3-opus-20240229
6 benchmarks · 59 observations · 6 sources
Scores at a glance
6 benchmarks · large dot: claude-3-opus-20240229; small dots: closest models where reported
claude-3-opus-20240229 · RepoQA & LongCodeArena · RepoQA retrieval accuracy · similarity threshold 0.8 ↗
Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Disabled oversight · Goal nudging · 300 rollouts · Apollo scheming evaluations ↗
19 more reported results
6% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Deleted successor model · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
11% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Attempted to delete successor model · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
10% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Copied itself onto other server · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
15% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Attempted to copy to other server · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
5% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Replaced successor model · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
10% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Removed successor prompt · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
6% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Replaced successor prompt · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Covertly produced false ranking · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Manipulated input data · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Overwrote output file · Goal nudging · 300 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0.3% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Disabled oversight · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Deleted successor model · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Attempted to delete successor model · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Copied itself onto other server · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Attempted to copy to other server · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Replaced successor model · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0.3% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Removed successor prompt · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Replaced successor prompt · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
0% · Claude 3 Opus 20240229 · Apollo Scheming & Sandbagging Evals · Covertly produced false ranking · No goal nudging · 400 rollouts · Apollo scheming evaluations · Frontier Models are Capable of In-context Scheming ↗
Claude 3 Opus (20240229) · MMLU-Redux (2.0) · Virology · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison ↗
4 more reported results
96% · Claude 3 Opus (20240229) · MMLU-Redux (2.0) · Logical Fallacies · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
72% · Claude 3 Opus (20240229) · MMLU-Redux (2.0) · College Chemistry · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
72% · Claude 3 Opus (20240229) · MMLU-Redux (2.0) · Professional Law · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
96% · Claude 3 Opus (20240229) · MMLU-Redux (2.0) · Human Sexuality · corrected-instance accuracy · MMLU-Redux 2.0 paper · audited subject comparison · Are We Done with MMLU? ↗
ForecastBench ↗62.4 Brier index
claude-3-opus-20240229† · Overall difficulty-adjusted Brier index ↗
claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · LitQA2 ↗
32 more reported results
2% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SuppQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
24% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · FigQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
67% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · TableQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
6.7% · claude-3-opus-20240229 · LAB-Bench · DbQA · subtask macro average · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
52% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · ProtocolQA · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
44.9333% · claude-3-opus-20240229 · LAB-Bench · SeqQA · subtask macro average · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
27% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · CloningScenarios · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_dga_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_gene_location_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_mirna_targets_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
36% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_mouse_tumor_gene_sets-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
12% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_oncogenic_signatures_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_tfbs_GTRD_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_variant_from_sequence_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
5% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_variant_multi_sequence_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
14% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_vax_response_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
0% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · DbQA_viral_ppi_task-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
3% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-AAid-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
67% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-AAseq-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
7% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-seq-numlen-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
64% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_ORF-transeff-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
75% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-enzprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
48% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-gibshindprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
34% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-gene-gibssmaprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
76% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-geneprimers-enz-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
37% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-len-primers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
29% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-primers-len-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
80% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-seq-enzprimers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
83% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_PCR-seq-primers-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
45% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_Prop-seq-gcpercent-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
13% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_RE-seq-lenfrags-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
13% · claude-3-opus-20240229 · LAB-Bench (v1) · Accuracy · SeqQA_RE-seq-numfrags-v1 · LAB-Bench: Measuring Capabilities of Language Models for Biology Research ↗
Closest models by score
Similar positions on shared benchmark leaderboards. Smaller gaps are closer.
There are not enough shared scored settings to identify close comparisons reliably.
Sources
Every plotted observation links to its evaluation. Reported configurations, benchmark versions and task subsets can differ; similarity describes shared scores, not training lift.
- Aider refactoring benchmark ↗
- Are We Done with MMLU? — published result table ↗
All 16 authors
Gema, Aryo Pradipta, Leang, Joshua Ong Jun, Hong, Giwon, Devoto, Alessio, Mancino, Alberto Carlo Maria, Saxena, Rohit, He, Xuanli, Zhao, Yu, Du, Xiaotang, Madani, Mohammad Reza Ghasemi, Barale, Claire, McHardy, Robert, Harris, Joshua, Kaddour, Jean, van Krieken, Emile, Minervini, Pasquale
- RepoQA: Evaluating Long Context Code Understanding — published result table ↗
All 8 authors
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, Lingming Zhang
- LAB-Bench: Measuring Capabilities of Language Models for Biology Research — published result table ↗
All 9 authors
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, Samuel G. Rodriques
- Frontier Models are Capable of In-context Scheming — published result table ↗
- Forecastbench — methodology and published results ↗
Names used in source tables
Claude 3 Opus (20240229) · Claude 3 Opus 20240229 · claude-3-opus-20240229 · claude-3-opus-20240229†