NYU CTF Bench

Published agent submissions are evaluated on 200 cybersecurity competition challenges across six categories. Compare the full-set result with category breakdowns while preserving the agent and attempt budget.26 Oct 2026

Sources

NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner et al. · 2024
All 13 authors
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique
Dates & evaluation details
NYU CTF Bench · pass@1 · 200-challenge full set · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@1 · total; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@1 · crypto subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@1 · crypto; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@1 · forensics subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@1 · forensics; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@1 · misc subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@1 · misc; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@1 · pwn subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@1 · pwn; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@1 · rev subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@1 · rev; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@1 · web subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@1 · web; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@5 · 200-challenge full set · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@5 · total; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@5 · crypto subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@5 · crypto; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@5 · forensics subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@5 · forensics; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@5 · misc subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@5 · misc; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@5 · pwn subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@5 · pwn; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@5 · rev subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@5 · rev; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench · pass@5 · web subset · 2026-10-27
Published model comparison, 2024. NYU CTF Bench · official submitted results. Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported. Official leaderboard JSON · pass@5 · web; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
NYU CTF Bench — Cotool evaluation, February 2026 ↗
Cotool · 2026
Dates & evaluation details
NYU CTF Bench · Cotool 81-challenge subset · accuracy · 2026-10-27
Published model comparison, 2026. Cotool · February 2026 historical evaluation. Independent provider evaluation on 81 selected challenges spanning reverse engineering, miscellaneous tasks and forensics. This is not the official 200-challenge benchmark cohort. Cotool has deprecated this public subset after observing evidence of model memorization, so the rows are retained as historical measurements with that limitation. Only aggregate accuracy is imported. Published result table; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

Reading the measurement

Look beyond a plausible code sample to the executable artifact. Tests, repository context, language coverage and time limits shape the result. Separate first-attempt correctness from success after debugging, retries or agent assistance.

Published evaluation

Published solved-challenge counts divided by the official full-set or category denominator. Pass@1 and pass@5 remain different measures, and agent systems and submission dates remain visible. This is NYU CTF Bench; CyBench is a different benchmark with its own page. Only aggregate scores are imported.

Versions and evaluation settings

The imported official 200-challenge results and historical Cotool subset are NYU CTF Bench. Rename this page; Cybench is a different benchmark and remains separate.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security — official leaderboard ↗
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner et al. · 2024
All 13 authors
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, Muhammad Shafique
NYU CTF Bench — Cotool evaluation, February 2026 ↗
Cotool · 2026

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All code & engineering benchmarks →

		Evidence	What the result shows
Claude Opus 4.6 →Anthropic	79% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	Leads this displayed comparison. Source notes Source cell: 79.0%
GPT-5.2 →OpenAI	63% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#2 of 10 in this displayed comparison; 16 pp below the leader. Source notes Source cell: 63.0%
GPT-5.2 Codex →OpenAI	61.7% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#3 of 10 in this displayed comparison; 17.3 pp below the leader.
Gemini 3.0 Flash →Google	54.3% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#4 of 10 in this displayed comparison; 24.7 pp below the leader.
Gemini 3.0 Pro →Google	45.7% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#5 of 10 in this displayed comparison; 33.3 pp below the leader.
Claude Sonnet 4.5 →Anthropic	27.2% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#6 of 10 in this displayed comparison; 51.8 pp below the leader.
Claude Haiku 4.5 →Anthropic	23.5% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#7 of 10 in this displayed comparison; 55.5 pp below the leader.
GLM-4.7 →Z.ai	19.8% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#8 of 10 in this displayed comparison; 59.2 pp below the leader.
MiniMax M2.1 →MiniMax	12.3% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#9 of 10 in this displayed comparison; 66.7 pp below the leader.
Qwen3 235B →Qwen	11.1% 1 report	NYU CTF Bench · NYU CTF · Cotool 81-challenge subset · accuracy[2]Cotool · February 2026 historical evaluation	#10 of 10 in this displayed comparison; 67.9 pp below the leader.
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 44/200; Submission date: 2025/08/12	22% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 44/200; Submission date: 2025/08/12	Leads this displayed comparison. Source notes Source cell: 22.0
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 42/200; Submission date: 2025/08/12	21% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 42/200; Submission date: 2025/08/12	#2 of 12 in this displayed comparison; 1 pp below the leader. Source notes Source cell: 21.0
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 38/200; Submission date: 2025/03/01	19% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 38/200; Submission date: 2025/03/01	#3 of 12 in this displayed comparison; 3 pp below the leader. Source notes Source cell: 19.0
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 37/200; Submission date: 2025/08/12	18.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 37/200; Submission date: 2025/08/12	#4 of 12 in this displayed comparison; 3.5 pp below the leader. Source notes Source cell: 18.5
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 35/200; Submission date: 2025/08/12	17.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 35/200; Submission date: 2025/08/12	#5 of 12 in this displayed comparison; 4.5 pp below the leader. Source notes Source cell: 17.5
claude-3.5-sonnet-20240620 →Anthropic	13.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 27/200; Submission date: 2024/10/02	#6 of 12 in this displayed comparison; 8.5 pp below the leader. Source notes Source cell: 13.5
GPT-4.1 →OpenAINYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 27/200; Submission date: 2025/08/12	13.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 27/200; Submission date: 2025/08/12	#6 of 12 in this displayed comparison; 8.5 pp below the leader. Source notes Source cell: 13.5
GPT-4.1 →OpenAINYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 23/200; Submission date: 2025/08/12	11.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 23/200; Submission date: 2025/08/12	#8 of 12 in this displayed comparison; 10.5 pp below the leader. Source notes Source cell: 11.5
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 23/200; Submission date: 2025/08/12	11.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 23/200; Submission date: 2025/08/12	#8 of 12 in this displayed comparison; 10.5 pp below the leader. Source notes Source cell: 11.5
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 21/200; Submission date: 2025/03/01	10.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 21/200; Submission date: 2025/03/01	#10 of 12 in this displayed comparison; 11.5 pp below the leader. Source notes Source cell: 10.5
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 19/200; Submission date: 2024/10/02	9.5% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 19/200; Submission date: 2024/10/02	#11 of 12 in this displayed comparison; 12.5 pp below the leader. Source notes Source cell: 9.5
GPT-4-1106-preview →OpenAI	7% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · 200-challenge full set[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 14/200; Submission date: 2024/10/02	#12 of 12 in this displayed comparison; 15 pp below the leader. Source notes Source cell: 7.0
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 8/52; Submission date: 2025/08/12	15.38% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 8/52; Submission date: 2025/08/12	Leads this displayed comparison (tied). Source notes Source cell: 15.384615384615385
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 8/52; Submission date: 2025/03/01	15.38% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 8/52; Submission date: 2025/03/01	Leads this displayed comparison (tied). Source notes Source cell: 15.384615384615385
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 7/52; Submission date: 2025/08/12	13.46% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 7/52; Submission date: 2025/08/12	#3 of 12 in this displayed comparison; 1.92 pp below the leader. Source notes Source cell: 13.461538461538462
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 6/52; Submission date: 2025/08/12	11.54% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 6/52; Submission date: 2025/08/12	#4 of 12 in this displayed comparison; 3.85 pp below the leader. Source notes Source cell: 11.538461538461538
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 6/52; Submission date: 2025/08/12	11.54% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 6/52; Submission date: 2025/08/12	#4 of 12 in this displayed comparison; 3.85 pp below the leader. Source notes Source cell: 11.538461538461538
GPT-4.1 →OpenAINYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 5/52; Submission date: 2025/08/12	9.62% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 5/52; Submission date: 2025/08/12	#6 of 12 in this displayed comparison; 5.77 pp below the leader. Source notes Source cell: 9.615384615384615
claude-3.5-sonnet-20240620 →Anthropic	7.69% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 4/52; Submission date: 2024/10/02	#7 of 12 in this displayed comparison; 7.69 pp below the leader. Source notes Source cell: 7.6923076923076925
GPT-4.1 →OpenAINYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 4/52; Submission date: 2025/08/12	7.69% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 4/52; Submission date: 2025/08/12	#7 of 12 in this displayed comparison; 7.69 pp below the leader. Source notes Source cell: 7.6923076923076925
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/52; Submission date: 2025/08/12	5.77% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/52; Submission date: 2025/08/12	#9 of 12 in this displayed comparison; 9.62 pp below the leader. Source notes Source cell: 5.769230769230769
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 3/52; Submission date: 2025/03/01	5.77% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 3/52; Submission date: 2025/03/01	#9 of 12 in this displayed comparison; 9.62 pp below the leader. Source notes Source cell: 5.769230769230769
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 2/52; Submission date: 2024/10/02	3.85% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 2/52; Submission date: 2024/10/02	#11 of 12 in this displayed comparison; 11.54 pp below the leader. Source notes Source cell: 3.8461538461538463
GPT-4-1106-preview →OpenAI	1.92% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · crypto subset[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 1/52; Submission date: 2024/10/02	#12 of 12 in this displayed comparison; 13.46 pp below the leader. Source notes Source cell: 1.9230769230769231
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 4/15; Submission date: 2025/08/12	26.67% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 4/15; Submission date: 2025/08/12	Leads this displayed comparison. Source notes Source cell: 26.666666666666668
claude-3.5-sonnet-20240620 →Anthropic	20% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 3/15; Submission date: 2024/10/02	#2 of 12 in this displayed comparison; 6.67 pp below the leader. Source notes Source cell: 20.0
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	20% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	#2 of 12 in this displayed comparison; 6.67 pp below the leader. Source notes Source cell: 20.0
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 3/15; Submission date: 2025/03/01	20% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 3/15; Submission date: 2025/03/01	#2 of 12 in this displayed comparison; 6.67 pp below the leader. Source notes Source cell: 20.0
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	20% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	#2 of 12 in this displayed comparison; 6.67 pp below the leader. Source notes Source cell: 20.0
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 3/15; Submission date: 2025/08/12	20% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 3/15; Submission date: 2025/08/12	#2 of 12 in this displayed comparison; 6.67 pp below the leader. Source notes Source cell: 20.0
GPT-4.1 →OpenAINYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	20% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	#2 of 12 in this displayed comparison; 6.67 pp below the leader. Source notes Source cell: 20.0
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	20% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 3/15; Submission date: 2025/08/12	#2 of 12 in this displayed comparison; 6.67 pp below the leader. Source notes Source cell: 20.0
GPT-4-1106-preview →OpenAI	13.33% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 2/15; Submission date: 2024/10/02	#9 of 12 in this displayed comparison; 13.33 pp below the leader. Source notes Source cell: 13.333333333333334
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 2/15; Submission date: 2025/03/01	13.33% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 2/15; Submission date: 2025/03/01	#9 of 12 in this displayed comparison; 13.33 pp below the leader. Source notes Source cell: 13.333333333333334
GPT-4o →OpenAINYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 2/15; Submission date: 2024/10/02	13.33% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: EnIGMA; Challenges solved: 2/15; Submission date: 2024/10/02	#9 of 12 in this displayed comparison; 13.33 pp below the leader. Source notes Source cell: 13.333333333333334
GPT-4.1 →OpenAINYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 1/15; Submission date: 2025/08/12	6.67% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · forensics subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 1/15; Submission date: 2025/08/12	#12 of 12 in this displayed comparison; 20 pp below the leader. Source notes Source cell: 6.666666666666667
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 7/24; Submission date: 2025/08/12	29.17% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · misc subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG + Graph-RAG); Challenges solved: 7/24; Submission date: 2025/08/12	Leads this displayed comparison (tied). Source notes Source cell: 29.166666666666668
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 7/24; Submission date: 2025/08/12	29.17% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · misc subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 7/24; Submission date: 2025/08/12	Leads this displayed comparison (tied). Source notes Source cell: 29.166666666666668
claude-3.7-sonnet-20250219 →AnthropicNYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 7/24; Submission date: 2025/08/12	29.17% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · misc subset[1]NYU CTF Bench · official submitted results · Agent: D-CIPHER; Challenges solved: 7/24; Submission date: 2025/08/12	Leads this displayed comparison (tied). Source notes Source cell: 29.166666666666668
claude-3.5-sonnet-20241022 →AnthropicNYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 6/24; Submission date: 2025/08/12	25% 1 report	NYU CTF Bench · NYU CTF Bench · pass@1 · misc subset[1]NYU CTF Bench · official submitted results · Agent: CRAKEN (Self-RAG); Challenges solved: 6/24; Submission date: 2025/08/12	#4 of 12 in this displayed comparison; 4.17 pp below the leader. Source notes Source cell: 25.0

Sources

What it measures

Reading the measurement

Published evaluation

Versions and evaluation settings

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-3D

Terminal-Bench 4.0

FrontierCode 1.1

Terminal-Bench 2.1