SWE-bench Multimodal

517 visual software engineering issues (UI rendering, charting, front-end layout, diagram specs) across JS/TS/HTML/CSS repos.26 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
Claude Opus 5.5 →Anthropic	70.21% 1 report	SWE-bench Multimodal[1]Cost / Test: $4.92; Input / Output Cost: $4 / $20; Input / Output Tokens: 3.18B / 38.08M; Duration: 18m32s	Leads this displayed comparison. $4.92 / test18m 32s
Claude Fable 5.1 →Anthropic	61.25% 1 report	SWE-bench Multimodal[1]Cost / Test: $5.46; Input / Output Cost: $10 / $50; Input / Output Tokens: 2.21B / 27.94M; Duration: 15m12s	#2 of 20 in this displayed comparison; 8.96 pp below the leader. $5.46 / test15m 12s
Claude Sonnet 5.5 →Anthropic	58.33% 1 report	SWE-bench Multimodal[1]Cost / Test: $4.53; Input / Output Cost: $2 / $10; Input / Output Tokens: 6.21B / 67.48M; Duration: 24m13s	#3 of 20 in this displayed comparison; 11.88 pp below the leader. $4.53 / test24m 13s
Gemini 4 Argon →Google	56.04% 1 report	SWE-bench Multimodal[1]Cost / Test: $11.40; Input / Output Cost: $4 / $20; Input / Output Tokens: 6.54B / 56.31M; Duration: 33m38s	#4 of 20 in this displayed comparison; 14.17 pp below the leader. $11.40 / test33m 38s
Wonder Full →	47.29% 1 report	SWE-bench Multimodal[1]Cost / Test: $1.86; Input / Output Cost: $4 / $20; Input / Output Tokens: 2.00B / 15.55M; Duration: 14m22s	#5 of 20 in this displayed comparison; 22.92 pp below the leader. $1.86 / test14m 22s
DeepSeek V4.1 Flash →DeepSeek	46.46% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.23; Input / Output Cost: $0.3 / $1.2; Input / Output Tokens: 7.13B / 41.50M; Duration: 19m24s	#6 of 20 in this displayed comparison; 23.75 pp below the leader. $0.23 / test19m 24s
Wonder →	45.63% 1 report	SWE-bench Multimodal[1]Cost / Test: $1.70; Input / Output Cost: $4 / $20; Input / Output Tokens: 1.77B / 14.25M; Duration: 9m44s	#7 of 20 in this displayed comparison; 24.58 pp below the leader. $1.70 / test9m 44s
Kimi K3 →Kimi	44.17% 1 report	SWE-bench Multimodal[1]Cost / Test: $1.65; Input / Output Cost: $3 / $15; Input / Output Tokens: 1.38B / 18.63M; Duration: 22m45s	#8 of 20 in this displayed comparison; 26.04 pp below the leader. $1.65 / test22m 45s
Claude Haiku 5.5 →Anthropic	43.96% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.70; Input / Output Cost: $0.1 / $0.5; Input / Output Tokens: 4.64B / 62.24M; Duration: 16m16s	#9 of 20 in this displayed comparison; 26.25 pp below the leader. $0.70 / test16m 16s
GPT-6.1 Sol →OpenAI	43.54% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.35; Input / Output Cost: $2 / $10; Input / Output Tokens: 242.14M / 8.75M; Duration: 6m56s	#10 of 20 in this displayed comparison; 26.67 pp below the leader. $0.35 / test6m 56s
GPT-6 Astra →OpenAI	43.33% 1 report	SWE-bench Multimodal[1]Cost / Test: $1.81; Input / Output Cost: $10 / $50; Input / Output Tokens: 222.91M / 7.78M; Duration: 7m03s	#11 of 20 in this displayed comparison; 26.88 pp below the leader. $1.81 / test7m 03s
Muse Spark 1.3 →Meta	43.13% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.86; Input / Output Cost: $1.25 / $4.25; Input / Output Tokens: 1.83B / 16.56M; Duration: 10m52s	#12 of 20 in this displayed comparison; 27.08 pp below the leader. $0.86 / test10m 52s
Grok 4.7 →xAI	42.92% 1 report	SWE-bench Multimodal[1]Cost / Test: $4.66; Input / Output Cost: $2 / $6; Input / Output Tokens: 3.19B / 35.75M; Duration: 21m01s	#13 of 20 in this displayed comparison; 27.29 pp below the leader. $4.66 / test21m 01s
Qwen 3.8 Max →Qwen	42.5% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.99; Input / Output Cost: $2 / $6; Input / Output Tokens: 1.15B / 19.06M; Duration: 18m44s	#14 of 20 in this displayed comparison; 27.71 pp below the leader. $0.99 / test18m 44s
MiMo V2.6 Pro →Xiaomi	41.25% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.12; Input / Output Cost: $0.435 / $0.87; Input / Output Tokens: 1.81B / 32.39M; Duration: 46m03s	#15 of 20 in this displayed comparison; 28.96 pp below the leader. $0.12 / test46m 03s
Gemini 3.8 Flash →Google	40.83% 1 report	SWE-bench Multimodal[1]Cost / Test: $3.69; Input / Output Cost: $1.5 / $7.5; Input / Output Tokens: 6.22B / 38.86M; Duration: 22m18s	#16 of 20 in this displayed comparison; 29.38 pp below the leader. $3.69 / test22m 18s
Step 5 Preview →StepFun	40.83% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.61; Input / Output Cost: $1 / $2.7; Input / Output Tokens: 2.91B / 27.19M; Duration: 26m06s	#16 of 20 in this displayed comparison; 29.38 pp below the leader. $0.61 / test26m 06s
GPT-6 Luna →OpenAI	37.5% 1 report	SWE-bench Multimodal[1]Cost / Test: $0.03; Input / Output Cost: $0.1 / $0.5; Input / Output Tokens: 524.11M / 13.09M; Duration: 5m16s	#18 of 20 in this displayed comparison; 32.71 pp below the leader. $0.03 / test5m 16s
MiniMax M3 →MiniMax	33.33% 1 report	SWE-bench Multimodal[1]Cost / Test: $1.02; Input / Output Cost: $0.6 / $2.4; Input / Output Tokens: 3.57B / 16.50M; Duration: 13m09s	#19 of 20 in this displayed comparison; 36.88 pp below the leader. $1.02 / test13m 09s
Mistral Large 4 →Mistral	33.13% 1 report	SWE-bench Multimodal[1]Cost / Test: $5.14; Input / Output Cost: $1.36 / $4.18; Input / Output Tokens: 2.08B / 34.74M; Duration: 27m27s	#20 of 20 in this displayed comparison; 37.08 pp below the leader. $5.14 / test27m 27s
GPT-4o →OpenAISWE-bench Multimodal · original baselines · System: SWE-agent M	12.2% 1 report	SWE-bench Multimodal · Resolved tasks[2]SWE-bench Multimodal · original baselines · System: SWE-agent M	Leads this displayed comparison. Source notes Source cell: 12.2
GPT-4o →OpenAISWE-bench Multimodal · original baselines · System: SWE-agent Base	12% 1 report	SWE-bench Multimodal · Resolved tasks[2]SWE-bench Multimodal · original baselines · System: SWE-agent Base	#2 of 6 in this displayed comparison; 0.2 pp below the leader. Source notes Source cell: 12.0
Claude 3.5 Sonnet →Anthropic	9.36% 5 reports · range 5–12.2%	SWE-bench Multimodal · Resolved tasks[2] View 5 reports 12.2% · SWE-bench Multimodal · Resolved tasks · original baselines · SWE-bench Multimodal · original baselines · System: [2] 12% · SWE-bench Multimodal · Resolved tasks · original baselines · SWE-bench Multimodal · original baselines · System: [2] 11.4% · SWE-bench Multimodal · Resolved tasks · original baselines · SWE-bench Multimodal · original baselines · System: [2] 6.2% · SWE-bench Multimodal · Resolved tasks · original baselines · SWE-bench Multimodal · original baselines · System: [2] 5% · SWE-bench Multimodal · Resolved tasks · original baselines · SWE-bench Multimodal · original baselines · System: [2]	Published observations span 7.2 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #3 of 6 in this displayed comparison; 2.84 pp below the leader.
GPT-4o →OpenAISWE-bench Multimodal · original baselines · System: SWE-agent JS	9.2% 1 report	SWE-bench Multimodal · Resolved tasks[2]SWE-bench Multimodal · original baselines · System: SWE-agent JS	#4 of 6 in this displayed comparison; 3 pp below the leader. Source notes Source cell: 9.2
GPT-4o →OpenAISWE-bench Multimodal · original baselines · System: RAG	6% 1 report	SWE-bench Multimodal · Resolved tasks[2]SWE-bench Multimodal · original baselines · System: RAG	#5 of 6 in this displayed comparison; 6.2 pp below the leader. Source notes Source cell: 6.0
GPT-4o →OpenAISWE-bench Multimodal · original baselines · System: Agentless JS	3.1% 1 report	SWE-bench Multimodal · Resolved tasks[2]SWE-bench Multimodal · original baselines · System: Agentless JS	#6 of 6 in this displayed comparison; 9.1 pp below the leader. Source notes Source cell: 3.1

Sources

SWE-bench Multimodal — evaluation methodology and results ↗
Vals AI · 2026
Dates & evaluation details
SWE-bench Multimodal · Accuracy · 2026-10-26
Public source leaderboard captured 26 October 2026. Model names, configurations, native score units and source-reported uncertainty are retained. Version and task split follow this specific Vals methodology page. These are published observations, not evaluations run by RLTasks.
SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? — published result table ↗
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu et al. · 2024
All 13 authors
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, Ofir Press
Dates & evaluation details
SWE-bench Multimodal · Resolved tasks · original baselines · 2026-10-26
Published model comparison, 2024. SWE-bench Multimodal · original baselines. Resolved tasks; SWE-bench Multimodal · original baselines. Model configurations and missing observations follow the original publication. Table 3; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

Score vs. cost per test

Vals AI · 20 observations

SWE-bench Multimodal · Accuracy · higher is better

Costs are Vals AI’s published Cost / Test in USD, paired with the score from the same result. They reflect that evaluation’s workload and configuration; they are not token prices or cost per successful task. Missing costs are omitted. Explore all reported results ↓

The dashed line traces the best observed score as reported cost increases. It describes this selection of reports, with their original configurations. It does not estimate unmeasured models or costs.

View plotted values and sources (20)

SWE-bench Multimodal · SWE-bench Multimodal · Accuracy. Published measurements in their original score units.
Model	Score	Cost per test (USD)	Configuration & source
Claude Opus 5.5 →	70.21 %	$4.92	Cost / Test: $4.92; Input / Output Cost: $4 / $20; Input / Output Tokens: 3.18B / 38.08M; Duration: 18m32s Vals AI · SWE-bench Multimodal · Accuracy ↗
Claude Fable 5.1 →	61.25 %	$5.46	Cost / Test: $5.46; Input / Output Cost: $10 / $50; Input / Output Tokens: 2.21B / 27.94M; Duration: 15m12s Vals AI · SWE-bench Multimodal · Accuracy ↗
Claude Sonnet 5.5 →	58.33 %	$4.53	Cost / Test: $4.53; Input / Output Cost: $2 / $10; Input / Output Tokens: 6.21B / 67.48M; Duration: 24m13s Vals AI · SWE-bench Multimodal · Accuracy ↗
Gemini 4 Argon →	56.04 %	$11.40	Cost / Test: $11.40; Input / Output Cost: $4 / $20; Input / Output Tokens: 6.54B / 56.31M; Duration: 33m38s Vals AI · SWE-bench Multimodal · Accuracy ↗
Wonder Full →	47.29 %	$1.86	Cost / Test: $1.86; Input / Output Cost: $4 / $20; Input / Output Tokens: 2.00B / 15.55M; Duration: 14m22s Vals AI · SWE-bench Multimodal · Accuracy ↗
DeepSeek V4.1 Flash →	46.46 %	$0.23	Cost / Test: $0.23; Input / Output Cost: $0.3 / $1.2; Input / Output Tokens: 7.13B / 41.50M; Duration: 19m24s Vals AI · SWE-bench Multimodal · Accuracy ↗
Wonder →	45.63 %	$1.70	Cost / Test: $1.70; Input / Output Cost: $4 / $20; Input / Output Tokens: 1.77B / 14.25M; Duration: 9m44s Vals AI · SWE-bench Multimodal · Accuracy ↗
Kimi K3 →	44.17 %	$1.65	Cost / Test: $1.65; Input / Output Cost: $3 / $15; Input / Output Tokens: 1.38B / 18.63M; Duration: 22m45s Vals AI · SWE-bench Multimodal · Accuracy ↗
Claude Haiku 5.5 →	43.96 %	$0.70	Cost / Test: $0.70; Input / Output Cost: $0.1 / $0.5; Input / Output Tokens: 4.64B / 62.24M; Duration: 16m16s Vals AI · SWE-bench Multimodal · Accuracy ↗
GPT-6.1 Sol →	43.54 %	$0.35	Cost / Test: $0.35; Input / Output Cost: $2 / $10; Input / Output Tokens: 242.14M / 8.75M; Duration: 6m56s Vals AI · SWE-bench Multimodal · Accuracy ↗
GPT-6 Astra →	43.33 %	$1.81	Cost / Test: $1.81; Input / Output Cost: $10 / $50; Input / Output Tokens: 222.91M / 7.78M; Duration: 7m03s Vals AI · SWE-bench Multimodal · Accuracy ↗
Muse Spark 1.3 Max →	43.13 %	$0.86	Cost / Test: $0.86; Input / Output Cost: $1.25 / $4.25; Input / Output Tokens: 1.83B / 16.56M; Duration: 10m52s Vals AI · SWE-bench Multimodal · Accuracy ↗
Grok 4.7 →	42.92 %	$4.66	Cost / Test: $4.66; Input / Output Cost: $2 / $6; Input / Output Tokens: 3.19B / 35.75M; Duration: 21m01s Vals AI · SWE-bench Multimodal · Accuracy ↗
Qwen 3.8 Max →	42.5 %	$0.99	Cost / Test: $0.99; Input / Output Cost: $2 / $6; Input / Output Tokens: 1.15B / 19.06M; Duration: 18m44s Vals AI · SWE-bench Multimodal · Accuracy ↗
MiMo V2.6 Pro →	41.25 %	$0.12	Cost / Test: $0.12; Input / Output Cost: $0.435 / $0.87; Input / Output Tokens: 1.81B / 32.39M; Duration: 46m03s Vals AI · SWE-bench Multimodal · Accuracy ↗
Gemini 3.8 Flash →	40.83 %	$3.69	Cost / Test: $3.69; Input / Output Cost: $1.5 / $7.5; Input / Output Tokens: 6.22B / 38.86M; Duration: 22m18s Vals AI · SWE-bench Multimodal · Accuracy ↗
Step 5 Preview →	40.83 %	$0.61	Cost / Test: $0.61; Input / Output Cost: $1 / $2.7; Input / Output Tokens: 2.91B / 27.19M; Duration: 26m06s Vals AI · SWE-bench Multimodal · Accuracy ↗
GPT-6 Luna →	37.5 %	$0.03	Cost / Test: $0.03; Input / Output Cost: $0.1 / $0.5; Input / Output Tokens: 524.11M / 13.09M; Duration: 5m16s Vals AI · SWE-bench Multimodal · Accuracy ↗
MiniMax-M3 →	33.33 %	$1.02	Cost / Test: $1.02; Input / Output Cost: $0.6 / $2.4; Input / Output Tokens: 3.57B / 16.50M; Duration: 13m09s Vals AI · SWE-bench Multimodal · Accuracy ↗
Mistral Large 4 →	33.13 %	$5.14	Cost / Test: $5.14; Input / Output Cost: $1.36 / $4.18; Input / Output Tokens: 2.08B / 34.74M; Duration: 27m27s Vals AI · SWE-bench Multimodal · Accuracy ↗

What it measures

517 visual software engineering issues (UI rendering, charting, front-end layout, diagram specs) across JS/TS/HTML/CSS repos.

Reading the measurement

Look beyond a plausible code sample to the executable artifact. Tests, repository context, language coverage and time limits shape the result. Separate first-attempt correctness from success after debugging, retries or agent assistance.

Published evaluation

The linked publication documents the task and scoring procedure. Model results retain the source’s evaluation setting, native score units and available uncertainty.

Published evaluation

Resolved tasks; SWE-bench Multimodal · original baselines. Model configurations and missing observations follow the original publication.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Resolved tasks; SWE-bench Multimodal · original baselines. Model configurations and missing observations follow the original publication.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? — published result table ↗
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu et al. · 2024
All 13 authors
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, Ofir Press
SWE-bench Multimodal — evaluation methodology and results ↗
Vals AI · 2026

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All code & engineering benchmarks →

Sources

Score vs. cost per test

What it measures

Reading the measurement

Published evaluation

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-3D

Terminal-Bench 4.0

FrontierCode 1.1

Terminal-Bench 2.1