SWE-Lancer

1,400+ real freelance Upwork software engineering tasks ($1M total payout value) spanning IC implementation and SWE Management.27 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
3.5 Sonnet →Anthropic	26.2% 1 report	SWE-Lancer · IC SWE (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: N/A	Leads this displayed comparison.
o1 →OpenAISWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: High	16.5% 1 report	SWE-Lancer · IC SWE (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: High	#2 of 7 in this displayed comparison; 9.7 pp below the leader.
o1 →OpenAISWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: Medium	15.6% 1 report	SWE-Lancer · IC SWE (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: Medium	#3 of 7 in this displayed comparison; 10.6 pp below the leader.
o1 →OpenAISWE-Lancer · original evaluation · User Tool: No; Reasoning Effort: High	13.1% 1 report	SWE-Lancer · IC SWE (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: No; Reasoning Effort: High	#4 of 7 in this displayed comparison; 13.1 pp below the leader.
o1 →OpenAISWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: Low	9.3% 1 report	SWE-Lancer · IC SWE (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: Low	#5 of 7 in this displayed comparison; 16.9 pp below the leader.
GPT-4o →OpenAISWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: N/A	8% 1 report	SWE-Lancer · IC SWE (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: N/A	#6 of 7 in this displayed comparison; 18.2 pp below the leader. Source notes Source cell: 8.0%
GPT-4o →OpenAISWE-Lancer · original evaluation · User Tool: No; Reasoning Effort: N/A	8% 1 report	SWE-Lancer · IC SWE (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: No; Reasoning Effort: N/A	#6 of 7 in this displayed comparison; 18.2 pp below the leader. Source notes Source cell: 8.0%
3.5 Sonnet →Anthropic	21.1% 1 report	SWE-Lancer · IC SWE (Full) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: N/A	Leads this displayed comparison.
o1 →OpenAI	20.3% 1 report	SWE-Lancer · IC SWE (Full) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: High	#2 of 3 in this displayed comparison; 0.8 pp below the leader.
GPT-4o →OpenAI	8.6% 1 report	SWE-Lancer · IC SWE (Full) · pass@1[1]SWE-Lancer · original evaluation · User Tool: Yes; Reasoning Effort: N/A	#3 of 3 in this displayed comparison; 12.5 pp below the leader.
3.5 Sonnet →Anthropic	44.9% 1 report	SWE-Lancer · SWE Manager (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: N/A	Leads this displayed comparison.
o1 →OpenAI	41.5% 1 report	SWE-Lancer · SWE Manager (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: High	#2 of 3 in this displayed comparison; 3.4 pp below the leader.
GPT-4o →OpenAI	37% 1 report	SWE-Lancer · SWE Manager (Diamond) · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: N/A	#3 of 3 in this displayed comparison; 7.9 pp below the leader. Source notes Source cell: 37.0%
3.5 Sonnet →Anthropic	47% 1 report	SWE-Lancer · SWE Manager (Full) · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: High	Leads this displayed comparison. Source notes Source cell: 47.0%
o1 →OpenAI	46.3% 1 report	SWE-Lancer · SWE Manager (Full) · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: High	#2 of 3 in this displayed comparison; 0.7 pp below the leader.
GPT-4o →OpenAI	38.7% 1 report	SWE-Lancer · SWE Manager (Full) · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: N/A	#3 of 3 in this displayed comparison; 8.3 pp below the leader.
3.5 Sonnet →Anthropic	36.1% 1 report	SWE-Lancer · SWE-Lancer Diamond · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: N/A	Leads this displayed comparison.
o1 →OpenAI	29.7% 1 report	SWE-Lancer · SWE-Lancer Diamond · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: High	#2 of 3 in this displayed comparison; 6.4 pp below the leader.
GPT-4o →OpenAI	23.3% 1 report	SWE-Lancer · SWE-Lancer Diamond · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: N/A	#3 of 3 in this displayed comparison; 12.8 pp below the leader.
3.5 Sonnet →Anthropic	33.7% 1 report	SWE-Lancer · SWE-Lancer Full · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: N/A	Leads this displayed comparison.
o1 →OpenAI	32.9% 1 report	SWE-Lancer · SWE-Lancer Full · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: High	#2 of 3 in this displayed comparison; 0.8 pp below the leader.
GPT-4o →OpenAI	23.3% 1 report	SWE-Lancer · SWE-Lancer Full · pass@1[1]SWE-Lancer · original evaluation · User Tool: N/A; Reasoning Effort: N/A	#3 of 3 in this displayed comparison; 10.4 pp below the leader.

Sources

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — published result table ↗
Miserendino, Samuel, Wang, Michele, Patwardhan, Tejal, Heidecke, Johannes · 2025
Dates & evaluation details
SWE-Lancer · IC SWE (Diamond) · pass@1 · original evaluation · 2026-10-27
Published model comparison, 2025. SWE-Lancer · original evaluation. Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
SWE-Lancer · IC SWE (Full) · pass@1 · original evaluation · 2026-10-27
Published model comparison, 2025. SWE-Lancer · original evaluation. Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
SWE-Lancer · SWE Manager (Diamond) · pass@1 · original evaluation · 2026-10-27
Published model comparison, 2025. SWE-Lancer · original evaluation. Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
SWE-Lancer · SWE Manager (Full) · pass@1 · original evaluation · 2026-10-27
Published model comparison, 2025. SWE-Lancer · original evaluation. Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
SWE-Lancer · SWE-Lancer Diamond · pass@1 · original evaluation · 2026-10-27
Published model comparison, 2025. SWE-Lancer · original evaluation. Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
SWE-Lancer · SWE-Lancer Full · pass@1 · original evaluation · 2026-10-27
Published model comparison, 2025. SWE-Lancer · original evaluation. Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

1,400+ real freelance Upwork software engineering tasks ($1M total payout value) spanning IC implementation and SWE Management.

Reading the measurement

Look beyond a plausible code sample to the executable artifact. Tests, repository context, language coverage and time limits shape the result. Separate first-attempt correctness from success after debugging, retries or agent assistance.

Published evaluation

Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Published pass@1 under the named paid-software-task subset. Task success and dollar earnings are distinct metrics; user tools and reasoning effort remain in configuration.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — published result table ↗
Miserendino, Samuel, Wang, Michele, Patwardhan, Tejal, Heidecke, Johannes · 2025

Reviewed 2026-10-27. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All code & engineering benchmarks →

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-3D

Terminal-Bench 4.0

FrontierCode 1.1

Terminal-Bench 2.1