WindowsAgentArena

150+ real Windows OS desktop tasks across Edge, Office, File Explorer, VS Code, and Windows Settings.27 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
GPT-4V-1106 →OpenAIWindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✓; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser	19.5% 1 report	WindowsAgentArena · Task success[1]WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✓; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser	Leads this displayed comparison.
GPT-4V-1106 →OpenAIWindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✓; —————————— Inputs ——————————: Pytesseract+ DOM; —————————— Inputs ——————————: Grounding DINO; —————————— Inputs ——————————: Grounding DINO	13.1% 1 report	WindowsAgentArena · Task success[1]WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✓; —————————— Inputs ——————————: Pytesseract+ DOM; —————————— Inputs ——————————: Grounding DINO; —————————— Inputs ——————————: Grounding DINO	#2 of 9 in this displayed comparison; 6.4 pp below the leader.
GPT-4V-1106 →OpenAIWindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✓; —————————— Inputs ——————————: OneOCR; —————————— Inputs ——————————: Proprietary models; —————————— Inputs ——————————: Proprietary models	13.1% 1 report	WindowsAgentArena · Task success[1]WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✓; —————————— Inputs ——————————: OneOCR; —————————— Inputs ——————————: Proprietary models; —————————— Inputs ——————————: Proprietary models	#2 of 9 in this displayed comparison; 6.4 pp below the leader.
GPT-4V-1106 →OpenAIWindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✗; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser	12.5% 1 report	WindowsAgentArena · Task success[1]WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✗; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser; —————————— Inputs ——————————: Omniparser	#4 of 9 in this displayed comparison; 7 pp below the leader.
GPT-4V-1106 →OpenAIWindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✗; —————————— Inputs ——————————: OneOCR; —————————— Inputs ——————————: Proprietary models; —————————— Inputs ——————————: Proprietary models	11.3% 1 report	WindowsAgentArena · Task success[1]WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✗; —————————— Inputs ——————————: OneOCR; —————————— Inputs ——————————: Proprietary models; —————————— Inputs ——————————: Proprietary models	#5 of 9 in this displayed comparison; 8.2 pp below the leader.
GPT-4o →OpenAI	9.27% 6 reports · range 5.2–13.3%	WindowsAgentArena · Task success[1] View 6 reports 13.3% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 9.8% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 9.4% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 9.3% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 8.6% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 5.2% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1]	Published observations span 8.1 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #6 of 9 in this displayed comparison; 10.23 pp below the leader.
GPT-4V-1106 →OpenAIWindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✗; —————————— Inputs ——————————: Pytesseract + DOM; —————————— Inputs ——————————: Grounding DINO; —————————— Inputs ——————————: Grounding DINO	8.6% 1 report	WindowsAgentArena · Task success[1]WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ✗; —————————— Inputs ——————————: Pytesseract + DOM; —————————— Inputs ——————————: Grounding DINO; —————————— Inputs ——————————: Grounding DINO	#7 of 9 in this displayed comparison; 10.9 pp below the leader.
GPT-4o-mini →OpenAI	4.88% 6 reports · range 2.6–7.3%	WindowsAgentArena · Task success[1] View 6 reports 7.3% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 7.2% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 5.3% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 4.2% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 2.7% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 2.6% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1]	Published observations span 4.7 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #8 of 9 in this displayed comparison; 14.62 pp below the leader.
Phi3-V →Microsoft	2.67% 6 reports · range 1.3–4%	WindowsAgentArena · Task success[1] View 6 reports 4% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 3.5% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 3.2% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 2% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 2% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1] 1.3% · WindowsAgentArena · Task success · Navi configurations · WindowsAgentArena · Navi configurations · —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: ; —————————— Inputs ——————————: [1]	Published observations span 2.7 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty. #9 of 9 in this displayed comparison; 16.83 pp below the leader.

Sources

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale — published result table ↗
Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng et al. · 2024
All 12 authors
Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng, Lu, Yadong, Wagle, Justin, Koishida, Kazuhito, Bucker, Arthur, Jang, Lawrence, Hui, Zack
Dates & evaluation details
WindowsAgentArena · Task success · Navi configurations · 2026-10-27
Published model comparison, 2024. WindowsAgentArena · Navi configurations. Success across 154 Windows tasks. Accessibility trees, OCR, icon and image detectors are input-setting variables; source configurations remain visible. Table 4; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

150+ real Windows OS desktop tasks across Edge, Office, File Explorer, VS Code, and Windows Settings.

Reading the measurement

The useful outcome is a verified change or deliverable. Inspect the tool interface, initial state, user simulator and policy constraints. An agent can claim completion without reaching the required state, or reach it through an action the protocol forbids.

Published evaluation

Success across 154 Windows tasks. Accessibility trees, OCR, icon and image detectors are input-setting variables; source configurations remain visible.

Where this fits in an evaluation suite

A transcript that says “done” is weaker evidence than the verified final state. Track policy violations, recoveries, and invalid attempts alongside success.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Success across 154 Windows tasks. Accessibility trees, OCR, icon and image detectors are input-setting variables; source configurations remain visible.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale — published result table ↗
Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng et al. · 2024
All 12 authors
Bonatti, Rogerio, Zhao, Dan, Bonacci, Francesco, Dupont, Dillon, Abdali, Sara, Li, Yinheng, Lu, Yadong, Wagle, Justin, Koishida, Kazuhito, Bucker, Arthur, Jang, Lawrence, Hui, Zack

Reviewed 2026-10-27. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All agents & tool use benchmarks →

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-Briefcase v1.1

GDPval-AA v2.1

APEX-Agents-AA

AA-AnalystAgent