What it measures

Customer-service agents converse with simulated users while using tools and following domain policy.

How to read the result

Verified final-state success and repeated-run reliability reveal whether the assistant consistently completes the user’s actual request.

Limits and comparison traps

Retail and airline tasks, user simulator models and pass@k or pass^k metrics define different cohorts. Keep the original suite separate from τ² and τ³.

Published evaluation

Pass^1 success under user interaction, policy constraints and final-state checks. The overall average weights the two domains equally, not by task count. Llama-3 uses the reported text-ReAct exception.

Where this fits in an evaluation suite

A transcript that says “done” is weaker evidence than the verified final state. Track policy violations, recoveries, and invalid attempts alongside success.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Pass^1 success under user interaction, policy constraints and final-state checks. The overall average weights the two domains equally, not by task count. Llama-3 uses the reported text-ReAct exception.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

  1. $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — published result table ↗

    Yao, Shunyu, Shinn, Noah, Razavi, Pedram, Narasimhan, Karthik · 2024

  2. Primary benchmark documentation or model evaluation report ↗

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.