What it measures
Long-context full-filing (100+ page) and multi-turn conversational extensions of FinQA.
Reading the measurement
Check the source documents, dates, accounting conventions and calculation chain. A financial answer needs evidence and consistent units. Benchmark success does not establish live investment performance or suitability for an individual decision.
Published evaluation
Execution accuracy checks the computed answer; program accuracy checks the generated reasoning program. Oracle-evidence systems are identified in source model labels.
Where this fits in an evaluation suite
Synthetic research rewards do not establish live investment performance. Preserve transaction costs, held-out regimes, constraints, and the exact policy behind a reported result.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
Execution accuracy checks the computed answer; program accuracy checks the generated reasoning program. Oracle-evidence systems are identified in source model labels.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.
PRIMARY EVIDENCE & FURTHER READING
Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.