FinQA

What it measures

Questions over financial reports require retrieving the relevant figures and composing a numerical calculation.

How to read the result

Execution accuracy checks the numerical outcome; program accuracy tests a more specific computation representation. Keep the metric and financial units explicit.

Limits and comparison traps

A correct number can result from a different valid program. Document extraction, answer normalization and retrieval settings affect comparisons.

Published evaluation

Execution accuracy checks the computed answer; program accuracy checks the generated reasoning program. Oracle-evidence systems are identified in source model labels.

Where this fits in an evaluation suite

Synthetic research rewards do not establish live investment performance. Preserve transaction costs, held-out regimes, constraints, and the exact policy behind a reported result.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Execution accuracy checks the computed answer; program accuracy checks the generated reasoning program. Oracle-evidence systems are identified in source model labels.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.