Two places to attach feedback
Outcome supervision evaluates the final result. Process supervision evaluates intermediate work, such as individual reasoning steps. A final-answer checker can give a clear signal on a well-specified task, while step-level judgments can identify where a promising approach first becomes invalid.
More detailed labels create more obligations
A process label needs a definition of a valid step and enough context to judge it. A locally plausible statement can depend on an earlier error. Review the boundary between a genuinely valid inference, a useful heuristic, and an unsupported leap rather than rewarding the surface appearance of careful reasoning.
Do not confuse selection with learning
A process reward model can rank candidate solutions at inference time, supply training rewards, or both. Report which role it played. If a system samples many solutions and selects the best, show that additional search budget separately from changes to the base policy.
Evaluate with independent checks
Use fresh problems and a verifier or expert review that was not optimized as the training reward. Track valid complete solutions, useful partial work, and polished but invalid attempts. A stronger process score should ultimately improve the work the model can justify, not merely its ability to please a familiar critic.
References
- Let's Verify Step by Step ↗
All 10 authors
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe
- ErdősBench: research mathematics, scoped progress and proof audits ↗