PRIME (Process Reinforcement through Implicit Rewards)

Trains an implicit Process Reward Model $\pi_\psi$ online alongside the policy $\pi_\theta$ using only outcome labels $r \in \{0,1\}$, where token-level implicit process rewards are defined automatically via the log-ratio $r_t^{\text{PRM}} = \beta_{\text{PRM}} \log \frac{\pi_\psi(y_t\mid y_{<t})}{\pi_{\text{ref}}(y_t\mid y_{<t})}$, and combines dense implicit process advantages with sparse outcome advantages in an online RLOO/GRPO loop.

Achieves dense token-level process credit assignment using only binary outcome verifiers—no step-level human/MCTS annotations required.