SPO (Minimax Self-Play Policy Optimization)

Directly feeds the empirical multi-sample win-rate payoff $\widehat{P}(y_i \succ y_{-i}\mid x) - \frac{1}{2}$ into an on-policy PPO/REINFORCE loop without fitting a parametric reward model, converging to the minimax-optimal policy under arbitrary preference judges.