SPIN (Self-Play Fine-Tuning)

Allows an SFT model to improve without any additional human preference data or external reward model via a two-player minimax game: at iteration $t$, the opponent (old policy $\pi_{\theta_t}$) generates synthetic responses $y' \sim \pi_{\theta_t}(\cdot\mid x)$, while the main player $\pi_{\theta_{t+1}}$ is trained via a DPO-like logistic loss to assign higher likelihood to real human SFT demonstrations $y_{\text{real}} \sim p_{\text{data}}$ than to its own previous self-generated outputs $y'$, converging when $\pi_\theta = p_{\text{data}}$.

Extracts substantial extra capability from a fixed SFT dataset after standard 1–2 epoch SFT has saturated.