SimPO (Simple Preference Optimization)

Notices that at inference time, decoding is governed by the policy's own length-normalized log-likelihood $\frac{1}{|y|}\log \pi_\theta(y\mid x)$, whereas DPO optimizes the unnormalized log-ratio against $\pi_{\text{ref}}$. SimPO drops $\pi_{\text{ref}}$ entirely and optimizes the length-normalized average log-probability with a strict target reward margin $\gamma > 0$: $\mathcal{L}_{\text{SimPO}}(\theta) = -\mathbb{E}_{(x,y_w,y_l)}\left[\log \sigma\left(\frac{\beta}{|y_w|}\log \pi_\theta(y_w\mid x) - \frac{\beta}{|y_l|}\log \pi_\theta(y_l\mid x) - \gamma\right)\right]$

Saves 50% forward-pass compute vs. DPO (no $\pi_{\text{ref}}$) and strongly curbs verbosity inflation on AlpacaEval 2.0 and Arena-Hard.