ExPO (Extrapolated Policy Optimization)

Observes that an partially aligned DPO/RLHF checkpoint $\theta_{\text{aligned}}$ lies along a beneficial optimization trajectory from the SFT checkpoint $\theta_{\text{SFT}}$, and applies zero-cost linear weight extrapolation ($\alpha > 0$): $\theta_{\text{ExPO}} = \theta_{\text{SFT}} + (1 + \alpha)(\theta_{\text{aligned}} - \theta_{\text{SFT}}) = (1+\alpha)\theta_{\text{aligned}} - \alpha\theta_{\text{SFT}}$ boosting AlpacaEval 2.0 and MT-Bench scores by 3–8 points in seconds without additional training steps.