CPO (Contrastive Preference Optimization)

Approximates the ideal reference policy as a uniform prior to drop $\pi_{\text{ref}}$ memory overhead, combining a reference-free DPO contrastive loss with an explicit SFT negative log-likelihood regularizer on $y_w$: $\mathcal{L}_{\text{CPO}} = \mathcal{L}_{\text{DPO-uniform}}(\theta) + \mathcal{L}_{\text{NLL}}(\theta; y_w)$.