Mechanism
A relative preference objective can widen the chosen–rejected gap while reducing the probability of the chosen response. DPO-Positive adds a penalty when the chosen response loses likelihood relative to the reference. The Smaug paper studies this failure mode; Iterative Reasoning Preference Optimization combines preference learning with a likelihood term for reasoning tasks.
In practice
Track absolute chosen-response likelihood as well as pairwise preference accuracy. Regularization strength, data quality and the reference model change the trade-off. These related objectives should retain their published names and formulas.