SLiC-HF (Sequence Likelihood Calibration with Human Feedback)

Directly calibrates the sequence log-likelihood $\log \pi_\theta(y\mid x)$ using a large-margin rank hinge loss $\max(0, \delta - \log \pi_\theta(y_w\mid x) + \log \pi_\theta(y_l\mid x))$ regularized by cross-entropy on the SFT reference distribution.