KTO (Kahneman-Tversky Optimization)

Eliminates the requirement for paired $(y_w, y_l)$ preferences on the same prompt $x$. Inspired by Kahneman & Tversky's Prospect Theory (human utility is concave in gains, convex and steeper in losses relative to a reference point $z_0$), KTO trains directly on unpaired binary feedback $(x, y, \text{label} \in \{\text{desirable}, \text{undesirable}\})$: $\mathcal{L}_{\text{KTO}}(\theta) = \mathbb{E}_{x,y}\left[\lambda_y - v(x,y)\right], \quad v(x,y) = \begin{cases} \lambda_D \sigma\left(\beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)} - z_0\right) & y \sim \mathcal{D}_{\text{desirable}} \\ \lambda_U \sigma\left(z_0 - \beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\right) & y \sim \mathcal{D}_{\text{undesirable}} \end{cases}$ where $z_0 = \mathbb{E}_{x', y'\sim \pi_\theta}\left[\beta \log \frac{\pi_\theta(y'\mid x')}{\pi_{\text{ref}}(y'\mid x')}\right]$ estimates the KL reference point across mismatched batch items.