PRO (Preference Ranking Optimization)

Extends pairwise Bradley-Terry preference optimization to a full listwise Plackett-Luce permutation ranking over $K$ ordered candidates $y_1 \succ y_2 \succ \dots \succ y_K$: $\mathcal{L}_{\text{PRO}}(\theta) = -\sum_{k=1}^{K-1} \log \frac{\exp(r_\theta(x, y_k)/\tau_k)}{\sum_{j=k}^K \exp(r_\theta(x, y_j)/\tau_k)} + \lambda \mathcal{L}_{\text{SFT}}(x, y_1)$ with dynamic temperature $\tau_k$ scaled by reward gaps.