DNO (Direct Nash Optimization)

Batched on-policy iterative algorithm that samples $K$ outputs from the current policy $\pi_{\theta_t}$, compares them against each other and the gold/teacher output using a general pairwise win-rate oracle $\mathbb{P}(y_i \succ y_j)$ (which need not satisfy Bradley-Terry transitivity!), and regresses internal log-ratios toward the empirical win-rate margin.