Nash-MD (Nash Learning from Human Feedback via Mirror Descent)

Replaces the reward-maximization paradigm of RLHF with finding the Nash equilibrium of a KL-regularized preference model $\mathcal{P}(y \succ y' \mid x)$ against a geometric mixture opponent policy $\pi_t^{\text{mix}}(y\mid x) \propto \pi_{\theta_t}(y\mid x)^{1-\alpha}\pi_{\text{ref}}(y\mid x)^\alpha$, updated via policy gradient against the pairwise preference oracle.

Immune to Condorcet cycles (where $A \succ B \succ C \succ A$ in human preferences breaks scalar Bradley-Terry reward models).