Mechanism

Approximates the online Nash-MD / Online Mirror Descent update using an offline-compatible DPO-style direct loss over preference pairs sampled from the current policy πθt\pi_{\theta_t}, avoiding on-policy token-generation overhead during the gradient inner loop.

References

  1. Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning ↗

    Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song, Ye Tian, Mingyue Huo et al. · 2024

    All 9 authors

    Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song, Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, Dong Yu