Mechanism

Closes the off-policy gap of vanilla DPO by alternating across TT rounds: in round tt, sample KK responses {y1,…,yK}∼πθt(⋅∣x)\{y_1,\dots,y_K\} \sim \pi_{\theta_t}(\cdot\mid x), score them with a reward model or verifier to form fresh on-policy preference pairs (yw,yl)(y_w, y_l), update πθt+1\pi_{\theta_{t+1}} via DPO (with πref=πθt\pi_{\text{ref}} = \pi_{\theta_t}), and repeat.

Practical Notes

Consistently bridges 80–95% of the gap between offline DPO and full online PPO/GRPO.

References

  1. Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss ↗

    Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, Jason Weston · 2023

  2. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint ↗

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji et al. · 2023

    All 8 authors

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, Tong Zhang

  3. Direct Language Model Alignment from Online AI Feedback ↗

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares et al. · 2024

    All 12 authors

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, Mathieu Blondel