Mechanism

Samples two candidate responses (y1,y2)∼πθ(⋅∣x)(y_1, y_2) \sim \pi_\theta(\cdot\mid x) online at every training step, queries an LLM-as-a-Judge in real time to label which response is preferred, and immediately applies an online DPO/IPO gradient step.

References

  1. Direct Language Model Alignment from Online AI Feedback ↗

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares et al. · 2024

    All 12 authors

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, Mathieu Blondel