METHOD NOTEBOOK / sDPO

sDPO (Stepwise / Reference-Curriculum DPO)

Splits offline preference datasets into sequential difficulty or quality tranches and updates the reference model π_ref ← π_θ^(k-1) at each tranche boundary so the reference policy stays close…

Mechanism

Splits offline preference datasets into sequential difficulty or quality tranches and updates the reference model $\pi_{\text{ref}} \leftarrow \pi_{\theta^{(k-1)}}$ at each tranche boundary so the reference policy stays close to the active policy.

References

sDPO: Don't Use Your Data All at Once ↗
Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim et al. · 2024
All 7 authors
Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, Chanjun Park