Mechanism

Aligns a single LLM simultaneously to MM competing preference dimensions (e.g., helpfulness, safety, conciseness, citation faithfulness) by incorporating scalarized reward margins ∑m≠kwmrm(x,y)\sum_{m \neq k} w_m r_m(x, y) from M−1M-1 auxiliary reward models inside the primary DPO loss for objective kk.

References

  1. Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization ↗

    Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang et al. · 2023

    All 7 authors

    Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, Yu Qiao