Mechanism

Extends Self-Rewarding LMs by adding a third Meta-Judge role to the same model: for each response, the model generates multiple candidate judge critiques/scores, and the Meta-Judge evaluates which critique is most accurate and calibrated, constructing DPO pairs for the judge role itself to stop self-reward hacking and length drift.

References

  1. Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge ↗

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao et al. · 2024

    All 8 authors

    Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, Sainbayar Sukhbaatar