Self-Rewarding Language Models

Unifies the actor policy and the LLM-as-a-Judge reward model into a single model $\pi_\theta$: in Iteration $t$, $\pi_{\theta_t}$ generates new candidate prompts, samples $K$ candidate responses per prompt, grades its own responses via CoT LLM-as-a-Judge prompting (Instruction + Response -> Score 1-5), and trains $\pi_{\theta_{t+1}}$ via DPO on both the improved task preferences and judge traces, improving both generation ability and judging accuracy across iterations.