Mechanism

Formulates binary-reward self-training as formal Expectation-Maximization on latent correct trajectories:

  • E-step: Sample KK completions per problem from πθt\pi_{\theta_t} and filter by binary verifier r(x,y)∈{0,1}r(x,y) \in \{0, 1\}, weighting prompts to balance problem difficulty.
  • M-step: Reset to the base model πbase\pi_{\text{base}} and fine-tune to convergence on the E-step verified completions.

References

  1. Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models ↗

    Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia et al. · 2023

    All 41 authors

    Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel