Mechanism

Handles off-policy lag in asynchronous RL by combining asymmetric tapered importance-sampling weights on positive-advantage trajectories with unclipped/bounded updates on negative-advantage trajectories, accelerating learning from rare successful trajectories without off-policy divergence.

References

  1. Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs ↗

    Nicolas Le Roux, Marc G. Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette et al. · 2025

    All 10 authors

    Nicolas Le Roux, Marc G. Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer, Sándor Toth, Sam Work