Mechanism
Extends PPO for language generation with a parameterized top- masking policy that dynamically masks out low-probability vocabulary tokens during RL exploration, preventing grammatical collapse in large action spaces ().
Practical Notes
Early demonstration of vocabulary-constrained exploration in LLM policy gradients.
References
- Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization ↗
All 8 authors
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi