Mechanism

Exploits the equivalence between maximum-entropy RL and soft Bellman consistency along any sub-trajectory (st,at:t+k,st+k)(s_t, a_{t:t+k}, s_{t+k}), allowing unified off-policy and on-policy token-level updates without separate actor-critic instability.

Practical Notes

Connects directly to token-level DPO and GFlowNet trajectory balance objectives.

References

  1. Bridging the Gap Between Value and Policy Based Reinforcement Learning ↗

    Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans · 2017

  2. Efficient (Soft) Q-Learning for Text Generation with Limited Good Data ↗

    Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, Zhiting Hu · 2021

  3. Offline Regularised Reinforcement Learning for Large Language Models Alignment ↗

    Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov et al. · 2024

    All 18 authors

    Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, Bilal Piot