Mechanism

Optimizes LLM policies over multi-turn Partially Observable Markov Decision Processes (POMDPs) τ=(s0,a0,o1,a1,…,oT,aT)\tau = (s_0, a_0, o_1, a_1, \dots, o_T, a_T) where actions ata_t execute inside stateful external sandboxes (Docker Linux terminals, git repositories, headless browsers, GUI virtual machines, CAD engines, or MCP tool servers). Training masks environment observations oto_t from policy gradient updates (∇θlog⁡πθ(at∣s0,a<t,o≤t)\nabla_\theta \log \pi_\theta(a_t \mid s_0, a_{<t}, o_{\le t})) and combines terminal state verification (e.g., pytest fail-to-pass & pass-to-pass suites, git diff similarity, DOM/DB state assertions) with format, tool-efficiency, and step-budget rewards.

References

  1. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution ↗

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried et al. · 2025

    All 9 authors

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, Sida I. Wang

  2. WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning ↗

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao et al. · 2024

    All 14 authors

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, Yuxiao Dong

  3. ToolRL: Reward is All Tool Learning Needs ↗

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür et al. · 2025

    All 8 authors

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji