REINFORCE with Moving-Average / Learned Baseline for LLMs

Treats the entire generated sequence $y$ as a single bandit action (since intermediate environment transitions in text generation are deterministic), optimizing: $\nabla_\theta \mathcal{J}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot\mid x)}\left[\left(R(x,y) - b(x) - \beta \text{KL}(\pi_\theta(\cdot\mid x)\|\pi_{\text{ref}}(\cdot\mid x))\right)\nabla_\theta \log \pi_\theta(y\mid x)\right]$ Showing that token-level MDP machinery (GAE + learned token-level critic $V_\psi$) is often unnecessary when rewards are only given at sequence completion.

Saves ~30–50% GPU memory vs. PPO by dropping the value critic $V_\psi$.