IMPALA V-Trace / Asynchronous Actor-Learner LLM RL

Decouples rollout actors (running high-throughput vLLM/SGLang inference engines) from gradient learners (running FSDP/Megatron), correcting for policy staleness $\mu \neq \pi_\theta$ using truncated importance sampling weights $c_t = \min(\bar{c}, \frac{\pi_\theta(a_t\mid s_t)}{\mu(a_t\mid s_t)})$ and $\rho_t = \min(\bar{\rho}, \frac{\pi_\theta(a_t\mid s_t)}{\mu(a_t\mid s_t)})$.

Essential when rollout lengths vary from 500 to 64,000+ tokens and synchronous barriers leave GPUs 70% idle.