RLOO (REINFORCE Leave-One-Out)

Samples $K$ independent completions $\{y^{(1)}, \dots, y^{(K)}\} \sim \pi_\theta(\cdot\mid x)$ per prompt and uses the average reward of the other $K-1$ completions as an unbiased, low-variance prompt-specific baseline without any critic network or standard-deviation division: $\hat{A}_{\text{RLOO}}(x, y^{(i)}) = R(x, y^{(i)}) - \frac{1}{K-1}\sum_{j \neq i} R(x, y^{(j)})$

Unbiased gradient estimator (unlike GRPO, which divides by empirical group standard deviation and biases difficulty weighting).