From rollouts to policy updates

A policy generates trajectories; the training system scores them and estimates which sampled choices should become more likely. PPO uses a constrained-style surrogate objective to limit destructive policy updates while reusing a batch of sampled experience. Its original formulation is a general RL algorithm, not specifically an LLM method.

GRPO was introduced in DeepSeekMath as a PPO variant that uses group-relative reward information and avoids a separate value-model component in that setup. Relative scoring makes the composition of the sample group important: an uninformative group supplies little useful learning signal.

Record rollout count, sampling temperature, sequence limits, reward normalization, reference-policy regularization, and update schedule. Also record environment and verifier versions. Two systems called GRPO can have very different effective training and inference budgets.

Evaluate the trained policy under a matched test-time budget. If the new system samples more answers, uses more tools, or employs a stronger reranker, show that as a separate intervention. Track success per unit of compute and whether gains survive a fresh task distribution.