PPO-ptx (PPO with Pretraining Mix Gradients)

Augments the standard PPO RLHF objective with an auxiliary next-token log-likelihood loss on the original pretraining distribution $\mathcal{D}_{\text{pretrain}}$ to prevent the "alignment tax" (regressions on public NLP academic tasks during RLHF): $\mathcal{J}_{\text{PPO-ptx}}(\theta) = \mathcal{J}_{\text{PPO}}(\theta) + \gamma_{\text{ptx}} \mathbb{E}_{x \sim \mathcal{D}_{\text{pretrain}}}[\log \pi_\theta(x)]$

Largely superseded in 2024–2026 by multi-domain SFT/RL mixtures and weight averaging (WARP / Model Soups).