APA (Advantage-Induced Policy Alignment)

Replaces PPO's multiplicative ratio clipping with a squared-error loss between the log-policy ratio $\log \frac{\pi_\theta(y_t\mid s_t)}{\pi_{\text{ref}}(y_t\mid s_t)}$ and the scaled token advantage $\frac{\hat{A}_t}{\beta}$, enforcing explicit trust-region control around the reference policy.

Avoids flat zero-gradient regions outside the $[1-\epsilon, 1+\epsilon]$ PPO clip interval.