XPO (Exploratory Preference Optimization)

Augments Iterative Online DPO with an explicit theoretical optimism bonus $\alpha \sum_{y \sim \pi_{\text{ref}}} \log \pi_\theta(y\mid x)$ that encourages the policy to actively explore out-of-support responses where the preference model has high epistemic uncertainty, achieving sample-efficient convergence to the global RLHF optimum.