WARM (Weight Averaged Reward Models) & Reward Ensembles

Mitigates Goodhart's Law / reward overoptimization during RLHF without paying the $N\times$ inference cost of running an ensemble of $N$ separate reward models. WARM fine-tunes $N$ reward heads/models from a shared pretrained checkpoint (or along a training trajectory with Baklava/LoRA) and averages their weights in parameter space $\phi_{\text{WARM}} = \frac{1}{N}\sum_{m=1}^N \phi_m$, achieving the out-of-distribution robustness of a reward ensemble at single-model inference cost.