Mechanism

Replaces single-scalar reward modeling with a 5-dimensional regression Reward Model (HelpSteer / HelpSteer2 / HelpSteer3) predicting explicit human-interpretable attributes (Helpfulness, Correctness, Coherence, Complexity, Verbosity) ∈[0,4]5\in [0, 4]^5. During SteerLM SFT, responses are conditioned on their ground-truth attribute vector tag in the prompt, and at inference time or during online RL, attributes are set to maximum quality (Helpfulness=4, Correctness=4, Coherence=4) while explicitly controlling Verbosity to prevent length hacking.

References

  1. SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF ↗

    Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, Oleksii Kuchaiev · 2023

  2. HelpSteer2: Open-source dataset for training top-performing reward models ↗

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert et al. · 2024

    All 9 authors

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev