Mechanism
Replaces single-scalar reward modeling with a 5-dimensional regression Reward Model (HelpSteer / HelpSteer2 / HelpSteer3) predicting explicit human-interpretable attributes (Helpfulness, Correctness, Coherence, Complexity, Verbosity) . During SteerLM SFT, responses are conditioned on their ground-truth attribute vector tag in the prompt, and at inference time or during online RL, attributes are set to maximum quality (Helpfulness=4, Correctness=4, Coherence=4) while explicitly controlling Verbosity to prevent length hacking.
References
- SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF ↗
- HelpSteer2: Open-source dataset for training top-performing reward models ↗
All 9 authors
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev