Mechanism

Samples NN candidate completions {y1,…,yN}∼πθt(⋅∣x)\{y_1, \dots, y_N\} \sim \pi_{\theta_t}(\cdot\mid x) at temperature T∈[0.7,1.0]T \in [0.7, 1.0], evaluates all NN candidates with a reward model or executable verifier, selects the highest-scoring (and shortest/cleanest among ties) candidate y⋆y^\star, and performs standard SFT on the filtered dataset {(x,y⋆)}\{(x, y^\star)\}.

Practical Notes

The single most reliable, bug-resistant post-training workhorse in industrial frontier labs; DeepSeek-R1 uses RS-SFT as the bridge between pure Reasoning RL and general-purpose alignment.

References

  1. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment ↗

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan et al. · 2023

    All 10 authors

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang

  2. Llama 2: Open Foundation and Fine-Tuned Chat Models ↗

    Touvron, Hugo, Martin, Louis, Stone, Kevin, Albert, Peter, Almahairi, Amjad, Babaei, Yasmine et al. · 2023

    All 68 authors

    Touvron, Hugo, Martin, Louis, Stone, Kevin, Albert, Peter, Almahairi, Amjad, Babaei, Yasmine, Bashlykov, Nikolay, Batra, Soumya, Bhargava, Prajjwal, Bhosale, Shruti, Bikel, Dan, Blecher, Lukas, Ferrer, Cristian Canton, Chen, Moya, Cucurull, Guillem, Esiobu, David, Fernandes, Jude, Fu, Jeremy, Fu, Wenyin, Fuller, Brian, Gao, Cynthia, Goswami, Vedanuj, Goyal, Naman, Hartshorn, Anthony, Hosseini, Saghar, Hou, Rui, Inan, Hakan, Kardas, Marcin, Kerkez, Viktor, Khabsa, Madian, Kloumann, Isabel, Korenev, Artem, Koura, Punit Singh, Lachaux, Marie-Anne, Lavril, Thibaut, Lee, Jenya, Liskovich, Diana, Lu, Yinghai, Mao, Yuning, Martinet, Xavier, Mihaylov, Todor, Mishra, Pushkar, Molybog, Igor, Nie, Yixin, Poulton, Andrew, Reizenstein, Jeremy, Rungta, Rashi, Saladi, Kalyan, Schelten, Alan, Silva, Ruan, Smith, Eric Michael, Subramanian, Ranjan, Tan, Xiaoqing Ellen, Tang, Binh, Taylor, Ross, Williams, Adina, Kuan, Jian Xiang, Xu, Puxin, Yan, Zheng, Zarov, Iliyan, Zhang, Yuchen, Fan, Angela, Kambadur, Melanie, Narang, Sharan, Rodriguez, Aurelien, Stojnic, Robert, Edunov, Sergey, Scialom, Thomas