Beyond an algorithm name
An RL training recipe includes how prompts are sampled, how responses are generated, how rewards are assigned, and how policy updates are scheduled. DAPO presents an open-source large-scale LLM RL system and training choices. Reading the complete recipe is more informative than attributing a result to one acronym.
Watch the distribution of useful experience
If sampled responses have nearly identical rewards, a batch may provide little relative learning signal. Track the proportion of informative samples, output lengths, truncation, invalid answers, and reward variance. A mean reward alone can hide changes in which examples actually drive updates.
Keep systems improvements visible
Faster rollouts can allow more experience or more experiments within a budget. Report both throughput and learning outcomes. A systems improvement may be valuable even if the policy objective is unchanged, but it should not be mistaken for an isolated algorithmic advantage.
Publish a reproducible training contract
Record model and data revisions, objective settings, sequence limits, hardware, rollout policy, reward code, and evaluation checkpoints. Match inference budgets for the final comparison, then test transfer beyond the reward’s own task family. Treat reproducibility as part of the evidence for the claimed improvement.
References
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale ↗
All 35 authors
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, Mingxuan Wang
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models ↗
All 11 authors
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo