Mechanism

Formulates long-CoT RLVR as Online Mirror Descent with analytical partition function log⁡Z(x)≈log⁡1G∑i=1Gexp⁡(ri/τ)\log Z(x) \approx \log \frac{1}{G}\sum_{i=1}^G \exp(r_i / \tau), optimizing the squared score-matching loss between the log-policy ratio and the centered reward rˉ=1G∑i=1Gri\bar{r} = \frac{1}{G}\sum_{i=1}^G r_i: ∇θLKimi-OMD(θ)=−1G∑i=1G(ri−rˉ−τlog⁡πθ(yi∣x)πθold(yi∣x))∇θlog⁡πθ(yi∣x)\nabla_\theta \mathcal{L}_{\text{Kimi-OMD}}(\theta) = -\frac{1}{G}\sum_{i=1}^G \left(r_i - \bar{r} - \tau \log \frac{\pi_\theta(y_i\mid x)}{\pi_{\theta_{\text{old}}}(y_i\mid x)}\right)\nabla_\theta \log \pi_\theta(y_i\mid x) Paired with a Length Penalty Curriculum: rlen(i)=λ(0.5−∣yi∣−min⁡j∣yj∣max⁡j∣yj∣−min⁡j∣yj∣)(if correct)r_{\text{len}}(i) = \lambda \left(0.5 - \frac{|y_i| - \min_j |y_j|}{\max_j |y_j| - \min_j |y_j|}\right) \quad (\text{if correct}) and Long2Short methods (Model Merging, Shortest Rejection Sampling, Long2Short DPO, and Long2Short RL) to transfer long-CoT reasoning efficiency into fast short-CoT models.

References

  1. Kimi k1.5: Scaling Reinforcement Learning with LLMs ↗

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen et al. · 2025

    All 96 authors

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Weixin Xu, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, Zonghan Yang, Zongyu Lin