Reasoning

Incentivizing LLM Reasoning via Pure Reinforcement Learning

Original: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

By DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, Zhen Zhang Published 2025-01-22

Listen to the summary

Uses a voice available on your device

Speed Voice

On this page 4 sections

Related concepts 8 concepts

Key Takeaways

Applied pure reinforcement learning directly on DeepSeek-V3-Base without any prior supervised fine-tuning phase.
Achieved an increase in the average pass@1 score from an initial 15.6% to 77.9% on the AIME 2024 benchmark, reaching 86.7% using self-consistency decoding.
Developed a multi-stage training framework combining a cold-start conversational thinking process, language consistency rewards, rejection sampling, and reward models.
Distilled smaller open-source models from large-scale reasoning models to successfully transfer strong reasoning capabilities.

Summary & Methodology Analysis

Large language models traditionally rely heavily on extensive human-annotated demonstrations to achieve advanced reasoning capabilities. This dependence limits their scalability, introduces cognitive biases, and caps their performance based on human exemplars. To address this, the authors adopted Group Relative Policy Optimization (GRPO), a reinforcement learning framework that samples groups of outputs and optimizes policy models without requiring a value model. Traditional paradigms instead use Proximal Policy Optimization (PPO), which relies on reinforcement learning from human feedback, and supervised fine-tuning (SFT), which involves training on human-curated input-output pairs. The novel contribution of this work includes applying pure reinforcement learning directly on DeepSeek-V3-Base without any prior supervised fine-tuning phase, utilizing rule-based accuracy and format rewards to incentivize emergent reasoning patterns.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the core problem addressed by the paper?

Large language models often rely heavily on extensive human-annotated demonstrations to achieve advanced reasoning capabilities, which limits their scalability, introduces cognitive biases, and caps their performance based on human exemplars.

Q2. What is DeepSeek-R1?

DeepSeek-R1 is a large-scale reasoning model developed through a multi-stage training framework that integrates a cold-start conversational thinking process, a first reinforcement learning training stage, rejection sampling, supervised fine-tuning, and a secondary reinforcement learning stage.

Q3. What was the key result on the AIME 2024 benchmark?

On the AIME 2024 benchmark, the average pass@1 score of DeepSeek-R1-Zero increased from an initial 15.6% to 77.9% throughout reinforcement learning training, and reached 86.7% using self-consistency decoding.

Q4. What reinforcement learning framework is adopted in the paper?

Group Relative Policy Optimization (GRPO) is adopted as the reinforcement learning framework to sample groups of outputs and optimize policy models without requiring a value model.

Q5. What training methods were used in conventional paradigms?

Proximal Policy Optimization (PPO) and supervised fine-tuning (SFT) are used in conventional paradigms.

Q6. What are some limitations of DeepSeek-R1-Zero?

DeepSeek-R1-Zero faces challenges such as poor readability and language mixing, occasionally combining English and Chinese within a single chain-of-thought response, and its rule-based reinforcement learning training stage is narrowly focused on reasoning tasks.

Q7. Can DeepSeek-R1 leverage external tools?

No, DeepSeek-R1 cannot leverage tools, such as search engines and calculators, to improve the performance of output.

Q8. How does DeepSeek-R1 handle multilingual queries?

DeepSeek-R1 is currently optimized for Chinese and English, which may result in language mixing issues when handling queries in other languages.

Q9. What models or datasets are mentioned in connection with distillation?

The paper mentions distilling smaller open-source models, including DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Qwen-14B, DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Llama-70B, from the large-scale reasoning models to transfer strong reasoning capabilities.

Flag an issue