Brief Review — DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
DeepSeek for Mathematical Reasoning
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
DeepSeekMath, by DeepSeek
2024 arXiv v3, Over 1300 Citations ( @ Medium)Large Langauge Model (LLM)
2020 … 2023 [GPT-4] [LLaMA] [Koala] [BloombergGPT] [GLM-130B] [UL2] [PaLM 2] [Llama 2] [MultiMedQA, HealthSearchQA, Med-PaLM] [Med-PaLM 2] [Flan 2022, Flan-T5] [AlphaCode 2] [Mistral 7B] [Alpaca] [Inflection-1] 2024 [Nemotron-4 15B] [DeepSeek-v1] [DeepSeekMoE] [DeepSeek-Coder]
==== My Other Paper Readings Are Also Over Here ====
- Mathematical reasoning poses a significant challenge for language models.
- DeepSeekMath 7B is introduced, which continues pretraining DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data.
Outline
- DeepSeekMath: Data
- DeepSeekMath: Model Training
- Results
1. DeepSeekMath: Data
- DeepSeekMath Corpus is constructed from Common Crawl, which is a high-quality dataset of 120B tokens from web pages filtered for mathematical content, almost 7 times the size of the math web pages used by Minerva and 9 times the size of the recently released OpenWebMath.
- An iterative pipeline is used with the aid of fastText systematically to gather a large-scale mathematical corpus as shown above.
- A 1.3B DeepSeek is trained on mathematical corpus for 150B tokens, and we can see that there is a clear performance lead of the model trained on the DeepSeekMath Corpus.
2. DeepSeekMath: Model Training
2.1. Training and Evaluating DeepSeekMath-Base 7B
- DeepSeekMath-Base 7B is initialized with DeepSeek-Coder-Base-v1.5 7B and trained for 500B tokens.
- DeepSeekMath-Base’s performance of solving mathematical problems is evaluated using few-shot chain-of-thought prompting, across eight benchmarks in English and Chinese.
As shown in Table 2, DeepSeekMath-Base 7B leads in performance across all eight benchmarks among the open-source base models, e.g.: Mistral 7B and and the Llemma 34B (Azerbayev et al., 2023).
As shown in Table 3, DeepSeekMath-Base 7B outperforms the prior state-of-the-art Llemma 34B, and demonstrates strong performance in proof autoformalization.
DeepSeekMath-Base 7B effectively maintains the performance of DeepSeek-Coder-Base-v1.5 on the two coding benchmarks. Overall, DeepSeekMath-Base 7B significantly outperforms the general model Mistral 7B on the three reasoning and coding benchmarks.
2.2. Supervised Fine-Tuning (SFT)
- A mathematical instruction-tuning dataset is constructed covering English and Chinese problems. The total number of training examples is 776K.
- Training examples are randomly concatenated until reaching a maximum context length of 4K tokens.
- The model is trained for 500 steps.
As shown in Table 5, under the evaluation setting where tool use is disallowed, DeepSeekMath-Instruct 7B demonstrates strong performance of step-by-step reasoning.
2.3. Reinforcement Learning
- Reinforcement learning (RL) has been proven to be effective in further improving the mathematical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage.
- Proximal Policy Optimization (PPO) (Schulman et al., 2017) is an actor-critic RL algorithm that is widely used in the RL fine-tuning stage of LLMs (Ouyang et al., 2022).
- As the value function employed in PPO is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden. Additionally, during RL training, the value function is treated as a baseline in the calculation of the advantage for variance reduction.
- Group Relative Policy Optimization (GRPO), which obviates the need for additional value function approximation as in PPO, and instead uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline.
- Also note that, instead of adding KL penalty in the reward, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of ˆ𝐴𝑖,𝑡.
In Table 5 above, DeepSeekMath-RL 7B attains accuracies of 88.2% and 51.7% on GSM8K and MATH, respectively, utilizing chain-of-thought reasoning. This performance surpasses that of all open-source models in the 7B to 70B range, as well as the majority of closed-source models.
Crucially, DeepSeekMath-RL 7B is only trained on chain-of-thought-format instruction tuning data of GSM8K and MATH, starting from DeepSeekMath-Instruct 7B.
