Brief Review — DeepSeek-Coder: When the Large Language Model Meets Programming — The Rise of Code Intelligence
DeepSeek-Coder, DeepSeek for Coding
DeepSeek-Coder: When the Large Language Model Meets Programming — The Rise of Code Intelligence
DeepSeek-Coder, by Peking University, DeepSeek
2024 arXiv v2, Over 940 Citations ( @ Medium)Large Langauge Model (LLM)
2020 … 2023 [GPT-4] [LLaMA] [Koala] [BloombergGPT] [GLM-130B] [UL2] [PaLM 2] [Llama 2] [MultiMedQA, HealthSearchQA, Med-PaLM] [Med-PaLM 2] [Flan 2022, Flan-T5] [AlphaCode 2] [Mistral 7B] [Alpaca] [Inflection-1] 2024 [Nemotron-4 15B] [DeepSeek-v1] [DeepSeekMoE]
==== My Other Paper Readings Are Also Over Here ====
- DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens, is introduced.
- These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling.
Outline
- DeepSeek-Coder: Data
- DeepSeek-Coder: Model
- Results
1. DeepSeek-Coder: Data
1.1. Training Dataset
- The training dataset of DeepSeek-Coder is composed of 87% source code, 10% English coderelated natural language corpus, and 3% code-unrelated Chinese natural language corpus.
- English corpus consists of materials from GitHub’s Markdown and StackExchange, for the understanding of code-related concepts and improve its ability to handle tasks like library usage and bug fixing.
- Meanwhile, the Chinese corpus consists of high-quality articles aimed at improving the model’s proficiency in understanding the Chinese language.
1.2. GitHub Data Crawling and Filtering
- Public repositories created before February 2023 on GitHub and retain only 87 programming languages, are collected, as above.
- Filtering rules similar to those used in the StarCoder project (Li et al., 2023) are used to preliminarily filter out lower-quality code. By applying these filtering rules, the total amount of data is reduced to only 32.8% of its original size.
- Firstly, files with an average line length exceeding 100 characters or a maximum line length surpassing 1000 characters, are filtered out.
- Additionally, files with fewer than 25% alphabetic characters are removed. Except for the XSLT programming language, files where the string “<?xml version=” appeared in the first 100 characters are further filtered out.
- For HTML files, the ratio of visible text to HTML code is considered. Files where the visible text constitutes at least 20% of the code and is no less than 100 characters are retained.
- For JSON and YAML files, which typically contain more data, only files that have a character count ranging from 50 to 5000 characters are kept.
- This effectively removes most data-heavy files.
1.3. Dependency Parsing
- In previous works, large language models for code are mainly pre-trained on file-level source code, which ignores the dependencies between different files in a project. However, in practical applications, such models struggle to effectively scale to handle entire project-level code scenarios.
- Here, the dependencies between files are first parsed and then these files are arranged based on order. e.g.: regular expressions are used to extract them, such as “import” in Python, “using” in C#, and “include” in C.
1.4. Repo-Level Deduplication
- Deduplicating training datasets often brings significant performance improvements.
- In this paper, deduplication is performed at the repository level of code. The concatenated code from the repository level is treated as a single sample and near-deduplication algorithm is applied.
1.5. Quality Screening and Decontamination
- A compiler and a quality model, combined with heuristic rules, are employed, to further filter out low-quality data.
- The total data volume is 798 GB with 603 million files.
- n-gram filtering process is also used to ensure that our code training data is not contaminated by information from the test set.
2. DeepSeek-Coder: Model
2.1. Training Strategy
- The first training objective is the popular next token prediction.
- The second training objective is known as fill-in-the-middle since it is often necessary to generate corresponding inserted content based on the given context and subsequent text.
- Several approaches (Bavarian et al., 2022; Li et al., 2023) propose the pretraining method of Fill-in-the-Middle (FIM). This approach involves randomly dividing the text into three parts, then shuffling the order of these parts and connecting them with special characters.
- Within the FIM methodology, two distinct modes are employed: PSM (Prefix-Suffix-Middle) and SPM (Suffix-Prefix-Middle). In the PSM mode, the training corpus is organized in the sequence of 𝑃𝑟𝑒𝑓 𝑖𝑥, 𝑆𝑢 𝑓 𝑓 𝑖𝑥, 𝑀𝑖𝑑𝑑𝑙𝑒. Conversely, the SPM mode arranges the segments as 𝑆𝑢 𝑓 𝑓 𝑖𝑥, 𝑃𝑟𝑒 𝑓 𝑖𝑥, 𝑀𝑖𝑑𝑑𝑙𝑒.
- Weakest code completion capability is obtained when a 100% FIM rate is used.
- With a 50% PSM rate, the model outperforms the MSP strategy by T5.
2.2. Instruction Tuning
- DeepSeek-Coder-Instruct is developed by enhancing the DeepSeek-Coder-Base through instruction-based fine-tuning using high-quality data. This data comprises helpful and impartial human instructions structured by the Alpaca Instruction format.
- A unique delimiter token <|EOT|> is used to signify the conclusion of each segment.
- This example is a multi-turn dialogue scenario for building a snake game. Initially, authors ask the model to write a game snake using pygame. To improve the game, authors further request adding a scoring system in the top left corner.
3. Results
- HumanEval consists of 164 hand-written Python problems.
- MBPP benchmark includes 500 problems in a few-shot setting.
DeepSeek-Coder-Base achieves state-of- the-art performance with an average accuracy of 50.3% on HumanEval and 66.0% on MBPP.
In comparison to the similarly sized open-source model CodeLlama-Base 34B, the proposed model has demonstrated a notable improvement of 9% and 11% in accuracy, respectively.
- DS-1000 offers a comprehensive collection of 1,000 practical and realistic data science workflows across seven different libraries.
DeepSeek-Coder model achieves relatively high accuracy in all libraries, demonstrating that the model is not only capable of generating good code but also of using libraries more accurately in real data science workflows.
- Authors construct the LeetCode Contest benchmark for real-world programming problems.
- A total of 180 problems were collected from July 2023 to January 2024. For each problem, 100 test cases are collected to ensure the test coverage.
DeepSeek-Coder-Instruct 6.7B and 33B achieve Pass@1 scores of 19.4% and 27.8% respectively in this benchmark. This performance notably surpasses existing open-sourced models such as Code-Llama-33B. The DeepSeek-Coder-Instruct 33B is the only open-sourced model that outperforms OpenAI’s GPT-3.5-Turbo in this task.
However, there remains a substantial performance gap comparing with more advanced GPT-4-Turbo.
- By adding the directive, “You need first to write a step-by-step outline and then write the code.” following the initial prompt, enhancements in performance are observed using Chain-of-Though (CoT) prompting.
- The benchmark for this comparison was the Single-Line Infilling benchmarks.
DeepSeek-Coder outperforms its larger counterparts, StarCoder and CodeLlama, in these benchmarks. As the model size increases, there is a corresponding and responsible enhancement in performance.
- The performance of existing open-source models in cross-file code completion tasks is evaluated.
DeepSeek-Coder consistently outperforms other models in cross-file completion tasks across multiple languages, showcasing its superior practical application capabilities.
- To further enhance the natural language understanding and mathematical reasoning abilities of the DeepSeek-Coder model, additional pre-training is performed from the general language model DeepSeek-LLM-7B Base.
- Unlike DeepSeek-Coder, DeepSeek-Coder-v1.5 employs solely a next token prediction objective with a 4K context length during its pre-training phase.
DeepSeek-Coder-Base-v1.5 model, despite a slight decrease in coding performance, shows marked improvements across most tasks when compared to the DeepSeek-Coder-Base model.
