Brief Review — DeepSeek-VL: Towards Real-World Vision-Language Understanding

Vision-Language Model for DeepSeek

Sik-Ho Tsang

5 min read

·

Aug 5, 2025

--

Listen

Share

DeepSeek-VL: Towards Real-World Vision-Language Understanding
DeepSeek-VL, by DeepSeek-AI
2024 arXiv v2, Over 480 Citations (

Sik-Ho Tsang

@ Medium)

Visual/Vision/Video Language Model (VLM)
2023 [GPT-4] [GPT-4V(ision)] [MultiModal-CoT] [CoCa] [Florence-2] [PaLI] [PaLI-X] [OpenCLIP] 2024 [MiniGPT-4]
==== My Other Paper Readings Are Also Over Here ====

• In this paper, DeepSeek-VL is proposed, which is a Vision-Language Model (VLM) for DeepSeek.
• The data comes from diverse, scalable and extensively covers real-world scenarios including web screenshots, PDFs, OCR, charts, and knowledge-based content (expert knowledge, textbooks).
• The model incorporates a hybrid vision encoder that efficiently processes high-resolution images (1024 x 1024) within a fixed token budget, while maintaining a relatively low computational overhead.
• An effective VL pretraining strategy is designed by integrating LLM training from the beginning and carefully managing the competitive dynamics observed between vision and language modalities.

Outline

1. DeepSeek-VL: Data
2. DeepSeek-VL: Model
3. DeepSeek-VL: Results
4. DeepSeek-VL: Some Use Cases

1. DeepSeek-VL: Data

1.1. Pretraining Data

• The pretraining dataset utilized in our study encompasses a diverse range of publicly accessible sources, in addition to a selection of proprietary data.

1. Interleaved image-text data enable the models to have a better capability for in-context learning of multi-modality inputs, and we utilize three public datasets.
2. Image caption data come from three high-quality image-text paired datasets.
3. Table and chart data enable the models to learn the capability for general table and chart image understanding.
4. Web Code data empowers models with the capability to reconstruct code from graphical interfaces or visual plots.
5. Document Optical Character Recognition (OCR) data facilitates the recognition of optical characters at the document level, even in challenging real-world scenarios.
6. Scene text OCR data augment the capability of the model to recognize and extract text from images in which the text is integrated into the environment.
7. Text-only corpus serves to maintain proficiency in language-centric tasks.

1.2. Supervised Fine-Tuning (SFT) Data

• To enhance the quality of our multi-modality SFT data, a portion of high-quality in-house multi-modality SFT data is also curated, some of which are in the Chinese language. The proposed in-house instruction-tuning dataset is meticulously designed to reflect real-world usage scenarios and cover a wide range of tasks.
• This structured taxonomy serves as a guideline for selecting representative prompts for each test image, ensuring that our instruction-tuning dataset is both practical and relevant to real-world applications. Moreover, this taxonomy is also employed to construct a balanced and comprehensive evaluation dataset.

2. DeepSeek-VL: Model

• The system contains three modules: a hybrid vision encoder, a vision adaptor, and a language model.

2.1. Hybrid Vision Encoder

• A vision-only encoder based on the SAM-B is additionally utilized, a pre-trained ViTDet image encoder to process low-level features, which accepts high-resolution 1024 x 1024 image inputs.
• In addition to the SAM-B encoder, the SigLIP-L vision encoder is used with low-resolution 384 x 384 image inputs.

2.2. Vision-Language Adaptor

• The VL Adaptor initially interpolates it into a size of 96 x 96 x 256. Subsequently, it employs two convolutional layers with a stride of 2, producing a feature map of 24 x 24 x 1024, and reshapes it to 576 x 1024.
• Alongside this, the low-resolution feature map of size 576 x 1024 generated by SigLIP-L is concatenated with the high-resolution features, resulting in 576 visual tokens with 2048 dimensions.
• Then they undergo GeLU activation and are directed through an embedding layer to establish a connection with the language model.

2.3. Language Model

• The language model is built upon DeepSeek LLM.

2.4. Training Pipeline

• The primary objective of stage 1, in which both the vision encoder and the LLM remain frozen during this stage, is to establish a conceptual link between visual and linguistic elements within the embedding space.
• During stage-2 training, not only multimodal data is engaged in training but also a large proportion of language data is incorporated. This approach aims to balance the training focus, mitigating the adverse effects observed.
• In stage 3, the pretrained DeepSeek-VL model is fine-tuned with instruction-based fine-tuning to bolster its ability to follow instructions and engage in dialogue, culminating in the creation of the interactive DeepSeek-VL-Chat model.

3. DeepSeek-VL: Results

• DeepSeek-VL-7B surpasses most open-source models of similar size across a wide range of benchmarks.
• DeepSeek-VL outperforms open-source models of similar size in benchmarks such as MMB, MMC, and SEEDbench, even approaching proprietary models (DeepSeek-VL vs. GPT-4V = 70.4 vs. 71.6 on seedbench),

• As shown in Table 6, DeepSeek-VL-1.3B significantly outperforms models of comparable size.

• Across the majority of language benchmarks, DeepSeek-VL performs comparably to, or even surpasses, DeepSeek-7B.

• DeepSeek-VL-7B achieves better results in overall performance, reaching outcomes close to GPT-4V in Recognition, Conversion, and Commonsense Reasoning.

• DeepSeek-VL is judged to be superior in over 60% of instances when compared to opensource multimodal models, including Fuyu-8B, CogVLM-17B, and InternLM-XComposer2-VL.
• Moreover, in comparison with other proprietary models, such as GPT-4V itself, DeepSeek-VL demonstrates comparably exceptional performance.