DeepSeek R-1 Model Overview and how it Ranks Versus OpenAI’s O1

DeepSeek is a Chinese AI company “dedicated to making AGI a truth” and open-sourcing all its designs. They began in 2023, but have actually been making waves over the past month approximately, and especially this previous week with the release of their 2 most current reasoning models: DeepSeek-R1-Zero and the more advanced DeepSeek-R1, also referred to as DeepSeek Reasoner.

They have actually launched not just the models but likewise the code and assessment prompts for public usage, along with a detailed paper detailing their method.

Aside from creating 2 extremely performant models that are on par with OpenAI’s o1 model, the paper has a great deal of important info around reinforcement learning, chain of idea reasoning, prompt engineering with reasoning designs, and more.

We’ll start by concentrating on the training procedure of DeepSeek-R1-Zero, which distinctively relied entirely on support learning, rather of conventional monitored knowing. We’ll then move on to DeepSeek-R1, how it’s reasoning works, and some timely engineering finest practices for thinking models.

Hey everybody, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s newest model release and comparing it with OpenAI’s thinking designs, particularly the A1 and A1 Mini models. We’ll explore their training process, thinking capabilities, and some key insights into prompt engineering for reasoning models.

DeepSeek is a Chinese-based AI business dedicated to open-source advancement. Their recent release, the R1 thinking design, is groundbreaking due to its open-source nature and ingenious training methods. This includes open access to the designs, prompts, and research papers.

Released on January 20th, DeepSeek’s R1 accomplished remarkable performance on numerous standards, measuring up to OpenAI’s A1 models. Notably, they also released a precursor model, R10, which acts as the structure for R1.

Training Process: R10 to R1

R10: This model was trained exclusively using reinforcement learning without supervised fine-tuning, making it the first open-source design to accomplish high efficiency through this technique. Training involved:

– Rewarding correct answers in deterministic tasks (e.g., mathematics issues).
– Encouraging structured reasoning outputs utilizing templates with “” and “” tags

Through thousands of models, R10 established longer reasoning chains, self-verification, and even reflective behaviors. For example, during training, the design demonstrated “aha” moments and self-correction behaviors, which are unusual in traditional LLMs.

R1: Building on R10, R1 included several enhancements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human preference positioning for sleek reactions.
– Distillation into smaller models (LLaMA 3.1 and 3.3 at different sizes).

Performance Benchmarks

DeepSeek’s R1 design carries out on par with OpenAI’s A1 models across lots of reasoning benchmarks:

Reasoning and Math Tasks: R1 rivals or outshines A1 designs in precision and depth of thinking.
Coding Tasks: A1 designs typically carry out better in LiveCode Bench and CodeForces jobs.
Simple QA: R1 typically exceeds A1 in structured QA tasks (e.g., 47% precision vs. 30%).

One significant finding is that longer thinking chains generally enhance performance. This aligns with insights from Microsoft’s Med-Prompt framework and OpenAI’s observations on test-time calculate and thinking depth.

Challenges and Observations

Despite its strengths, R1 has some limitations:

– Mixing English and Chinese reactions due to a lack of supervised fine-tuning.
– Less sleek reactions compared to chat models like OpenAI’s GPT.

These concerns were attended to throughout R1’s improvement process, including supervised fine-tuning and human feedback.

Prompt Engineering Insights

An interesting takeaway from DeepSeek’s research is how few-shot triggering abject R1’s efficiency compared to zero-shot or concise tailored prompts. This aligns with findings from the Med-Prompt paper and OpenAI’s recommendations to limit context in thinking models. Overcomplicating the input can overwhelm the model and minimize accuracy.

DeepSeek’s R1 is a substantial advance for open-source reasoning models, showing abilities that measure up to OpenAI’s A1. It’s an interesting time to explore these models and their chat user interface, which is complimentary to use.

If you have questions or want to discover more, take a look at the resources linked listed below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only approach

DeepSeek-R1-Zero stands out from the majority of other cutting edge designs due to the fact that it was trained using just support knowing (RL), no supervised fine-tuning (SFT). This challenges the current conventional technique and opens brand-new chances to train reasoning designs with less human intervention and effort.

DeepSeek-R1-Zero is the very first open-source model to verify that advanced reasoning capabilities can be established purely through RL.

Without pre-labeled datasets, the model finds out through experimentation, improving its behavior, parameters, and weights based solely on feedback from the solutions it generates.

DeepSeek-R1-Zero is the base model for DeepSeek-R1.

The RL process for DeepSeek-R1-Zero

The training process for DeepSeek-R1-Zero included presenting the design with various reasoning jobs, ranging from mathematics issues to abstract reasoning difficulties. The design created outputs and was evaluated based on its performance.

DeepSeek-R1-Zero received feedback through a benefit system that helped assist its learning process:

Accuracy rewards: Evaluates whether the output is correct. Used for when there are deterministic outcomes (mathematics issues).

Format benefits: Encouraged the model to structure its reasoning within and tags.

Training prompt template

To train DeepSeek-R1-Zero to create structured chain of thought sequences, the scientists used the following timely training design template, changing timely with the reasoning question. You can access it in PromptHub here.

This template triggered the model to clearly outline its idea procedure within tags before providing the last response in tags.

The power of RL in reasoning

With this training process DeepSeek-R1-Zero began to produce advanced reasoning chains.

Through thousands of training steps, DeepSeek-R1-Zero evolved to resolve progressively complex issues. It discovered to:

– Generate long reasoning chains that enabled deeper and more structured problem-solving

– Perform self-verification to cross-check its own responses (more on this later).

– Correct its own mistakes, showcasing emerging self-reflective habits.

DeepSeek R1-Zero efficiency

While DeepSeek-R1-Zero is primarily a precursor to DeepSeek-R1, it still accomplished high performance on numerous standards. Let’s dive into a few of the experiments ran.

Accuracy improvements throughout training

– Pass@1 accuracy started at 15.6% and by the end of the training it improved to 71.0%, equivalent to OpenAI’s o1-0912 design.

– The red strong line represents efficiency with majority voting (similar to ensembling and self-consistency techniques), which increased precision further to 86.7%, going beyond o1-0912.

Next we’ll take a look at a table comparing DeepSeek-R1-Zero’s performance across several thinking datasets versus OpenAI’s thinking models.

AIME 2024: 71.0% Pass@1, a little listed below o1-0912 however above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a rating of 73.3%.

– Performed much worse on coding jobs (CodeForces and LiveCode Bench).

Next we’ll take a look at how the response length increased throughout the RL training procedure.

This graph reveals the length of reactions from the design as the training process advances. Each “step” represents one cycle of the design’s knowing procedure, where feedback is supplied based upon the output’s performance, examined using the prompt template discussed previously.

For each concern (representing one action), 16 actions were sampled, and the was calculated to make sure stable evaluation.

As training progresses, the model generates longer thinking chains, enabling it to solve increasingly complex thinking tasks by leveraging more test-time compute.

While longer chains don’t constantly guarantee much better results, they usually correlate with improved performance-a pattern also observed in the MEDPROMPT paper (check out more about it here) and in the original o1 paper from OpenAI.

Aha moment and self-verification

One of the coolest elements of DeepSeek-R1-Zero’s advancement (which likewise applies to the flagship R-1 model) is simply how excellent the design became at thinking. There were sophisticated reasoning habits that were not clearly configured however emerged through its support discovering process.

Over thousands of training steps, the design started to self-correct, review problematic reasoning, and verify its own solutions-all within its chain of idea

An example of this kept in mind in the paper, described as a the “Aha minute” is below in red text.

In this circumstances, the design actually said, “That’s an aha minute.” Through DeepSeek’s chat function (their version of ChatGPT) this type of thinking usually emerges with phrases like “Wait a minute” or “Wait, but … ,”

Limitations and difficulties in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to carry out at a high level, there were some drawbacks with the model.

Language mixing and coherence problems: The model periodically produced responses that mixed languages (Chinese and English).

Reinforcement knowing trade-offs: The absence of monitored fine-tuning (SFT) indicated that the model did not have the refinement needed for fully polished, human-aligned outputs.

DeepSeek-R1 was developed to address these issues!

What is DeepSeek R1

DeepSeek-R1 is an open-source thinking design from the Chinese AI lab DeepSeek. It constructs on DeepSeek-R1-Zero, which was trained entirely with support knowing. Unlike its predecessor, DeepSeek-R1 integrates monitored fine-tuning, making it more refined. Notably, it exceeds OpenAI’s o1 design on several benchmarks-more on that later.

What are the main distinctions in between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 develops on the structure of DeepSeek-R1-Zero, which acts as the base model. The two differ in their training techniques and total efficiency.

1. Training method

DeepSeek-R1-Zero: Trained entirely with support knowing (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that consists of supervised fine-tuning (SFT) initially, followed by the exact same support discovering procedure that DeepSeek-R1-Zero damp through. SFT helps enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Struggled with language mixing (English and Chinese) and readability problems. Its thinking was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these concerns with cold-start fine-tuning, making responses clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still a very strong thinking model, in some cases beating OpenAI’s o1, but fell the language blending problems decreased use significantly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on a lot of reasoning benchmarks, and the responses are far more polished.

Simply put, DeepSeek-R1-Zero was a proof of concept, while DeepSeek-R1 is the fully enhanced version.

How DeepSeek-R1 was trained

To deal with the readability and coherence issues of R1-Zero, the scientists integrated a cold-start fine-tuning stage and a multi-stage training pipeline when building DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a high-quality dataset of long chains of idea examples for initial monitored fine-tuning (SFT). This information was gathered utilizing:- Few-shot prompting with detailed CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, fine-tuned by human annotators.

Reinforcement Learning:

DeepSeek-R1 went through the very same RL procedure as DeepSeek-R1-Zero to improve its thinking capabilities further.

Human Preference Alignment:

– A secondary RL phase enhanced the design’s helpfulness and harmlessness, guaranteeing much better alignment with user needs.

Distillation to Smaller Models:

– DeepSeek-R1’s reasoning capabilities were distilled into smaller sized, effective designs like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 criteria efficiency

The scientists evaluated DeepSeek R-1 throughout a variety of criteria and against leading designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The benchmarks were broken down into a number of categories, shown below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were applied across all designs:

Maximum generation length: 32,768 tokens.

Sampling configuration:- Temperature: 0.6.

– Top-p worth: 0.95.

– DeepSeek R1 surpassed o1, Claude 3.5 Sonnet and other designs in the bulk of reasoning standards.

o1 was the best-performing design in 4 out of the five coding-related benchmarks.

– DeepSeek performed well on creative and long-context task job, like AlpacaEval 2.0 and ArenaHard, outshining all other models.

Prompt Engineering with reasoning models

My preferred part of the post was the researchers’ observation about DeepSeek-R1’s level of sensitivity to triggers:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which references Microsoft’s research study on their MedPrompt framework. In their study with OpenAI’s o1-preview model, they found that frustrating reasoning models with few-shot context broken down performance-a sharp contrast to non-reasoning models.

The crucial takeaway? Zero-shot triggering with clear and concise instructions appear to be best when using reasoning designs.