DeepSeek R-1 Model Overview and how it Ranks Versus OpenAI’s O1

DeepSeek is a Chinese AI company “devoted to making AGI a truth” and open-sourcing all its models. They started in 2023, but have been making waves over the past month or so, and specifically this previous week with the release of their two latest reasoning designs: DeepSeek-R1-Zero and the advanced DeepSeek-R1, likewise referred to as DeepSeek Reasoner.

They have actually launched not only the models but likewise the code and assessment triggers for public use, along with a detailed paper describing their technique.

Aside from producing 2 highly performant designs that are on par with OpenAI’s o1 model, the paper has a great deal of important details around support learning, chain of thought reasoning, timely engineering with reasoning designs, and more.

We’ll start by focusing on the training procedure of DeepSeek-R1-Zero, which uniquely relied solely on support learning, rather of standard monitored learning. We’ll then move on to DeepSeek-R1, how it’s thinking works, and some prompt engineering best practices for thinking designs.

Hey everybody, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s most current design release and comparing it with OpenAI’s thinking models, specifically the A1 and A1 Mini models. We’ll explore their training process, thinking abilities, and some essential insights into timely engineering for reasoning designs.

DeepSeek is a Chinese-based AI company dedicated to open-source advancement. Their current release, the R1 thinking design, is groundbreaking due to its open-source nature and ingenious training techniques. This consists of open access to the designs, triggers, and research study papers.

Released on January 20th, DeepSeek’s R1 accomplished impressive performance on numerous benchmarks, matching OpenAI’s A1 models. Notably, they also introduced a precursor model, R10, which functions as the structure for R1.

Training Process: R10 to R1

R10: This model was trained exclusively utilizing reinforcement learning without supervised fine-tuning, making it the first open-source model to attain high performance through this technique. Training involved:

– Rewarding correct answers in deterministic tasks (e.g., math problems).
– Encouraging structured reasoning outputs using templates with “” and “” tags

Through countless iterations, R10 developed longer reasoning chains, self-verification, and even reflective habits. For instance, throughout training, the model demonstrated “aha” minutes and self-correction habits, which are unusual in standard LLMs.

R1: Building on R10, R1 added numerous enhancements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated thinking chains.
– Human choice alignment for refined responses.
– Distillation into smaller models (LLaMA 3.1 and 3.3 at numerous sizes).

Performance Benchmarks

DeepSeek’s R1 design carries out on par with OpenAI’s A1 models across lots of reasoning criteria:

Reasoning and Math Tasks: R1 rivals or exceeds A1 models in accuracy and depth of thinking.
Coding Tasks: A1 designs typically perform much better in LiveCode Bench and CodeForces jobs.
Simple QA: R1 frequently exceeds A1 in structured QA jobs (e.g., 47% precision vs. 30%).

One notable finding is that longer thinking chains typically enhance performance. This lines up with insights from Microsoft’s Med-Prompt framework and OpenAI’s observations on test-time compute and reasoning depth.

Challenges and Observations

Despite its strengths, R1 has some constraints:

– Mixing English and Chinese reactions due to an absence of monitored fine-tuning.
– Less polished responses compared to chat models like OpenAI’s GPT.

These concerns were attended to throughout R1’s refinement procedure, including monitored fine-tuning and human feedback.

Prompt Engineering Insights

A fascinating takeaway from DeepSeek’s research study is how few-shot triggering degraded R1’s performance compared to zero-shot or concise customized prompts. This lines up with findings from the Med-Prompt paper and OpenAI’s suggestions to restrict context in thinking designs. Overcomplicating the input can overwhelm the design and decrease accuracy.

DeepSeek’s R1 is a substantial advance for open-source thinking designs, showing abilities that match OpenAI’s A1. It’s an interesting time to try out these designs and their chat interface, which is free to utilize.

If you have questions or desire to discover more, take a look at the resources linked below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only approach

DeepSeek-R1-Zero stands apart from most other cutting edge models because it was trained utilizing only reinforcement learning (RL), no supervised fine-tuning (SFT). This challenges the present traditional method and opens brand-new chances to train reasoning designs with less human intervention and effort.

DeepSeek-R1-Zero is the first open-source design to verify that advanced thinking capabilities can be established simply through RL.

Without pre-labeled datasets, the design learns through experimentation, refining its behavior, parameters, and weights based solely on feedback from the services it generates.

DeepSeek-R1-Zero is the base design for DeepSeek-R1.

The RL process for DeepSeek-R1-Zero

The training procedure for DeepSeek-R1-Zero involved providing the model with numerous thinking jobs, ranging from mathematics problems to abstract logic obstacles. The design produced outputs and was assessed based upon its performance.

DeepSeek-R1-Zero got feedback through a reward system that helped direct its learning process:

Accuracy rewards: Evaluates whether the output is proper. Used for when there are deterministic results (math problems).

Format rewards: Encouraged the model to structure its reasoning within and tags.

Training prompt design template

To train DeepSeek-R1-Zero to create structured chain of thought series, the researchers used the following prompt training template, replacing prompt with the thinking concern. You can access it in PromptHub here.

This design template triggered the model to clearly describe its idea procedure within tags before providing the final answer in tags.

The power of RL in thinking

With this training process DeepSeek-R1-Zero began to produce sophisticated reasoning chains.

Through countless training steps, DeepSeek-R1-Zero progressed to resolve increasingly complex issues. It found out to:

– Generate long thinking chains that made it possible for deeper and more structured problem-solving

– Perform self-verification to cross-check its own responses (more on this later).

– Correct its own errors, showcasing emerging self-reflective habits.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is mostly a precursor to DeepSeek-R1, it still attained high efficiency on several benchmarks. Let’s dive into some of the experiments ran.

Accuracy improvements during training

– Pass@1 precision began at 15.6% and by the end of the training it enhanced to 71.0%, equivalent to OpenAI’s o1-0912 design.

– The red solid line represents efficiency with majority voting (similar to ensembling and self-consistency strategies), which increased accuracy further to 86.7%, exceeding o1-0912.

Next we’ll look at a table comparing DeepSeek-R1-Zero’s efficiency throughout numerous reasoning datasets versus OpenAI’s reasoning designs.

AIME 2024: 71.0% Pass@1, a little listed below o1-0912 but above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

– Performed much worse on coding jobs (CodeForces and LiveCode Bench).

Next we’ll take a look at how the reaction length increased throughout the RL training procedure.

This chart shows the length of reactions from the design as the training procedure advances. Each “action” represents one cycle of the model’s learning process, where feedback is provided based upon the output’s performance, evaluated using the prompt template discussed earlier.

For each question (corresponding to one step), 16 actions were tested, and the typical accuracy was calculated to ensure steady assessment.

As training advances, the model creates longer reasoning chains, enabling it to solve progressively complex thinking tasks by leveraging more test-time calculate.

While longer chains do not constantly guarantee better outcomes, they typically correlate with enhanced performance-a pattern also observed in the MEDPROMPT paper (find out more about it here) and in the original o1 paper from OpenAI.

Aha moment and self-verification

Among the coolest aspects of DeepSeek-R1-Zero’s development (which likewise uses to the flagship R-1 design) is simply how good the model became at thinking. There were advanced thinking habits that were not explicitly programmed but occurred through its reinforcement learning process.

Over countless training actions, the model started to self-correct, review flawed reasoning, and validate its own solutions-all within its chain of idea

An example of this kept in mind in the paper, described as a the “Aha minute” is below in red text.

In this instance, the model actually said, “That’s an aha minute.” Through DeepSeek’s chat feature (their version of ChatGPT) this type of reasoning generally emerges with expressions like “Wait a minute” or “Wait, but … ,”

Limitations and difficulties in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to perform at a high level, there were some disadvantages with the design.

Language blending and coherence problems: The model occasionally produced actions that mixed languages (Chinese and English).

Reinforcement learning compromises: The lack of monitored fine-tuning (SFT) indicated that the model lacked the refinement needed for fully polished, human-aligned outputs.

DeepSeek-R1 was established to resolve these concerns!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning model from the Chinese AI lab DeepSeek. It builds on DeepSeek-R1-Zero, which was trained entirely with support learning. Unlike its predecessor, DeepSeek-R1 incorporates monitored fine-tuning, making it more refined. Notably, it outperforms OpenAI’s o1 model on several benchmarks-more on that later.

What are the primary differences between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 builds on the structure of DeepSeek-R1-Zero, which serves as the base model. The 2 differ in their training techniques and total performance.

1. Training approach

DeepSeek-R1-Zero: Trained entirely with support knowing (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes supervised fine-tuning (SFT) initially, followed by the very same support finding out procedure that DeepSeek-R1-Zero wet through. SFT assists enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Battled with language mixing (English and Chinese) and readability problems. Its reasoning was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these problems with cold-start fine-tuning, making actions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still an extremely strong thinking model, sometimes beating OpenAI’s o1, but fell the language blending concerns minimized usability considerably.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on a lot of reasoning standards, and the actions are a lot more polished.

Simply put, DeepSeek-R1-Zero was a proof of idea, while DeepSeek-R1 is the completely enhanced version.

How DeepSeek-R1 was trained

To tackle the readability and coherence problems of R1-Zero, the scientists included a cold-start fine-tuning phase and a multi-stage training pipeline when constructing DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers a top quality dataset of long chains of thought examples for initial supervised fine-tuning (SFT). This information was gathered utilizing:- Few-shot prompting with in-depth CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, fine-tuned by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the same RL procedure as DeepSeek-R1-Zero to improve its thinking abilities further.

Human Preference Alignment:

– A secondary RL stage improved the model’s helpfulness and harmlessness, ensuring better positioning with user requirements.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking capabilities were distilled into smaller, effective models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 benchmark performance

The scientists checked DeepSeek R-1 throughout a range of criteria and against top designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The standards were broken down into a number of classifications, revealed below in the table: English, Code, Math, and Chinese.

Setup

The following specifications were used across all designs:

Maximum generation length: 32,768 tokens.

Sampling setup:- Temperature: 0.6.

– Top-p worth: 0.95.

– DeepSeek R1 exceeded o1, Claude 3.5 Sonnet and other designs in the bulk of reasoning criteria.

o1 was the best-performing model in four out of the five coding-related benchmarks.

– DeepSeek performed well on imaginative and long-context task job, like AlpacaEval 2.0 and ArenaHard, outshining all other models.

Prompt Engineering with thinking designs

My favorite part of the article was the researchers’ observation about DeepSeek-R1’s level of sensitivity to triggers:

This is another datapoint that aligns with insights from our Prompt Engineering with Reasoning Models Guide, which references Microsoft’s research on their MedPrompt framework. In their research study with OpenAI’s o1-preview model, they found that overwhelming reasoning designs with few-shot context degraded performance-a sharp contrast to non-reasoning designs.

The key takeaway? Zero-shot triggering with clear and concise guidelines seem to be best when using reasoning designs.