Harel Lidar

Paper summary · May 2025

RLVR - DeepSeek R1

Summary of DeepSeek-R1 by DeepSeek

Opening (3 minutes)

Main Innovation - DeepSeek-R1-Zero (4 minutes)

Reference pages 3-9

The Multi-Stage Pipeline (5 minutes)

Reference pages 9-11

Stage 1: Cold Start (Section 2.3.1)

Stage 2: Reasoning-Oriented RL (Section 2.3.2)

Stage 3: Rejection Sampling + SFT (Section 2.3.3)

Stage 4: Full-Scenario RL (Section 2.3.4)

Results Deep Dive (3 minutes)

Reference Table 4 (page 13)

Distillation Results (2 minutes)

Reference Table 5 (page 14)

What Didn’t Work (2 minutes)

Reference Section 4.2 (pages 15-16)

  1. Process Reward Models:
    • Suffer from reward hacking
    • Hard to define fine-grained steps
    • Add complexity without big gains
  2. Monte Carlo Tree Search:
    • Works for inference, not iterative improvement
    • Search space too large compared to chess/Go
    • Value model too hard to train
  3. Pure RL limitations:
    • Poor readability
    • Language mixing issues
    • Needs careful prompt engineering

Key Insights & Philosophy (2 minutes)

Technical Insights:

Philosophical Questions:

Questions for me

  1. “If reasoning is emergent, what does this tell us about the nature of intelligence?”
  2. “Should we view the need for ‘thinking time’ as more human-like or as a limitation?”
  3. “What are the implications for AI safety if sophisticated behaviors emerge unexpectedly?”

Quick Reference Points to Emphasize: