Harel Lidar

Paper summary · May 2025

ProRL

Summary of ProRL by Minjie Liu, Yi Dong · paper ↗

TL;DR

1. Problem Statement

  1. Over reliance on specialized domains, like math, where models are overtrained during pre/post-training, restricting the potential for exploration
  2. The premature termination of RL training before models can fully explore and develop new reasoning capabilities based on a limited amount of RL training, typically no more than hundreds of steps.

2. Methodology / Architecture

  1. They introduce ProRL, a recipe designed to enable extended RL training periods that facilitate deeper exploration of reasoning strategies.
    1. This method demonstrates continued performance improvements after an unprecedented 2k training steps, suggesting that RL training scales effectively with increased compute
  2. They show that RL can indeed discover genuinely new solution pathways entirely absent in base models, when given sufficient training time and applied to novel reasoning tasks.
    1. They show that their model generates novel insights and performs exceptionally well on tasks with increasingly difficult and out-of-domain tasks
    2. They identify many tasks where the base model fails to produce ANY correct solution and the RL model achieves 100% correct solutions.
    3. They show that the amount of gain from RL on each task is predictable given the base model’s performance — RL expands reasoning boundary most effectively in domains where the base initially struggles
      1. They introduce a “Creativity Index” which measures the amount of overlap with the pretraining corpus, they show that ProRL leads to higher novelty trajectories.

3. Results & Performance

4. Limitations

5. Personal Takeaways / Action Items

Train Recipe

Training Monitoring. We construct a validation data blend to closely monitor training progress across steps. This validation set includes subsets from our evaluation benchmark, specifically AIME2024, Codeforces, GPQA-diamond, IFEval, and the logic puzzle graph_color from Reasoning Gym. We evaluate model performance using similar sampling parameters as in evaluation settings (other than we use the same context window as in training).

Reference Model and Optimizer Reset. Occasionally, we perform a hard reset of the reference model and optimizer, as described in Section 2.3.1, particularly when validation metrics significantly degrade or when improvements plateau. Interestingly, the hard reset not only restores training stability but also provides an opportunity to adjust training hyperparameters and introduce enhancements such as additional training data and reward shaping. Figure 8 presents KL divergence across training runs. The final training recipe comprises several sequential stages, described in the following.