Harel Lidar

Paper summary · Oct 2026

Improving Data Efficiency for LLM Reinforcement Fine-tuning through Difficulty-targeted Online Data Selection and Rollout Replay

Summary of Improving Data Efficiency for LLM Reinforcement Fine-tuning through Difficulty-targeted Online Data Selection and Rollout Replay by Yifan Sun, Jingyan Shen, Yibin Wang, Huan Zhang · paper ↗

TL;DR

1. Problem Statement

2. Methodology / Architecture

JGRPO-RR(θ)=Eq∼D, {oi}i=1G∼πθbehavior(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(r~i,t(θ)Ai, clip(r~i,t(θ),1−ϵ,1+ϵ)Ai)−βDKL(πθ ∥ πref))],\begin{aligned} \mathcal{J}_{\text{GRPO-RR}}(\theta) = {} & \mathbb{E}_{q \sim \mathcal{D},\ \color{red}{\{o_i\}_{i=1}^{G} \sim \pi_{\theta_{\text{behavior}}}(\cdot \mid q)}} \\ & \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Big( \min\big( \tilde{r}_{i,t}(\theta) A_i,\ \text{clip}\big(\tilde{r}_{i,t}(\theta), 1-\epsilon, 1+\epsilon\big) A_i \big) - \beta D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) \Big) \right], \end{aligned}

The full algorithm: Algorithm 1 GRPO with DOTS and RR

Require: Initial policy model πθ\pi_\theta, reward model rφr_\varphi, training dataset D\mathcal{D}, target difficulty α\alpha, batch size BB, total steps TT, reference set size KK, sampling temperature τ\tau, adaptive difficulty prediction framework DiffPred (§4.1), fresh rollout fraction δ∈(0,1]\delta \in (0, 1], buffer capacity CC


3. Results & Performance

4. Limitations

5. Personal Takeaways / Action Items