Harel Lidar

Paper summary · Oct 2026

Finetuning with Sampling - SFT Learns Better Than You Think

Summary of Finetuning with Sampling - SFT Learns Better Than You Think · paper ↗

Provenance

Claim

By utilizing a MCMC sampling algorithm one can progressively transform an off-policy expert trace to be more on-policy given a reference model for fine tuning -> this minimizes KLD directly rather than through a regularizer -> improved capabilities while avoiding the catastrophic forgetting that hinders SFT usually.

Techniques and evidence

Here is the pseudocode from the original paper: Algorithm 1: Projection Sampling with Autoregressive Models


Let’s understand the components one by one:

  1. we assume that trajectories can be grouped into “equivalence classes” based on task-specific notion of “correctness”. We note this group with C\mathbb{C}.
  2. We note with PCP_\mathbb{C} the set of all distributions that are only output trajectories which are equivalent to the expert traces
  3. They prove that pc:=argmin⁡π∈PCKL(π∣p)=.p(x)1(x∈C)Σx′∈X′p(x′)⋅1(x′∈C)p_c := arg \min _{\pi \in P_C} KL (\pi|p)=. \frac{p(x) \boldsymbol{1}(x \in \mathbb{C})}{\Sigma_{x' \in \mathbb{X'}}p(x')\cdot \boldsymbol{1}(x' \in \mathbb{C})} i.e the restriction of the base model onto equivalent trajectories.
  4. They rewrite the Metropolis-Hastings algorithm
A(x,xi)=min⁡{1, p(x)⋅κ(xi∣x)⋅1(x∈C)p(xi)⋅κ(x∣xi)⋅1(xi∈C)}=min⁡{1, p(x)⋅κC(xi∣x)p(xi)⋅κC(x∣xi)}.(6)A(\mathbf{x}, \mathbf{x}^i) = \min\left\{1,\ \frac{p(\mathbf{x}) \cdot \kappa(\mathbf{x}^i \mid \mathbf{x}) \cdot \mathbf{1}(\mathbf{x} \in \mathcal{C})}{p(\mathbf{x}^i) \cdot \kappa(\mathbf{x} \mid \mathbf{x}^i) \cdot \mathbf{1}(\mathbf{x}^i \in \mathcal{C})}\right\} = \min\left\{1,\ \frac{p(\mathbf{x}) \cdot \kappa_{\mathcal{C}}(\mathbf{x}^i \mid \mathbf{x})}{p(\mathbf{x}^i) \cdot \kappa_{\mathcal{C}}(\mathbf{x} \mid \mathbf{x}^i)}\right\}. \tag{6}
  1. This gives us a convenient recipe for boosting an off-policy dataset: initialize at the expert trajectory, mutate the candidates with some information-preserving proposal distribution κC\kappa_\mathbb{C} and use (6) to accept or reject candidates. In fact, not only does this process eventually converge to πC\pi_\mathbb{C} , but it also guarantees a progressive decrease in the policy gap between the data generation policy and the base model.
  2. A direct implementation of Metropolis-Hastings for LLMs is infeasible as it requires regenerating full-length token sequences with repeated LLM inference calls. This infeasibility is further compounded by slow mixing, where convergence can require exponentially many MCMC steps due to high dimensional sample spaces or poor choice of proposals and initializations. To help avoid these issues, they adapt the implementation of Metropolis-Hastings for power sampling in Karan and Du [2025], which we refer the reader to for further details. In short, they define restriction distributions of the target projection pC over sequences of increasing length in units of block size B. They use Metropolis-Hastings to sample within each distribution, providing a strong initialization to begin sampling from the next one. T

Anki