Paper summary · Oct 2026
Finetuning with Sampling - SFT Learns Better Than You Think
Provenance
- Link — https://arxiv.org/pdf/2610.02140
- Authors / affiliation —
- Venue, date —
- How it reached me —
- Why I clipped it —
Claim
By utilizing a MCMC sampling algorithm one can progressively transform an off-policy expert trace to be more on-policy given a reference model for fine tuning -> this minimizes KLD directly rather than through a regularizer -> improved capabilities while avoiding the catastrophic forgetting that hinders SFT usually.
Techniques and evidence
Here is the pseudocode from the original paper: Algorithm 1: Projection Sampling with Autoregressive Models
- Input: base ; proposal ; expert trajectory ; length
- Hyperparams: block size ; MCMC steps
- Output:
1Notation: Define the unnormalized intermediate target2for to do3Given prefix , we wish to sample from . Construct initialization by extending autoregressively with : Set the current state .4for to do5Sample an index uniformly.6Construct proposal sequence with prefix and resampled completion:7Compute acceptance ratio (5): Draw ;8if then accept and set
9end10Set to fix the new prefix sequence for the next stage.
11end12return
Let’s understand the components one by one:
- we assume that trajectories can be grouped into “equivalence classes” based on task-specific notion of “correctness”. We note this group with .
- We note with the set of all distributions that are only output trajectories which are equivalent to the expert traces
- They prove that i.e the restriction of the base model onto equivalent trajectories.
- They rewrite the Metropolis-Hastings algorithm
- This gives us a convenient recipe for boosting an off-policy dataset: initialize at the expert trajectory, mutate the candidates with some information-preserving proposal distribution and use (6) to accept or reject candidates. In fact, not only does this process eventually converge to , but it also guarantees a progressive decrease in the policy gap between the data generation policy and the base model.
- A direct implementation of Metropolis-Hastings for LLMs is infeasible as it requires regenerating full-length token sequences with repeated LLM inference calls. This infeasibility is further compounded by slow mixing, where convergence can require exponentially many MCMC steps due to high dimensional sample spaces or poor choice of proposals and initializations. To help avoid these issues, they adapt the implementation of Metropolis-Hastings for power sampling in Karan and Du [2025], which we refer the reader to for further details. In short, they define restriction distributions of the target projection pC over sequences of increasing length in units of block size B. They use Metropolis-Hastings to sample within each distribution, providing a strong initialization to begin sampling from the next one. T
Anki
- The pseudocode as well as the computational optimizations needed on top in order to functionalize this for LLMs