Paper summary · Oct 2026
Razor & Higher Order Pruning of Experts in MoE language Models
Provenance
- Link — https://arxiv.org/pdf/2609.18916, https://arxiv.org/pdf/2609.30465
- Authors / affiliation —
- Venue, date —
- How it reached me —
- Why I clipped it —
Claim
Both of these papers introduce more principled ways to prune experts in MoE models to reduce memory/compute constraints while maintaining performance on a specific test-set/domain
Techniques and evidence
Both papers note that by inspecting , the routed output of each layer we can identify the consensus. Which experts are more/less important on the dataset on which we’d like to maintain performance. They both note that in order to go from consensus residual to exact deletion effect, one needs to take into account the higher order effects that result from the deletion beyond just the magnitude of the removed component. They differ in the way they go about this. Razor - HOPE -
Generalizable vs specific
Limitations and flaws
Both papers note the same issue but use severe relaxations to make the problem tractable, relaxtions that lose a significant amount of information. A principled approach would approximate the full counterfactual - how would removing this subset of experts effect the ENTIRE delta. i.e we need to consider all of:
- The entire subset we remove rather than each expert in isolation
- The experts that replace them (we now that models are highly redundant We clearly see how each of these papers handles only one of the two components and a relaxed version of it at that.
Anki
- The two methods, how they decide which experts are removed along with a consolidation card that compares and contrasts them.