Harel Lidar

Paper summary · Oct 2026

Razor & Higher Order Pruning of Experts in MoE language Models

Summary of Razor & Higher Order Pruning of Experts in MoE language Models · paper ↗

Provenance

Claim

Both of these papers introduce more principled ways to prune experts in MoE models to reduce memory/compute constraints while maintaining performance on a specific test-set/domain

Techniques and evidence

Both papers note that by inspecting c(x)=Σi∈S(x)wi(x)fi(x)c(x)= \Sigma_{i \in S(x) }w_i(x)f_i(x) , the routed output of each layer we can identify the consensus. Which experts are more/less important on the dataset on which we’d like to maintain performance. They both note that in order to go from consensus residual to exact deletion effect, one needs to take into account the higher order effects that result from the deletion beyond just the magnitude of the removed component. They differ in the way they go about this. Razor - HOPE -

Generalizable vs specific

Limitations and flaws

Both papers note the same issue but use severe relaxations to make the problem tractable, relaxtions that lose a significant amount of information. A principled approach would approximate the full counterfactual - how would removing this subset of experts effect the ENTIRE delta. i.e we need to consider all of:

  1. The entire subset we remove rather than each expert in isolation
  2. The experts that replace them (we now that models are highly redundant We clearly see how each of these papers handles only one of the two components and a relaxed version of it at that.

Anki