You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs

All per-dataset FIXED-K scores and unweighted row means for five additional architectures across every tested integer budget are reported in Table 6.

Table 13 reports Qwen3-30B-A3B-Instruct scores on nine likelihood-scored text QA datasets for every FIXED-K budget from k=8 to k=1.

Some appendix rule configurations do not land exactly at their target tier: overspending rows are daggered and excluded from Table 3, whereas underspending rows remain visible but a deficit on them is not treated as evidence against the rule.

At aggressive target budgets retaining one third to one half of native K, the best eligible dynamic rule improves the eleven-dataset mean on every core model, by 0.90–2.98 points, averaging 2.26 points.

The ordering of dynamic rules changes between conservative and aggressive tiers on the same model, and the spread among rules at a matched tier is wider than the gap between the best rule and FIXED-K.

On Qwen3-30B-A3B at the conservative tier, damping DIEP’s similarity term raises the suite mean from 54.32 to 61.60, while removing it entirely (NAEE) yields 63.92; conservative DIEP deficits versus FIXED-K do not vary monotonically with native K.

Knowledge QA declines gradually, whereas generative reasoning initially holds steady and then drops sharply; mathematical and code reasoning is the least robust family at the aggressive end on eight of nine models.

Table 3 reports, per dataset, the unpruned score and the matched-tier FIXED-K and best eligible dynamic-rule scores for all four core models at conservative and aggressive budgets.

Table 8 reports every evaluated rule at both conservative and aggressive budget tiers for Qwen3-Next-80B-A3B-Instruct, including per-dataset scores, measured budgets, and suite means.

Table 9 reports every evaluated rule at both conservative and aggressive budget tiers for Ling-lite-1.5, including per-dataset scores, measured budgets, and suite means.

Table 7 reports every evaluated rule at both conservative and aggressive budget tiers for Qwen3-30B-A3B-Instruct, including per-dataset scores, measured budgets, and suite means.

On prompt tokens, Qwen3-VL’s selected routing weights are substantially flatter than the text model’s: its top three hold less of the selected-eight mass, and the selected eight hold less of the full 128-way softmax mass.

Uniformly retaining the paper's stated ceil(2K/3) operating point preserves 98.8% of unpruned performance on average across nine models and three task families; six of 27 model-family combinations match or exceed their unpruned scores.

Qwen3-VL routing becomes somewhat more concentrated during decoding than during prompting but remains flatter than the text model; bootstrap intervals over layers support stable differences in top-eight mass and entropy.

Table 10 reports every evaluated rule at both conservative and aggressive budget tiers for GPT-OSS-20B, including per-dataset scores, measured budgets, and suite means.

Thinking and Instruct have similar normalized retention from eight to five experts, but Thinking leads on the four hardest generative datasets by 2.7 points at k=4 and 5.1 points at k=3; the reported task breakdown shows little Knowledge QA difference.

FIXED-K, NAEE, DYNROUTE, DIEP/DIEP-d, and BAN are implemented at the router-selection point; retained gate weights are renormalized, all experts remain resident, and only routed experts may be dropped.

Repeated FIXED-K measurements differ by up to about one point; the paper keeps the older, lower reading in every listed pair so dynamic rules are not credited against a high reference fluctuation.

Uniform truncation remains near-flat at high retained fractions but breaks down sharply at aggressive budgets, and the same absolute k has very different meaning across native router widths.

Within the multimodal suite, ChartQA falls furthest at every reduced budget; at k=2, the three image-text transcription datasets are at or near zero, while multiple-choice datasets retain roughly one third to one half of their own unpruned scores.

Qwen3-235B has no normalized-retention advantage over Qwen3-30B from k=7 through k=5 on the four hardest generative datasets, but leads by 2.5 points at k=4 and 14.4 points at k=3.

For both prompt-token concentration measures, the text and multimodal models’ per-dataset values do not overlap, and the text model remains more concentrated in the same direction across all 48 MoE layers.

The gains from dynamic allocation under aggressive pruning are concentrated in generative tasks, including large recoveries on MATH-500 and LiveCodeBench.

All per-dataset FIXED-K scores and unweighted row means for the four core models across every tested integer budget are reported in Table 5.

Reported k-bar counts routed experts only; shared experts always execute, so equal routed-expert budgets are not equal total expert evaluations across architectures.

At conservative budgets, the best eligible dynamic rule differs from FIXED-K by −0.53 to +0.67 percentage points across the four core models, averaging +0.06 points.

Table 12 reports Qwen3-VL-30B-A3B-Instruct scores on nine multimodal datasets for every FIXED-K budget from k=8 to k=1.

At the tested reduced FIXED-K budgets, prefill and decode throughput improve on both vLLM and HF Transformers, with all per-model ratios reported in Table 2.

Table 11 reports the full per-dataset FIXED-K sweeps for the Qwen3-30B Instruct reference, its Thinking counterpart, and the Qwen3-235B Instruct scale comparison.

The twelve reported checkpoints span routed-expert pools of 32–512 and native selections of 4–10 routed experts per token, with model roles and parameter counts reported in Table 1.

The multimodal and text short-answer suites have normalized means within one point from k=8 through k=4, but the multimodal model falls 11 points behind at k=3 and 42 points behind at k=2.