Five Ways to Prune Experts: What Each Rule Keeps and Loses
Between October 22 and November 8, six public checkpoints deleted a quarter to four fifths of the routed experts from four mixture-of-experts models. Kimi K3.1 Pro got three of them, each by a different rule. A pruned model stores fewer bytes. It does not read fewer bytes per token, because the router still picks the same number of experts. Each build makes one choice, which experts to delete, and its card reports what that choice cost.
autotrust published its E384 build of Qwen3.8-Flash-Next on October 22 and an RL-merged V2 on October 26. On November 2, IST Austria's DASLab published the GSQ-RCO Coder of DeepSeek-V4.1-Flash-1017, and Scantist published GLM-5.3-SLIM-GGUF. Three Kimi K3.1 Pro builds followed: the vLLM project's pruned75 on November 5, then Nota's gated Global-Pruned-30 and pipenetwork's REAP80 on November 6. Grawnch's Kimi-K3.1-Pro-Coder of November 8 deletes nothing and is here as the contrast.
I read the cards, configs, and file trees of all seven on November 9. I recomputed the vLLM ranking from two Kimi K3.1 Pro router matrices. I did not run any of these models. Every speed and score below is card-reported, with its harness named. The GLM-5.3 guide explains how a router picks experts.
One published build at a time. The shelf holds the checkpoint, the plate holds one MoE layer with one tile per routed expert, and the selector station changes with the rule. Bytes come from the safetensors headers, the GGUF type tables, and config.json. The tiles are the published kept list where a build publishes one (REAP80, the RCO Coder, E384) and my own recomputation for pruned75 layers 1 and 2. Everywhere else they are an illustrative pattern with the card's count. Nota's per-layer counts are illustrative too, inside the card's 600 to 712 range. The Tile pattern chip under the drawing says which case you see. Picks per token are illustrative. Scores are card-reported.
An expert is worth what the next tokens want
An MoE layer is a set of small feed-forward networks behind one router. For each token, the router scores every expert and runs the top few. Kimi K3.1 Pro runs 16 of 896 experts in each of 92 layers, and GLM-5.3 runs 8 of 256 in 75 layers. DeepSeek-V4.1-Flash-1017 runs 6 of 384 in 40 layers, and Qwen3.8-Flash-Next runs 10 of 512 in 48.
Pruning deletes whole experts. A deleted expert loses its three weight matrices, its router row, and its entry in the load-balance correction bias. The survivors are renumbered, and the expert count in the config or the GGUF header drops. The vLLM card lists these edits.
GLM-5.3 and DeepSeek set norm_topk_prob, and Kimi sets moe_renormalize, so the picked gate values are rescaled and the layer output keeps its scale. A token that wanted a deleted expert gets a substitute. The REAP paper bounds the error at the gate value of the deleted expert times the sum of the two output norms. So the cost of a deletion depends on how often the expert fires on your tokens, how strongly, and how large its output is. Each rule below estimates that cost from a different guess about your tokens. The machine above runs the steps that all five rules share, and only the scoring step differs.
Rule 1, router magnitude: no data at all
The vLLM project's pruned75 removes 672 of the 896 experts in every MoE layer of Kimi K3.1 Pro. The score is the signed sum of each expert's router row, and nothing else is read. The card calls the checkpoint calibration-free, for vLLM smoke tests on fewer GPUs, and tells you to expect a large quality regression. It gives about 443 GB. The file tree lists 475.5 GB, which is 443 GiB, against 1,561 GB for the source.
A router row scores one expert against the hidden state. Its signed sum is the score for an all-ones input, so it is a prior on how easily the gate opens. The card reports GSM8K at 43.7% for this ranking and 14.4% for the earlier library default, the L1 sum of absolute values. It has no base run. The compressed-tensors PR #1005 made the signed sum the default on November 6.
I read the router matrices of layers 1 and 2 and reproduced the ranking. In layer 1, 455 of the 896 rows have a negative sum, and the 224th row from the top is +3.29. The L1 set shares 71 of its 224 experts with the signed set, and 68 in layer 2. Two random sets of 224 would share about 56. The two scores have a correlation of 0.02, so they rank the experts almost independently. The GSM8K pair says that only the signed sum is a usable prior.
Every dot is one routed expert of one Kimi K3.1 Pro layer. The top strip places it by the signed sum of its router row, the bottom strip by the L1 sum. The values come from the 896 by 7,168 BF16 gate.weight tensor, read from the published shard with an HTTP range request. Move the cut to keep more or fewer experts. Blue dots survive both rankings, dark dots survive one, hollow dots survive neither.
Data: language_model.model.layers.N.block_sparse_moe.gate.weight of moonshotai/Kimi-K3.1-Pro, shards 2 and 3, read on November 9. Dots are spread vertically so dense ranges stay visible. The vertical line on the top strip is zero.
The row sum also ignores Kimi's e_score_correction_bias, 896 values per layer between -0.08 and +0.04 in the two layers I read. The engine adds the bias before the top-k, so a large row sum with a negative bias fires less often than its sum suggests.
Rule 2, architecture search, applied by byte copy
autotrust's E384 keeps 384 of the 512 routed experts in each decoder layer of Qwen3.8-Flash-Next and in the MTP layer. The card calls the selection a neural architecture search and gives no more detail. The V2 repository publishes expert_selection.json, which names a metric, acp, and 14 weighted calibration captures. They include web text, code, math, agent traces, exam questions, and vision. That is a scored selection over a weighted corpus. Only the router matrices change. The kept tensors are byte copies of NVIDIA's NVFP4 checkpoint, and the file goes from 102.14 GB to 85.66 GB.
Scantist's GLM-5.3-SLIM-GGUF applies autotrust's 192-expert selection for GLM-5.3 to Unsloth's UD-IQ2_M GGUF, with no quantizer in the loop. In a GGUF, the expert index is the slowest dimension of the expert tensors, the router, and the bias. So each expert is one contiguous byte range. Scantist copied the 192 kept ranges of each layer in order, set glm-dsa.expert_count to 192, and dropped the MTP layer. The file went from 238.58 GB to 179.86 GB.
The autotrust GGUF card describes the same selection as a frequency count on "representative workloads" that removes the 64 least used experts in each layer. Those workloads are the calibration set. Both autotrust cards report same-harness pairs of the pruned and original models, which Figure 2 collects. The GGUF itself has no pair, so Scantist's 92.7% on HumanEval stands alone.
Rule 3, calibrated KL under the budget the format sets
ISTA's Coder keeps 192 of the 384 experts in each of the 40 layers of DeepSeek-V4.1-Flash-1017. The transformer goes from 138.2 GB to 72.1 GB. The two Engram tables stay at 110.6 GB on the SSD, as in the previous essay. The selection comes from RCO, the lab's budgeted optimizer. It minimizes the KL divergence from the unpruned model on calibration data of code, tool-using agents, long contexts, and vision.
Two details matter. RCO ran on the IQ2_XXS build, so it kept the experts that matter most to that build at its shipped precision. The budget is also exactly 192 in every layer, because a GGUF header holds one expert count for the whole model. A global budget could take more experts from the layers that tolerate it. The format cannot hold that without padding, and padding gives the memory back.
The card reports SWE-bench Verified falling from 81.20 to 77.00 at reasoning effort 50, and LiveCodeBench v6 from 89.71 to 88.57. The base numbers are for the FP8 and FP4 original, so each drop includes the 2-bit quantization as well as the pruning. The card calls degradation outside the calibration domains an accepted cost and does not measure it.
Rule 4, REAP saliency: gate times output norm
REAP comes from a Cerebras paper by Lasby and others, revised in May 2026. The score of expert j is its gate value times its output norm, averaged over the tokens routed to j. The mean is conditional on a pick. So a rare specialist with a large output keeps a high score, and a frequency count would delete it. The paper also argues that merging two experts adds an error that pruning avoids.
pipenetwork's REAP80 keeps 179 of 896 experts in each Kimi K3.1 Pro layer, scored on a corpus that is 60% code. The kept experts are byte copies of Moonshot's MXFP4 weights, everything else is quantized, and the q6 file is 313 GiB. The config publishes the kept list for all 92 layers, and the machine draws it. The card reports held-out perplexity of 5.736 at q6 with no unpruned number, so the cost of the cut is not measured. In one chat test on the q6 build, the model gave 323 for 17 times 23.
pipenetwork's GLM-5.3 builds from August 28 give the cleanest published loss curve, because one harness scores all of them. REAP25, REAP37, and REAP50 keep 192, 160, and 128 of 256 experts at 4 bits. The card reports wikitext-2 perplexity 14.8%, 34.5%, and 75.6% above the unpruned 4-bit build, from 2.864 to 3.287, 3.852, and 5.030. Ranked on two disjoint halves of the calibration set, 93.0% of the kept set is the same. So 7% of the kept set depends on which half of the corpus was scored.
Rule 5, a global budget, and the engine it needs
Nota's Global-Pruned-30 removes 30.4% of the routed experts of Kimi K3.1 Pro under one budget for the whole model. The model page is public and the files are gated, so I use only the page text. Layers keep 600 to 712 experts, and the first two and last two MoE layers keep all 896. In total, 82,432 experts become 57,408, and the checkpoint goes from 1,454 GiB to 1,045 GiB. That moves the model from eight B300 GPUs to eight B200 GPUs, with a 663,070-token FP8 KV cache. Nota's blog says that the method measures a whole block and sets a different removal ratio for each layer.
The price is the format and the engine. GGUF cannot hold ragged counts. The card ships a patch for the vLLM image and lists expert-parallel load balancing and the latency_specialized decode path as unsupported. The compressed-tensors PR #938 added a per-layer num_experts list to the checkpoint schema on October 15, but engine support is a separate step.
The card puts its GPQA-Diamond of 94.44 on Nota's harness beside 94.20 from the Kimi card. That is a card-against-card comparison on two harnesses, and the table labels the base column as the original's model card. Nota's blog claims 2.52 points over REAP on GPQA-Diamond for Kimi K3 at 50% pruning, under Nota's own setup. The autotrust GLM card reports the opposite: a non-uniform per-layer study at equal budget gave no gain. No third party has reproduced either result.
The related design: keep every expert, thin the rare ones
Grawnch's Kimi-K3.1-Pro-Coder deletes no expert. It sorts the 896 experts of each layer into three tiers by routing frequency on code and math. The 134 most used sit at 2.81 bits a weight in GPU memory, and the next 269 at 2.56 bits in pinned RAM. The 493 least used sit at 1.44 bits on NVMe, and the file is 775 GB. The card reports 29 of 30 on both AIME 2025 and a LiveCodeBench window, equal to its own base run. It needs the author's Strata fork and its dr-triton kernels.
This is a three-level expert cache inside the checkpoint. Pruning assumes that an expert will never run. Tiering assumes that it runs rarely, so it can sit at low precision far from the GPU, and every token can still reach all 896.
Pruning saves memory, not time
The arithmetic for Kimi K3.1 Pro comes from the safetensors headers. One routed expert is three matrices of 3,072 by 3,584 weights in the latent expert width, 33,030,144 weights. MXFP4 with one 8-bit scale per 32 values is 4.25 bits a weight, so one expert is 17,547,264 bytes. A token runs 16 experts in each of 92 MoE layers, which is 1,472 expert reads and 25.8 GB. The file holds 82,432 experts, 1,446 GB of its 1,561 GB.
Delete 672 of the 896 experts in every layer, and the stored experts fall to 362 GB. A token still reads 16 a layer, so its expert bytes stay at 25.8 GB. Attention, shared experts, routers, and the output head add about 111.5 GB a token at BF16. So a decode step reads about 137 GB before pruning and about 137 GB after. The file is 70% smaller, and the bytes a step reads did not change. REAP80 reads about 73 GB a token, because its non-expert tensors are quantized. Its card says it in one line: "Pruning buys memory, not speed."
GLM-5.3 gives the same answer. One expert is 3 by 6,144 by 2,048 weights. With gate and up at 2.0625 bits and down at 3.0625 in UD-IQ2_M, that is 11.3 MB. A token reads 8 experts in each of 75 layers, 6.8 GB, plus about 15 GB of everything else. Deleting 64 experts a layer removes 54 GB from the file and nothing from the step. An independent tester ran the 192- and 224-expert builds on two DGX Sparks and got the same speed from both.
What pruning changes is the machine that holds the model. The 192-expert GLM fits one B200 with about 100K tokens of f16 KV cache left. The Nota build moves from eight B300s to eight B200s, and REAP80 fits a 512 GB Mac. The ISTA Coder halves the expert bytes that a 12 GB desktop keeps in RAM and on the SSD. The bandwidth essay explains why the time of a step follows the bytes it reads.
Where the published loss is
Every card that reports a same-harness pair shows the same pattern. Knowledge recall drops most, and code holds. The E384 card reports GPQA-Diamond down 8.6 and 5.1 points at two budgets, MMLU down 5.1, and C-Eval down 4.5, while HumanEval gained 0.6. On the 192-expert GLM, autotrust reports C-Eval down 7.3 and GPQA-Diamond down 3.6, while HumanEval did not move. Figure 2 collects every pair.
The calibration sets explain the pattern. By their cards, all three selections target code and agent work. A question about Chinese history routes to experts that a code corpus rarely reaches, so those experts score low under every rule that reads tokens. The REAP paper reports the reverse case: models calibrated on C4 web text collapsed on code generation, some to 0% accuracy.
Every pair is a single run, and GPQA-Diamond has 198 questions, so one question is half a point. The E384 card treats 2 points as noise. The pruned E384 also thinks 1.8 times longer, with a median of 16,817 reasoning tokens against 9,458. A tight budget turns its unfinished answers into wrong ones. A fair pair gives the pruned model the budget that it uses.
Each row is one benchmark that a card reports for both the base and the pruned model, grouped by what the benchmark asks for. The bar is the change in points, red for a loss and green for a gain. Same harness means the publisher ran both models on one setup. Two cards means the base number comes from the base model's own card, on another harness. Perplexity rows show the percentage change, and higher is worse. Every number is card-reported, and the ISTA base is the FP8 and FP4 original, not the 2-bit build.
Sources: the autotrust E384 and GLM-5.3-GGUF-DGX-Spark cards, the ISTA Coder card, and the Nota Global-Pruned-30 and -37 cards. Also the pipenetwork GLM-5.3-REAP cards, the Grawnch card, and the vLLM pruned75 card. Single runs throughout. autotrust says a 2-point difference is noise on sets this small. The bar scale is 10 points for scores and 80% for perplexity.
Run one
GLM-5.3-SLIM has the most complete recipe of the six, measured on one B200. You need llama.cpp built with CUDA, about 180 GB of disk, and about 170 GiB of free GPU memory. Scantist tested llama.cpp commit 8e5a58d. Pass the first shard, and llama.cpp opens the other five.
hf download Scantist/GLM-5.3-SLIM-GGUF \
--include "GLM-5.3-UD-IQ2_M-E192-*.gguf" "eval/*" --local-dir GLM-5.3-E192
llama-server -m GLM-5.3-E192/GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf \
-ngl 99 -kvu -c 102400 --kv-unified-per-slot 68608 -np 10 -bs \
--host 127.0.0.1 --port 8080 --metrics
The -kvu flag makes one KV pool that all slots share, and -c 102400 sizes it at about 100K tokens of 93 KiB each. The --kv-unified-per-slot value caps one request, and -np 10 opens ten slots. The -bs flag samples on the GPU, which the card measured at 23% more throughput, with inaccurate logprobs. The card reports about 35 tokens a second for one sequence. If the pool runs out, every running request fails with Context size has been exceeded. Keep the slot count times the longest request under the pool, or pass -ctk q4_0 -ctv q4_0 to fit about 340K tokens.
The other builds use the commands on their cards. The E384 V2 checkpoint serves on one B200 with the vllm_patch/sitecustomize.py that its card bundles. REAP80 runs in mlx-lm on a 512 GB Mac after sudo sysctl iogpu.wired_limit_mb=480000. The pruned75 card serves on four GPUs with vllm serve vllm-project/Kimi-K3.1-Pro-pruned75 --tensor-parallel-size 4 --trust-remote-code.
Measure what it cost you
The cards measure what each publisher chose, so measure your own traffic. Take 200 to 300 of your prompts with a pass or fail check. Run them on the pruned build and on a reference with the same sampling. Give the pruned model a generation budget that it can use in full. Write one JSON line per prompt with an id and a pass field for each model, and compare the pairs.
import json, sys
from math import comb
def load(path):
return {r["id"]: bool(r["pass"]) for r in map(json.loads, open(path))}
ref, pruned = load(sys.argv[1]), load(sys.argv[2])
ids = sorted(set(ref) & set(pruned))
b = sum(1 for i in ids if ref[i] and not pruned[i])
c = sum(1 for i in ids if pruned[i] and not ref[i])
n = b + c
p = min(1.0, 2 * sum(comb(n, i) for i in range(min(b, c) + 1)) / 2 ** n) if n else 1.0
print(f"{len(ids)} prompts, reference {sum(ref[i] for i in ids)} pass, pruned {sum(pruned[i] for i in ids)} pass")
print(f"reference-only passes {b}, pruned-only passes {c}, exact McNemar p = {p:.3f}")
The discordant pairs carry the signal. Suppose the reference passed 20 prompts that the pruned build failed, and the pruned build passed 8 that the reference failed. The exact test says whether 20 against 8 is chance. Scantist ran HumanEval twice on the same pruned build. The runs differed on 9 problems, 7 one way and 2 the other, and the card reports p = 0.18. The evals guide works the interval arithmetic. Put your knowledge-heavy prompts in the set first, because the cards report the largest losses there.
When it goes wrong
The first failure is a router that still knows a deleted expert. Every expert-axis tensor, the router rows, the bias, and the header count must agree. The llama.cpp loader refuses a mismatch with tensor has wrong shape. A safetensors engine with a stale num_experts can index the wrong expert and report no error. After a prune, print the expert count and run one prompt whose answer you know.
The second is a ragged count in a GGUF. GGUF stores one count, and padding the short layers back gives the memory back. For a GGUF, keep one count in every layer, as RCO did. For a global budget, use safetensors with an engine that reads a per-layer list. Expect load balancing and the fast decode path to be off.
The third is a calibration set that is too narrow. Rank your experts on two halves of a held-out slice of your traffic, and read the overlap before you trust a set. A low overlap means that the corpus is too small or too narrow for the cut.
Choose the rule by the distribution you can name
- No data and no quality bar, only a smaller test rig: router magnitude with the signed sum. It costs nothing to compute, and the card says what to expect.
- A workload you can write down: REAP or RCO on a calibration set drawn from that workload, with half held out to check the choice.
- A GGUF to publish: one count per layer, which RCO enforces and which a byte copy can apply to an existing quant.
- Your own engine and the last tenth of memory: a global budget, with an engine patch and no expert-parallel load balancing.
- Knowledge you cannot lose: keep every expert. Store the rare ones at a lower bit width, as the Grawnch build does, or read them from RAM and SSD.
In every case, a token reads as many experts a layer as before. Size the machine by the stored bytes, and expect the speed of the read bytes. Then run your own prompts on the pruned build before you rely on it.
Keep reading