Local model guide · September 23, 2026 · 16 min read

How to Run MiMo-V2.6 Locally

On September 21 Xiaomi put three models on Hugging Face at once, all under MIT: a 9B dense distill that fits a laptop GPU, MiMo-V2.6-Flash-RL at 309B total and 15B active, and MiMo-V2.6-Pro-RL at 1.02T total and 42B active. The two big ones read text, images, video and audio and carry a 1M-token config. Unlike most day-one releases this one runs on stock software: llama.cpp merged conversion support the next day and the ggml-org account published official GGUFs within hours. This guide covers which of the three your machine should run, the sliding-window memory math that makes the Flash cache small, every quant size with its measured perplexity cost, the commands per engine, and what the built-in speculative decoder is worth while one of its load paths is broken. Everything here was checked on September 23, 2026.

Three models, one release

The release is a family, and the three members are different kinds of object. Pro-RL is the flagship: 70 layers, 384 routed experts, 8 active per token. Flash-RL is what the card calls "the efficiency-balanced checkpoint": the same design at 48 layers and 256 experts. Both are trained the same way, which the card sums up as one mixed reinforcement-learning run across coding, general agents, visual tasks and cybersecurity, with an agentic grader that ranks passing solutions within each group rather than scoring pass or fail. The third model is not a MiMo architecture at all. MiMo-V2.6-Distill-Qwen-9B is Qwen3.5-9B fine-tuned on 77.4B tokens of MiMo-generated data, released, per its card, "as a starting point for open research in agentic reinforcement learning." It inherits Qwen3.5's architecture, its 262,144-token context, and its engine support.

Two details change how you should think about the downloads. First, the big checkpoints ship their routed experts in MXFP4 already. The quantization_config in each config.json says FP8 compute with MXFP4 storage, and the Flash-RL safetensors total 177.74 GB for 309B parameters, about 4.6 bits per parameter. The download is a 4-bit model, so the "full quality" GGUF is the MXFP4 one, and anything smaller is a real step down. Second, the Hugging Face repositories were created on September 21 in UTC; Xiaomi's own announcement is dated September 22. Both dates are correct for their time zones.

The architecture

Both big models are MiMoV2ForCausalLM (model type mimo_v2). The values below are from the raw Flash-RL and Pro-RL configs, cross-checked against the architecture tables on the model cards.

The global layers sit at fixed positions in hybrid_layer_pattern: in Flash-RL at indices 0, 5, 11, 17, 23, 29, 35, 41 and 47, one every six layers after the first; in Pro-RL at 0, 7, 15, 23, 31, 39, 47, 55, 62 and 69. Every other layer attends only to the previous 128 tokens and carries a learned attention sink bias (add_swa_attention_sink_bias), which is how a sliding layer keeps a stable place to put attention mass when nothing in its window matters. The global layers use a long RoPE base of 10,000,000 over a third of each head's dimensions; the sliding layers use a base of 10,000. One mismatch to know about: the config lists num_nextn_predict_layers: 3 while the card describes a 5-layer MTP drafter that predicts 7 tokens per pass. The GGUF sidecars follow the card. If you have read the transformer internals guide, the pieces are all familiar; the proportions are what is new.

A 128-token window

The memory math for long context comes down to one question: which layers keep a cache that grows with the conversation? Here, only the global ones. A sliding layer holds at most 128 tokens of keys and values no matter how long the context gets, so it costs a fixed amount. The arithmetic for Flash-RL in BF16:

• Global layers, per token: 9 layers × 4 KV heads × (192 key + 128 value) × 2 bytes = 23,040 bytes, or 22.5 KB per token.
• Sliding layers, fixed: 39 layers × 8 KV heads × 320 × 2 bytes × 128 tokens = 25.6 MB, total, forever.
• At 262,144 tokens: 6.04 GB. At the full 1,048,576: 24.2 GB. Halve both with a q8_0 cache.

For contrast, if all 48 layers were global with the same 4 KV heads, the cache would be 122,880 bytes per token, 5.3 times larger, and a million tokens would need 129 GB before a single weight loaded. Pro-RL is 51,200 bytes per token (10 global layers with 8 KV heads each) plus 39.3 MB of sliding state, so 13.4 GB at 262K and 53.7 GB at 1M. The 9B distill is a different design again: 8 full-attention layers at 32,768 bytes per token, plus a fixed recurrent state in its 24 linear-attention layers (about 50 MB by my arithmetic from the config, since that state is stored in float32). At its 262K maximum that is 8.6 GB of cache, which on a 16 GB card matters more than the weights do.

The practical reading is the reverse of most local guides: for Flash-RL the cache is small next to the weights until you are well past 128K tokens, so the weights decide what you can run. For the 9B the cache decides how much context you can afford. The router below accounts for both.

Every quant, with its cost

File sizes are sums of the shard files from the Hugging Face tree API on September 23, in decimal gigabytes. The perplexity and KL columns for Flash-RL are AesSedai's measurements against the MXFP4 base, which is the most useful table published so far because it tells you what each gigabyte you save costs.

Flash-RL (309B A15B)

The BPW builds were produced with a quantizer pull request (#15550) that sets per-tensor types to hit a target size; AesSedai keeps attention and shared tensors at Q8_0 or Q6_K and squeezes the expert FFNs. The ggml-org repo also carries the vision and audio projector (mmproj, 1.56 GB at Q8_0) and the MTP sidecars (2.38 GB at MXFP4 or Q8_0, 4.48 GB at BF16).

Pro-RL (1.02T A42B)

Distill-Qwen-9B

Which one your machine runs

Three ladders, one machine. The router lights every build across all three models that fits, and draws the route to the one I would run: the biggest model tier that fits without a heavy-loss quant, then the best quant within that tier. Green fits in fast memory; dashed runs with experts offloaded to system RAM; amber marks a build whose measured perplexity cost is above 10%.

A measured data point for the offload case, from the reporter of llama.cpp issue #29345: Flash-RL MXFP4 on eight 20 GB RTX 3080s with 768 GB of DDR4-2133, most expert layers placed on the GPUs, decoded at about 18.6 tokens per second. That is one person's box, not a benchmark, and it is the only speed figure for a local run I could source on day two. For a ceiling on unified-memory machines, use the rule from the memory bandwidth essay: 15B active parameters at about 4.6 bits read roughly 8.6 GB per token, so a machine's bandwidth divided by 8.6 GB is the most you can ever see, and real engines land well under it.

llama.cpp

Support arrived in PR #29257, "convert: add MiMo-V2.6 support," merged on September 22 and shipped in build b11102. The author's note is the short version of why it landed so fast: both models use MXFP4 experts the way DeepSeek V4 and Kimi K3 do, so the existing K3 repack was hoisted into shared code, and "runtime changes weren't necessary to support the architectures once converted." Any build at or after b11102 runs it. The audio decoder is excluded from the projector, so audio input works and audio output does not.

# Flash-RL on a unified-memory machine (192 GB class), official Q2_K
llama-server -hf ggml-org/MiMo-V2.6-Flash-RL-GGUF:Q2_K \
  --jinja -ngl 999 -fa on -c 65536 \
  --temp 1.0 --top-p 0.95 \
  --host 127.0.0.1 --port 8080

# Flash-RL on GPUs plus system RAM: attention on the cards, experts in RAM
llama-server -hf ggml-org/MiMo-V2.6-Flash-RL-GGUF:MXFP4 \
  --jinja -ngl 999 -fa on -c 131072 \
  -ot "blk\..*\.ffn_.*_exps\..*=CPU" \
  -ctk q8_0 -ctv q8_0 \
  --temp 1.0 --top-p 0.95

# The 9B distill on any 8 GB card
llama-server -hf bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q4_K_M \
  --jinja -ngl 999 -fa on -c 65536

The -ot regex sends every routed-expert tensor to the CPU; to keep some expert layers on the GPUs, put more specific patterns before the catch-all, one per device, which is what the #29345 reporter did with eight cards. Multi-shard files load from the first shard. -hf also fetches the projector when the repo has one; pass --mmproj explicitly if you download by hand, or the model silently ignores image and audio parts. The ggml-org card also shows a one-line llama serve -hf ggml-org/MiMo-V2.6-Flash-RL-GGUF for the newer llama launcher.

For the 9B there was one real bug on day one, now fixed: issue #29319 reported that llama.cpp's template detection routed the distill's chat template to the Qwen3-Coder tool-call parser by substring match, so tool calls never completed. The maintainer's reply: "Already fixed by #29257 b11102, update." If your tool calls run to the token cap, check your build number first. Ollama has no official library entry as of today (ollama.com/library/mimo-v2.6 returns 404); pulling a GGUF straight from Hugging Face with ollama run hf.co/bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q4_K_M should work on builds that already run Qwen3.5 GGUFs, which I have not verified for this model.

vLLM and SGLang

Both engines are documented on the model card, both through the MiMo-V2.5 recipes, because the architecture family did not change. The card warns that "stable vLLM may lag" and points to a prebuilt image, vllm/vllm-openai:mimov25-cu129. The Flash-RL example is tensor parallel 4, which with 177.74 GB of weights means four 80 GB cards:

vllm serve XiaomiMiMo/MiMo-V2.6-Flash-RL \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --max-model-len auto \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --enable-auto-tool-choice \
  --generation-config vllm

SGLang is the engine Xiaomi recommends "for best performance," via its MiMo cookbook and lmsysorg/sglang:latest. The card's Flash-RL command is sized for a node of eight GPUs (--tp 8 --dp 2 --enable-dp-attention) with the MTP head as an EAGLE drafter; the Pro-RL command spans two nodes with --tp 16 --ep 16, DeepEP all-to-all, and --swa-full-tokens-ratio 0.3, a flag that sizes the sliding-window cache pool against the full-attention one. That last flag is the serving-side version of the arithmetic above: most of the cache pool can go to the few global layers. For a single-node Flash-RL setup, the speculative flags are the part worth copying:

sglang serve --trust-remote-code \
  --model-path XiaomiMiMo/MiMo-V2.6-Flash-RL \
  --tp 8 --mem-fraction-static 0.65 \
  --chunked-prefill-size 16384 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-multi-layer-eagle \
  --reasoning-parser mimo --tool-call-parser mimo \
  --host 0.0.0.0 --port 30000

The 9B distill runs on a recent SGLang with Qwen3.5 support using only --model-path and --reasoning-parser mimo, per its card. Once several people or an agent fleet share the box, the serving engines are where you want to be; the vLLM internals guide explains why paged KV and continuous batching change the math.

On a Mac

Two paths. The GGUF path is the llama.cpp build above with Metal, and it works today at any size the machine holds. A 128 GB Mac has about 96 GB usable at the default wired limit, and the BPW2.5 build is 96.80 GB before any cache. BPW2.0 fits, but at a measured 41.7% perplexity cost, so on that machine I would run the 9B, or raise the limit with sudo sysctl iogpu.wired_limit_mb and take BPW2.5 with a short context. A 192 GB Mac Studio holds the official Q2_K or BPW3.5 with room for 128K of context. A 256 GB machine runs Q3_K or MXFP4.

The MLX path is ahead of its own library. mlx-community published Flash-RL and Pro-RL conversions on release day with the experts left at native MXFP4 and everything else at 8-bit: 4.334 bits per weight and 156 GB (167.26 GB decimal on disk) for Flash, which the card says takes "about 170 GB of unified memory" to load. The conversions are text only; the vision and audio encoders and the MTP weights are not included. The model code is in mlx-lm PR #1219, still open, so the card tells you to install the branch:

pip install git+https://github.com/kernelpool/mlx-lm.git@add-mimo-v2

mlx_lm.generate --model mlx-community/MiMo-V2.6-Flash-RL-mxfp4-q8 \
  --max-tokens 2048 --temp 1.0 --top-p 0.95 \
  --prompt "Summarize the failing test output below."

Pro-RL at 553.88 GB does not fit one Mac. Its MLX card describes tensor parallel across two 512 GB machines with mlx-lm's distributed support. For the 9B, mlx-community/MiMo-V2.6-Distill-Qwen-9B-OptiQ-4bit exists and should run on stock mlx-lm wherever Qwen3.5 does.

The MTP drafter

Both big models ship their own speculative decoder: per the card, a 5-layer sliding-window MTP block that "predicts 7 subsequent tokens per forward pass for parallel verification." The main model then checks the drafted tokens in one pass, keeps the prefix it agrees with, and adds one token of its own. How much that buys depends on two numbers you do not control directly: how often the target agrees with each drafted token, and how much a draft costs next to a full decode step. The mechanics are the same ones covered in the DFlash 2 essay; what matters here is which engine path actually works this week.

On SGLang the drafter runs through the EAGLE flags above. On llama.cpp the ggml-org repo ships the drafter as a separate sidecar file, and the documented way to use it (-md mtp-MiMo-V2.6-Flash-RL-MXFP4.gguf --spec-type draft-mtp) crashes at load with blk.0.attn_q.weight not found, per #29345, opened September 23 and still open. The reporter's analysis is that the sidecar holds only the MTP tensors and expects to borrow the main model's base weights, while the -md path loads drafts as complete standalone models. Until that is fixed, speculative decoding with MiMo on llama.cpp is worth zero. Drive the tape to see what it is worth when it works:

Two things the tape makes obvious. Longer draft windows pay only when acceptance is high: at 0.7 acceptance and a draft cost of 0.15, going from 4 to 7 drafts adds about 0.37 tokens per step but nearly doubles the drafting cost, so the speedup falls from 1.73x to 1.53x. And the drafter's cost matters more than people expect for this model, because the main model is cheap per token: 15B active parameters is not a heavy step, so a 5-layer drafter is a larger fraction of it than a drafter is for a dense 70B.

Thinking, sampling, tools

Sampling. The card recommends temperature=1.0 and top_p=0.95 for both big models, and those are the defaults in every command above.

Thinking. The chat template enables thinking by default. To switch it off in llama.cpp, pass --chat-template-kwargs '{"enable_thinking":false}'; over an OpenAI-compatible API, send "chat_template_kwargs": {"enable_thinking": false} in the request body, which is also how the 9B card's own example turns it on explicitly. In MLX, pass enable_thinking=False to apply_chat_template. With the mimo reasoning parser on vLLM and SGLang, the thinking comes back in reasoning_content, separate from the answer.

Tools. Tool calls use the Qwen3-Coder XML shape (<tool_call><function=...>), per the MLX card, which is exactly why llama.cpp's detection mixed it up before b11102. vLLM and SGLang ship a mimo tool parser; llama.cpp needs --jinja, which you should treat as required. For an agent, serve any of these as an OpenAI-compatible endpoint and point the harness at http://127.0.0.1:8080/v1.

Benchmarks, vendor-reported

These are Xiaomi's numbers from the model cards, under Xiaomi's harness choices, with the Claude columns as Xiaomi printed them. I have left out the OpenAI column the card also carries and rows where either Claude column is blank. No independent evaluation of the open weights existed on day two.

Read plainly: against its predecessor the jump is enormous, which is the story of a reinforcement-learning release. Flash sits within a few points of Pro on most agent rows, which makes it the model most people should run. Both trail the closed frontier on the newer, harder terminal benchmark by 14 to 20 points and on the offensive-security rows by more. The 9B card reports its own comparison against the base it was tuned from: SWE Pro 44.6 against Qwen3.5-9B's 32.0, Terminal Bench 2.1 37.1 against 27.0.

Rough edges, day two

• The MTP sidecar crashes llama.cpp at load (#29345, open). Run without -md until it is fixed; the reporter says an embedded-MTP packaging from a community repo loads on the same build.
• Tensor split across devices is not finished. The fused QKV projection has uneven K and V head sizes (192 and 128), and PR #29294 to fix the split is open; Georgi Gerganov reported that Flash-RL on two DGX Sparks "proceeds further with this change but still crashes" at a later assertion. Single-device and -ot expert offload are the paths people are running.
• Old builds misparse the 9B's tool calls. Fixed in b11102 (#29319).
• No imatrix on the official Q2_K. The ggml-org card says so. AesSedai's builds are imatrix-calibrated and come with measured perplexity.
• MLX is on a branch and text only (mlx-lm #1219, open).
• Audio in, not out. The llama.cpp conversion drops the audio decoder from the projector.
• The 1M context is a config value. Plan around what the router says your memory holds at the context you pick, not the maximum.

FAQ

Which of the three should I run?

If you have a single consumer GPU, the 9B distill; it is a different architecture and a much smaller model, so expect a Qwen3.5-9B that is better at agent tasks, not a small Flash. If you have 128 GB or more of fast memory, Flash-RL at the best quant that fits. Pro-RL is for 512 GB Macs, pairs of them, or GPU servers.

Is the MXFP4 GGUF a quantized model?

Only in the sense that the original is. The experts were trained and released in MXFP4, so the ggml-org MXFP4 build keeps them at native precision; AesSedai measures it at +0.07% perplexity against their base. Everything smaller trades quality for memory at the rates in the table.

Why is the long-context cache so small?

Only 9 of 48 layers in Flash-RL keep a growing cache, with 4 KV heads each. The other 39 look at a 128-token window. That is 22.5 KB per token against 120 KB if every layer were global.

Should I wait?

For the 9B and for Flash-RL on one machine, no; stock llama.cpp runs both today. Wait if you need speculative decoding on llama.cpp, tensor split across GPUs, or stock MLX. Each of those is one open pull request or issue away.