Local model guide · August 27, 2026 · 14 min read

How to Run Qwen3.8-Flash-Next Locally

Qwen3.8-Flash-Next is the first open-weight model whose biggest single tensor is not a weight matrix. It is a lookup table: 20 million rows of n-gram embeddings, 51 billion parameters, roughly a third of every file you download. A token touches two rows of it. That one fact decides how you run the model at home, because a tensor you read two rows of per token can sit on an NVMe drive behind mmap while the 6B active parameters stay in fast memory. This guide has the exact Unsloth GGUF sizes, an offload planner that places the table, the experts, and the KV cache on the machine you own, the ledger for a hybrid stack where only 12 of 48 layers keep a KV cache, speed ceilings by memory bandwidth, the llama.cpp and vLLM commands, thinking and effort settings, and the rough edges reported in the pull request that landed the architecture.

What it is

Qwen3.8-Flash-Next is Alibaba's open-weight preview of the architecture the Qwen team says will underpin Qwen4. The card is direct about the intent: the question is "no longer just how much we can scale, but how efficiently," and the answer is four architectural bets shipped as one 180B-parameter safetensors checkpoint. It is a causal language model with a vision encoder, 125B parameters in the mixture-of-experts stack with 6B activated per token, plus a 51B n-gram embedding table and a 4B multi-token-prediction head. The native context is 262,144 tokens, extensible to 1M with YaRN. Thinking is on by default. The blog post and the technical report ("On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability", August 2026) carry the design rationale.

Two names to keep apart. Qwen3.8-Flash-Next is the open checkpoint on Hugging Face. Qwen3.8-Flash is the hosted product on Qwen Cloud, which the card describes as "the official version based on Qwen3.8-Flash-Next with more production features, e.g., 1M context length by default, official built-in tools." Everything below is about the open one.

The local-inference community read it the way it reads any new 6B-active model: as a challenger to DeepSeek V4 Flash. An issue opened the day the llama.cpp work started put it plainly, "125B params plus 51B n-gram embeddings, only 6B active per token. Looks like an ideal model for local deployment and a serious contender to deepseek-v4-flash" (antirez/ds4 #867). The comparison is fair on active parameters and unfair on file size, and the difference is the table.

The architecture, briefly

Every row of this config feeds the memory math further down, so it is worth the minute. The numbers are from the model card; the "why" column is mine.

Three of the four bets are about memory traffic rather than FLOPs. Gated DeltaNet on 36 of 48 layers means most of the stack carries a constant-size state instead of a cache that grows with every token. Qwen Sparse Attention on the other 12 selects micro-blocks rather than individual tokens, capping what each attention layer reads at 2,048 tokens no matter how long the prompt is. And the n-gram table is, in the card's own words, "a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts." That last sentence is the whole local-inference story: 51B parameters that never need to be in fast memory.

Arithmetic you can check from the table. The routed experts are 48 layers × 512 experts × 3 matrices × 2,560 × 640 = 120.8B parameters. Ten routed experts per layer per token is 2.36B of that. Add the shared expert, the 36 Gated DeltaNet blocks (about 1.5B), the 12 sparse-attention blocks (about 0.4B), and the 636M-parameter output head, and the per-token active set lands near the 6B the card states. The n-gram table is 20,000,000 rows × 2,560 = 51.2B. If you have read the transformer guide, the only new object here is that table.

The n-gram table, in one paragraph

A normal embedding table has one row per vocabulary entry: the model looks up the current token and gets a vector. Qwen's n-gram table has rows for pairs and triples of tokens, hashed into 20 million slots, and it is consulted at layer 2, after the first two blocks have already run. For each position the model hashes the last two tokens and the last three, fetches those two rows, and adds them into the residual stream. That is the whole mechanism. It gives the network 51B parameters of memorized local context, the kind of thing a transformer otherwise spends attention and expert capacity re-deriving, at a cost of two row reads per token and no extra matrix multiplies. The llama.cpp port computes the row indices on the host from the token history and gathers the rows with a plain get_rows call, which is why the tensor can live wherever mmap can reach. It is, in effect, a giant cache of bigram and trigram knowledge, and caches are the one kind of memory that is happy to be slow.

Where the bytes go

Think of a download as three tensors with three different access patterns. The core (attention, DeltaNet, norms, embeddings, the shared expert, about 4.5B parameters) is read every token and belongs in the fastest memory you have. The routed experts (about 120B) are read ten-at-a-time per layer, a different ten for every token, so they want to be resident somewhere with real bandwidth, and if they are not, decode speed collapses to whatever paging them in costs. The n-gram table (51B) is read two rows per token, about 5 KB, which is why the llama.cpp port keeps it as a single tensor that "can be offloaded to RAM or Disk via mmap," in the words of the pull request author (llama.cpp #27742). The same PR measured the table at 97.7 GiB of a 337.6 GiB BF16 conversion and noted that the imatrix Q4_K_M recipe leaves the table at q8_0. vLLM does the equivalent with VLLM_PLE_CPU_OFFLOAD=1, which its recipe says needs at least 51 GB of host RAM.

Read the planner the way you would read a bill. On a 24 GB or 32 GB GPU the experts do not fit, and llama.cpp will keep them in system RAM, which puts decode on the CPU's memory bus. On a 128 GB unified-memory machine (DGX Spark, Strix Halo, a 128 GB Mac) the 3-bit tiers are fully resident and the 4-bit tier is a squeeze that depends on the table going to disk. Above 192 GB you stop thinking about it. The table row is the one to watch: at q8_0 it is 54 GB that you can keep in RAM for a few hundred microseconds saved per token, or leave on NVMe for a few hundred microseconds spent. At 5 KB per token, either choice is invisible next to the experts.

Pick a quant

Unsloth publishes twelve tiers plus a separate MTP folder. Sizes below are the sum of every shard in each folder, read from the Hugging Face API on September 3, 2026, in decimal gigabytes; divide by 1.074 for GiB. The vLLM recipe lists the official FP8 checkpoint at 172.78 GiB and BF16 at 335.28 GiB, which matches the GGUF BF16 within conversion overhead.

Two things the table hides. Unsloth has not published which tensor type each tier gives the n-gram table, and the file sizes say it varies: the llama.cpp PR's imatrix Q4_K_M recipe leaves the table at q8_0, about 54 GB, which fits the arithmetic of UD-Q4_K_XL and above and cannot fit the 1-bit through 3-bit tiers, so those must quantize the table harder; the planner above defaults to the file's average and flags the mismatch. And there is no per-tier quality table from Unsloth or Qwen yet, so the honest guidance is the usual one for dynamic quants of large mixtures: 4-bit is where you stop noticing, 3-bit is fine for agent loops that verify their own work, and 1-bit is for proving it boots.

Run with llama.cpp

The architecture landed in master on August 27, 2026 when #27742 merged, opened the day before by Daniel Hanchen of Unsloth, under the Hugging Face model type qwen4_exp. Packaged builds older than that fail to load the file, so build from master or wait for your package manager to catch up. The commands on the Unsloth card are the short ones:

./llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
./llama-cli    -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL

They work, and they hide everything that matters. A fuller invocation, with Qwen's recommended thinking-mode sampling, the template applied, and a context you chose on purpose:

./llama-server \
  -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  -c 32768 -ngl 99 \
  --host 127.0.0.1 --port 8080

Three flags need thought on this model.

-ngl and the experts. On a discrete GPU that cannot hold the routed experts, keep the core on the GPU and the experts in RAM. llama.cpp has had expert-offload switches for a while (--cpu-moe for all expert tensors, --n-cpu-moe N for the first N layers' experts), and they are the right tool here: the core is a few gigabytes, the experts are the rest. Start with --cpu-moe on a 24 GB card, then pull layers back onto the GPU until VRAM is full.

mmap and the table. Leave memory mapping on. The n-gram table is one tensor, and with mmap the pages you never touch never load; each token's two rows fault in on demand from the page cache or the drive. A tester on a DGX Spark reported that --no-mmap pushed the load to about 112 GB and crashed the CUDA driver, while the mmap path loaded and ran. If your RAM is large enough that the whole table stays in page cache, you get RAM speed on the table for free.

-c and the CUDA limit. Do not set the context to the native 262,144 on CUDA yet. A reviewer on the PR traced an abort at that size to rms_norm in core ggml-cuda, where the kernel's grid uses n_ctx / 4 channels (the gated residual's four branches) and 65,536 exceeds the launch limit. It is not an out-of-memory error, VRAM was at 68% in the report, and it is not in the PR; treat 131,072 as the ceiling on CUDA until that lands.

To disable thinking for a plain instruct endpoint, pass the template variable through:

./llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
  --jinja --chat-template-kwargs '{"enable_thinking": false}' \
  --temp 0.7 --top-p 0.8 --top-k 20 --presence-penalty 1.5 \
  -c 32768 -ngl 99 --port 8080

Those are Qwen's instruct-mode numbers: temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5. The presence penalty is the one people forget, and the card says it is there to reduce endless repetition.

Ollama, LM Studio, and Macs

Ollama and LM Studio both run on llama.cpp underneath, so the rule is the same: the bundled engine has to be newer than the August 27 merge before either will load the file. Until a release ships with it, Ollama's Modelfile route works with a locally built engine and any of the Unsloth GGUFs, and LM Studio picks the model up as soon as its runtime updates. The Unsloth card also lists a Jan and an LM Studio path for the same repository. Watch the memory dial in both apps: they default to loading everything they can into VRAM, and on this model you want the experts resident and the table left to the page cache, which is the opposite of the usual "put it all on the GPU" instinct on a discrete card.

Macs are the friendliest home for this model because unified memory turns the placement question into one number. A 128 GB Mac counts about 96 GB as usable for the model and runs the 3-bit tiers with a long context; a 192 GB Mac Studio takes UD-Q4_K_XL with the table in RAM; a 512 GB Mac Studio runs Q8_0 with room for the full native context. Metal builds of llama.cpp inherit the merged support. Native MLX conversions were not in the mlx-community index when this was written, so the Metal GGUF path is the one to use today; check the search before assuming that has not changed. The DeltaNet layers are the part that will decide Mac speed, because they are a recurrence rather than a matmul, and how well a Metal kernel handles them is a per-engine question rather than a hardware one.

Thinking, effort, and the template

The model thinks by default, emitting <think>...</think> before the answer, and it exposes three controls through the chat template and the API: enable_thinking, preserve_thinking, and reasoning_effort. The effort levels are xhigh, medium, and low, with xhigh as the default. Qwen's advice for agent loops is counterintuitive and worth quoting: "lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption."

Preserved thinking is on. By default the model keeps the thinking blocks from every earlier assistant turn in context, which the card says helps decision consistency across an agent run and improves KV cache reuse, because the prefix of the conversation stops changing between turns. It also means a long agentic session grows faster than you expect. Turn it off per request with {"chat_template_kwargs": {"preserve_thinking": false}} if you are context-bound and the task is stateless.

Output budgets. Qwen recommends allowing 262,144 tokens of reasoning and 131,072 tokens of final response inside a 1M context for agentic tasks. Locally, the sane reading is: do not cap the reasoning at a few thousand tokens and then blame the model for stopping mid-plan.

Through an OpenAI-compatible client, the shape is standard, and llama-server, vLLM, and SGLang all accept it:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="EMPTY")

r = client.chat.completions.create(
    model="qwen3.8-flash-next",
    messages=[{"role": "user", "content": "Refactor the retry loop in worker.py"}],
    reasoning_effort="medium",
    extra_body={"chat_template_kwargs": {"enable_thinking": True, "preserve_thinking": True}},
    stream=True,
)

Tool calls come out as XML, not JSON; vLLM's recipe uses --tool-call-parser qwen3_xml and llama.cpp's --jinja path handles the same template. If you assemble prompts by hand you will break tool calling, so do not.

Wire an agent

This is an agent model by design; every headline row on Qwen's table is an agentic benchmark, and the card's own advice on effort levels is written for multi-turn tool loops. The wiring is the standard one: serve an OpenAI-compatible endpoint and point your harness at it.

./llama-server -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL \
  --jinja -c 65536 -ngl 99 --cpu-moe \
  --temp 1.0 --top-p 0.95 --top-k 20 --port 8080

# in the agent's config:
#   base_url: http://127.0.0.1:8080/v1
#   model:    anything (llama-server ignores the name)

Three things change once an agent is on the other end. Context grows faster than the transcript. Preserved thinking keeps every earlier turn's reasoning in the prompt, so a 40-turn coding loop with xhigh effort can carry hundreds of thousands of tokens of thought; that is what the 262K native window is for, and it is also why the KV ledger in Fig. 2 matters more than it does for chat. Effort is a per-task decision. Qwen says low effort can cost more total tokens through retries; the practical split is medium for tool dispatch and file edits, xhigh for planning turns, and you set it per request rather than per server. Tool schemas go through the template. The XML tool format is produced by the chat template and parsed by the engine, so pass tools as structured definitions through the API and never paste them into a system prompt as text.

Which harness is a matter of taste, and the pattern is the same across all of them: Claude Code, Codex, opencode, Pi, and Kimi CLI all accept an OpenAI-compatible base URL, and the harness comparison covers how each one spends the tokens you just budgeted. Local-model harnesses were quick to ask for this model specifically; the request in antirez/ds4 that called it "an ideal model for local deployment" was filed the day the llama.cpp work began.

The KV ledger at 1M context

Here is where the hybrid stack pays off. A conventional 48-layer model with 2 KV heads of dimension 256 would store 48 × 2 × 256 × 2 × 2 bytes = 98 KB per token, about 25.8 GB at 262,144 tokens. Qwen3.8-Flash-Next stores that on 12 layers only, 24,576 bytes per token, because the other 36 layers are Gated DeltaNet blocks: each keeps a recurrent state of 48 heads × 128 × 128 values plus a short convolution buffer, about 1.6 MB per layer, that does not change size whether the prompt is 4K or 1M. The sparse-attention indexer adds its own small per-token key. Slide the context and watch which cells grow.

The number that matters for planning: about 6.4 GB of KV plus 0.8 GB of indexer keys at the full native context, and about 29 GB in total at 1M, in BF16, before KV quantization. That is small enough that a 128 GB unified box can hold a 3-bit tier and a 262K conversation at once. The indexer is the part people miss when they estimate compute: attention only scores 2,048 selected tokens per layer, but the indexer still has to scan every cached key to choose them, which is why prefill on very long prompts is not free even when decode is.

Speed: bytes touched per token

Decode speed on a memory-bound machine is bandwidth divided by bytes read per token, the arithmetic in the tokens-per-second essay. For this model the bytes are: the 6B active parameters at the tier's bytes per parameter, two rows of the n-gram table (5,120 values), the DeltaNet states read and written, the 2,048 attended KV entries on each of 12 layers, and the indexer's scan across all cached keys. Everything else on disk is untouched. The estimator gives the ceiling, then shows what week-one builds actually reported, so you can see how much of the gap is software.

Two honest readings of the estimator. First, the ceiling is high: at 4-bit a 128 GB unified box has room for tens of tokens per second, and a 5090 with experts in DDR5 is capped by the DDR5 bus, not the GPU, which is the usual fate of big mixtures on small cards (the DeepSeek V4 Flash guide walks the same trade-off). Second, the reported numbers in the merge week were an order of magnitude under the ceiling. That is what a brand-new architecture in a general-purpose engine looks like: the DeltaNet kernels, the indexer, the gathered expert matmuls, and the n-gram row fetches all run through generic paths until someone writes the fused ones. Expect the reported figures to move quickly and the ceilings to stay where they are.

MTP. The 4B multi-token-prediction head is the model's native drafter: it proposes extra tokens that the main model verifies in one pass, the same lossless trick as the DFlash drafter in the Muse Glimmer guide. vLLM enables it with --speculative-config '{"method":"mtp","num_speculative_tokens":3}'. In llama.cpp the author said MTP was "still WIP" when the PR opened and everything else worked, so check the repository before counting on it. The GGUF MTP folder is 24.62 GB and only helps if it is loaded.

vLLM and SGLang

This is a server model first, and Qwen ships recipes for vLLM, SGLang, and TokenSpeed. The vLLM recipe wants vLLM 0.29.0 or newer, or the vllm/vllm-openai:qwen38-flash-next image, and the FP8 checkpoint at 172.78 GiB. Its four-GPU configuration:

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

On eight H200s the recipe switches to expert parallelism (--enable-expert-parallel --moe-backend triton, tensor parallel 8) and notes that plain tensor parallel 8 is incompatible with the FP8 checkpoint on that card; pipeline parallelism is not supported at all. The offload switch for the table is an environment variable, VLLM_PLE_CPU_OFFLOAD=1, which the recipe says requires at least 51 GB of host RAM, exactly the table at one byte per value. Leave --max-model-len unset for the native 262,144, or set the YaRN override for 1M:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.8-Flash-Next-FP8 ... \
  --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
  --max-model-len 1000000

The card's warning applies to every engine: YaRN as implemented is static, the scaling factor applies at every length, so it can cost accuracy on short prompts. Set factor 2.0 if 524,288 is the longest you need, and leave the override off entirely for ordinary work. The mechanics of paged KV and the scheduler that makes 256 concurrent sequences sensible are in the vLLM internals guide.

Vision and video

The checkpoint is image-text-to-text with a vision encoder, and the card's examples pass images and videos through the standard OpenAI content array. The llama.cpp port carries the projector: the PR author verified image input end to end with llama-mtmd-cli, a Q4_K_M base, and the F16 mmproj file, after fixing a crash where every image request hit the end-of-segment token. For hour-scale video the card suggests raising longest_edge in video_preprocessor_config.json to 469,762,048, which corresponds to 224K video tokens, and vLLM lets you set frame sampling through mm_processor_kwargs with fps. Locally, treat video as a server feature; images work fine on a workstation.

Benchmarks, honestly

Every number below is Qwen's own, from the model card, with Qwen's harnesses and settings; nothing here is independently reproduced yet. The comparison set is Qwen3.8-27B (dense), Qwen3.7-Plus (397B, 17B active), and DeepSeek-V4-Flash-0731 (284B, 13B active). Rows selected for local-agent relevance:

Read it plainly. On Qwen's table the 6B-active model beats its own 27B dense sibling almost everywhere and edges DeepSeek V4 Flash on most agentic rows while losing two: repository-level code generation and the frontier agent exam. That is a coherent story for a model built to be cheap per token, and it is also the vendor's story on launch week. The footnotes matter: DeepSWE is the best of two harnesses, SWE-bench Pro was re-evaluated on a corrected task set by Qwen, and two of the agent benchmarks are in-house. Wait for the independent runs before you rebase a product on any row.

Rough edges, week one

• Old builds fail to load it. Support merged August 27; anything built before that has no qwen4_exp architecture. Build from master.
• Full GPU offload deadlocked on a DGX Spark. A tester with the 121 GiB unified GB10 reported CPU-only at 3.5 tok/s, 30 layers on the GPU at 4.9 tok/s, and a hang at -ngl 45 and above that looked like the driver blocking inside an allocation rather than returning an error; --no-mmap crashed at about 112 GB instead. Partial offload at 10 or 30 layers was slower than CPU-only because activations ping-pong across the bus. On that machine, as reported, the useful mode is full offload and it did not work yet.
• Strix Halo worked. A 128 GB Ryzen AI Max+ 395 box ran UD-IQ3_XXS with -ngl 99 -c 32768 on Vulkan, with a per-buffer mmap loader patch the tester wrote themselves. Numbers from patched forks vary; treat any tok/s you see for this platform as build-specific.
• Native 262K context aborts on CUDA. The rms_norm launch limit above. Use 131,072 or less on CUDA until the core fix lands.
• Slot save and restore re-prefills. Saving a server slot and restoring it is accepted but the restored hybrid state was not reused, so the next request re-prefilled the whole prompt. Plain prefix caching within a running server worked (57 tokens down to 4 on a repeated prompt in the report).
• Quantized tiers are not bit-identical. QSA versus dense attention under the budget is bit-identical at BF16 and F32 in the PR's tests; UD-IQ1_S showed a max logit delta of 2.84e-03. Expected, but worth knowing when you diff outputs across tiers.
• MTP in llama.cpp. In progress at merge time; the vLLM path is the one with a documented flag.
• Conversion needs a big host. The converter was rewritten to stream the 128 table shards instead of holding them, after the first version reached 300 GB of RSS on the real checkpoint. You will not convert this yourself on a laptop; use Unsloth's files.

What to do

If you own a 128 GB unified-memory machine, download UD-IQ3_XXS or UD-Q3_K_XL, build llama.cpp from master, leave mmap on, set the context to 32K or 64K, and measure your own decode rate before reading anyone else's. If you own a 24 GB or 32 GB card with 128 GB or more of system RAM, take UD-Q4_K_XL, keep the experts on the CPU with --cpu-moe, and expect the DDR5 bus to set your speed. If you have 192 GB or more, take the 4-bit or 5-bit tier and stop planning. If you serve a team, skip GGUF entirely and run the FP8 checkpoint on four GPUs with the vLLM recipe, MTP on, and the table in host RAM.

Then wait a month before deciding anything about speed. The architecture landed in the general-purpose engines on August 27, the first reported numbers were an order of magnitude under the bandwidth ceiling, and every part of that gap is the kind kernel writers close quickly. The memory story will not change, because it is arithmetic on the config: 6B active, 12 caching layers, and a 51B table that reads two rows per token.

FAQ

How much memory do I need?

Files run from 72.55 GB at UD-IQ1_S to 354.03 GB at BF16, and about a third of any file is the n-gram table you can leave on NVMe. A 128 GB unified-memory machine runs the 3-bit tiers resident; 192 GB runs UD-Q4_K_XL with the table in RAM; a 24 GB or 32 GB GPU runs the 4-bit tier with experts streamed from system RAM, paced by the DDR5 bus. Use the planner in Fig. 1 with your exact machine.

Why is a 125B model this small in memory at 262K context?

Only 12 layers keep a KV cache, with 2 KV heads each. The other 36 are Gated DeltaNet with a fixed state. About 7.3 GB of cache and indexer keys at the native context, about 29 GB at 1M, in BF16.

Is this Qwen4?

No. Qwen calls it "an experimental preview of the architecture that will underpin Qwen4," released as Qwen3.8-Flash-Next with the Qwen3.8 training recipe.

Should I keep the n-gram table in RAM or on disk?

Either. A token reads two rows, about 5 KB at q8_0. In RAM that is microseconds; on a decent NVMe it is on the order of a hundred microseconds for two page faults, which at tens of tokens per second is a few percent. Spend RAM on experts first.

What does it replace in a local stack?

The 6B-to-13B-active mixture slot: DeepSeek V4 Flash at 284B and the larger Qwen3.5 and 3.6 mixtures. On Qwen's numbers it trades a bigger download for a cheaper token and a smaller cache; on your numbers, measure the agent loop end to end, thinking on, before deciding.

Can I run it on a single 24 GB GPU?

Yes, with the routed experts in system RAM and the table on disk or in the page cache. The core is only a few gigabytes at 4-bit, so the GPU holds the core and the KV cache while --cpu-moe keeps every expert tensor on the CPU. What you give up is speed: the bus the experts sit on sets the decode rate, and dual-channel DDR5 is a fifth of a 4090's bandwidth. The estimator in Fig. 3 has a DDR5 row for exactly this case.

Why does the 1-bit tier exist at 72 GB when the table alone is 54 GB at q8_0?

Because at that tier the table is not q8_0. Unsloth has not published the per-tensor types for each tier, and the file arithmetic says the 1-bit and 2-bit builds quantize the table harder than the imatrix recipe in the llama.cpp PR does. The planner flags those tiers. If you care about the table's fidelity, which the bigram and trigram knowledge presumably does, stay at 3-bit or above.

Does the MTP head change memory?

It adds the MTP folder, 24.62 GB across six shards in Unsloth's GGUF repository, on top of whatever tier you chose, and it only pays off when the engine can use it. vLLM has the flag today; llama.cpp's MTP support for this architecture was still being written when the main port merged.