Model guide · October 30, 2026 · 15 min read

How to Run Mistral Large 4

Mistral released the weights of Mistral Large 4, which it calls Le Chonk, on October 29, 2026. It is a mixture of experts with about 1 trillion parameters and 52 billion active per token. It reads text and images, and it always reasons, at one of three effort levels. It is not a desktop model. The smallest GGUF is 233 GB, so the entry point is a machine with 256 GB of memory. Its cache is the interesting part. Mistral first cut the cache hard, then put most of it back to recover agent skill. What stayed are two memory tricks: 28 layers share a sparse indexer, and ten layers use one stored latent. This guide explains both, computes the cache from config.json, sizes the machines that hold the model, and gives the commands for llama.cpp, vLLM, and SGLang. For the 24B model that fits one GPU, read the Mistral Small guide.

Why Mistral put the cache back

The model you download is the second design of Mistral Large 4. Mistral explains the first one in its technical report. That design used three ways to shrink the key and value cache. Three of every four layers saw only the last 512 tokens through a sliding window. Every other full-attention layer reused the cache of the full-attention layer before it. In the second half of the stack, the full-attention layers read the cache that layer 35 wrote, a scheme called YOCO. The report says the three together made the cache 16 times smaller than full attention.

The small cache had a cost that perplexity did not show. Early fine-tuned checkpoints scored low on long-context agent work such as SWE-bench and BrowseComp, while knowledge benchmarks looked normal. The report calls this an agentic penalty. Mistral then changed a pretrained checkpoint in place. It removed the sliding window, so the window layers attend to the full context. It also removed the layer-to-layer reuse in the first half and gave those layers their own cache weights.

One mechanism stayed. A test at 8 × 10²³ FLOPs gave better scores with YOCO kept, so the second-half global layers still read layer 35. To control the attention cost at long context, Mistral then added DeepSeek Sparse Attention. A light indexer picks the 2,048 earlier tokens that each query attends to. The GLM-5.3 guide explains that indexer step by step. Mistral found that the former window layers in each group attend in similar ways, so each group now shares one indexer. The figure replays these versions on the same 72 layers.

The architecture

The values here come from the model's config.json. The class is MistralLarge4ForConditionalGeneration. The text model has 72 layers, a hidden size of 6,144, and 64 attention heads. Attention is multi-head latent attention, so each token stores a compressed latent of 512 values, not full keys and values per head. Layers with rotary position also store a 64-value rotary key. Every fourth layer, starting at layer 3, uses no rotary position at all. The Hugging Face release post calls this a 3:1 RoPE to NoPE pattern.

Layer 0 has a dense feed-forward block. The other 71 layers are mixture-of-experts blocks with 128 routed experts, four picked per token, plus one shared expert that always runs. The vocabulary has 204,800 tokens. The context limit is 1,048,576 positions. Mistral reaches it with LeRoPE, a YaRN-scaled rotary position with learned frequencies. The vision side is a 30-layer Pixtral encoder that the Hugging Face post puts at 1.6B parameters. The model card counts 49B active parameters per token, or 52B with the embeddings and the output layer.

So one layer has two kinds of sparsity. The router picks 4 of 128 expert blocks, which keeps compute per token low. The indexer picks 2,048 earlier positions, which keeps attention reads flat as the context grows. Neither one makes the model smaller. All 1T parameters still sit in memory, and that sets the size of the machine.

One indexer for three layers

The indexer is a small attention of its own: 32 heads of 128 dimensions that score every earlier token and keep the top 2,048. To score old tokens, a layer keeps one 128-value index key per token in its cache. In the transformers code, a layer marked shared has no indexer. It takes the picks of the previous layer that has one. So a shared layer stores no index key and scores nothing.

The config lists 44 full indexer layers and 28 shared ones. Layers 0 to 16 each run their own. From layer 16 on, the pattern repeats in groups of four. Layer 16 scores, and layers 17 and 18 reuse its picks. Layer 19 is a NoPE global layer with its own indexer. Then layer 20 scores for layers 21 and 22, and so on to layer 71. Each group of three former window layers shares one indexer, as the report describes.

The saving is in two places. The cache drops 28 index keys per token, 28 × 128 × 2 bytes = 7,168 bytes in BF16. Compute and memory reads also drop. Each indexer reads every cached index key once per decode step. At 1,048,576 tokens, one indexer reads 128 × 2 × 1,048,576 bytes, about 268 MB per step. With 44 indexers the scan is 11.8 GB per step. With one indexer per layer it would be 19.3 GB. At long context that scan, not the 2,048 picked tokens, is the main read of sparse attention.

Ten layers, one latent

The second trick is in kv_source_layers, a list with one entry per layer. Most layers point at themselves. Nine entries point at layer 35: layers 39, 43, 47, 51, 55, 59, 63, 67, and 71. So ten layers attend over one stored latent. Layer 35 writes it, and the nine later layers read it. A reader has no kv_a_proj_with_mqa, the projection that makes a latent, and it writes nothing to the cache. It keeps its own kv_b_proj, so it reads the same 512 stored values through its own weights.

All ten are NoPE layers. Their latent carries no rotary position, so a reader at layer 59 can use what layer 35 stored with no change. This sharing removes 9 × 512 × 2 bytes = 9,216 bytes per token in BF16. The machine at the top of the page draws it as a raised tank above the cell that holds layer 35. Pipes run down to the nine cells whose fourth layer reads it.

One point about the count. Some release notes say ten layers read layer 35. The config has nine readers. Ten is the number of layers that use the one latent, if you count layer 35 itself.

The cache per token

With both kinds of sharing known, the per-token cache is a sum over the 72 layers. In BF16:

• 54 RoPE layers each store 512 + 64 = 576 values: 54 × 576 × 2 = 62,208 bytes.
• 9 NoPE layers that own a latent store 512 values: 9 × 512 × 2 = 9,216 bytes.
• 9 NoPE readers of layer 35 store nothing.
• 44 indexer layers each store a 128-value key: 44 × 128 × 2 = 11,264 bytes.

The total is 82,688 bytes per token, about 80.8 KiB. Without either kind of sharing it would be 99,072 bytes. Sharing saves 16,384 bytes per token, or 16.5%. Multiply by the context to get one session. At 32,768 tokens the cache is 2.71 GB. At 131,072 it is 10.84 GB, and at the full 1,048,576 it is 86.70 GB. At that length the sharing keeps 17.18 GB out of memory.

Two engine details change these numbers. First, the servers default to an FP8 cache on Blackwell GPUs. The vLLM engine stores each latent in a 656-byte FP8 layout, described in its DeepSeek-V3.2 post. With FP8 index keys, the cache comes to about 47 KB per token. Second, tensor parallelism copies the whole cache to every GPU. A latent has one head, so the engine cannot split it across GPUs by head. Decode context parallelism, called DCP, splits it by position instead.

The vendor numbers show both effects. The vLLM recipe reports 3.60M cache tokens for NVFP4 on eight GB300 GPUs with tensor parallelism, and 28.76M with DCP8. That is eight times as many. The arithmetic above predicts 3.59M for the first case. The SGLang cookbook reports about 2.49M FP8 cache tokens on eight B300 GPUs, and the same arithmetic gives 2.39M.

Every build and its size

Mistral publishes two server checkpoints. The FP8 repository uses 128 × 128 block scales. The NVFP4 repository quantizes the routed experts and keeps attention and shared experts at FP8. It runs only on Blackwell GPUs. The llama.cpp team at ggml-org and Unsloth publish GGUF builds made from the FP8 release. Sizes below are sums of each Hugging Face file listing, in decimal gigabytes. Top-1 agreement is the share of tokens where the quant picks the same next token as the full model, from Unsloth's own tests.

Unsloth also publishes the DSpark draft head as a 2.25 GB Q8_0 GGUF, and both GGUF repositories carry a 3.4 GB vision projector. The ggml-org IQ2_XXS keeps attention, the indexer, the shared experts, the dense first layer, the embeddings, and the output head at Q8_0. Only the routed experts drop to 2 bits. AMD publishes an MXFP4 build for MI350 GPUs at about 578 GB, which the vLLM recipe lists.

Which machines hold it

Size the machine in three parts: the weights, the cache for the context you need, and a reserve for the runtime. The figure does that sum for seven real machines. On a Mac or a CPU server, llama.cpp puts everything in one memory. On a GPU server, vLLM and SGLang split the weights across GPUs, and the split mode decides how many cache tokens fit.

Read the figure from the smallest machine up. A 256 GB Mac Studio holds the 233 GB UD-IQ1_S. Its 256 GiB of memory is 274.88 GB. A GPU limit of 240 GiB leaves about 20.7 GB for the cache, or about 250,000 tokens. The limit there is quality, because the 1-bit build picks the same next token as the full model only 90.6% of the time. A 512 GB Mac Studio is the comfortable single box. With Q2_K or UD-Q2_K_XL it has room for the full 1,048,576-token cache in f16. The Hugging Face team measured Q2_K on one M3 Ultra at about 90 tokens per second for prefill and 15 for generation.

Four RTX PRO 6000 cards with 96 GB each hold IQ2_XXS. llama.cpp splits the layers, and the cache with them, so nothing is copied. Hugging Face ran that build at an 800K context on four of these cards. Generation was 30 tokens per second at short context and 3 at long context. A CPU server with 768 GB of RAM holds the 4-bit builds, but plain RAM bandwidth makes generation slow.

In a data center, the smallest deployment is four GB300 GPUs with NVFP4. That is the setup the technical report names. The FP8 build needs about 133 GB of weights per GPU across eight GPUs, so it fits eight B300 or B200 GPUs. It does not fit eight H200 GPUs. The vLLM recipe runs FP8 on 16 H200 across two nodes. For a first test with no hardware, use the Mistral API, which serves the model as mistral-large-4. The release post prices it at $0.70 per million input tokens, $0.07 for cached input, and $2.10 for output.

A worked example shows the arithmetic for one agent session on the 512 GB Mac. Q2_K is 380.71 GB, the buffers take about 4 GB, and a 131,072-token cache adds 10.84 GB. The sum is 395.55 GB. The GPU limit of 496 GiB is 532.58 GB, so the session fits with 137 GB left. Time is the real limit. At 90 tokens per second, a 131,072-token prompt takes about 24 minutes to prefill. Keep long prompts in the prompt cache so that the next turn reads them, not recomputes them.

llama.cpp on a big Mac

Support arrived in llama.cpp on October 29 with pull request #31376, which extends the sparse indexer and the DSpark head to the new model. Use a build from that day or later. The Unsloth card asks for its own llama.cpp build or Unsloth Desktop. On a Mac, first raise the GPU memory limit, here to 496 GiB on a 512 GB machine:

sudo sysctl iogpu.wired_limit_mb=507904

hf download unsloth/Mistral-Large-4-1T-A52B-GGUF \
  --include "UD-Q2_K_XL/*" --local-dir ml4
hf download unsloth/Mistral-Large-4-1T-A52B-GGUF \
  --include "dspark-Mistral-Large-4-1T-A52B-Q8_0.gguf" --local-dir ml4

llama-server \
  --model ml4/UD-Q2_K_XL/Mistral-Large-4-1T-A52B-UD-Q2_K_XL-00001-of-00009.gguf \
  -md ml4/dspark-Mistral-Large-4-1T-A52B-Q8_0.gguf \
  --spec-type draft-dspark --spec-draft-n-max 3 \
  --ctx-size 131072 \
  --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
  --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --host 127.0.0.1 --port 8080

llama.cpp loads the other eight shards on its own. The -md line and the two --spec flags turn on DSpark, the draft head. Unsloth reports a mean acceptance length of about 2.5 tokens and about twice the decode speed at batch 1. Delete those three flags to run without a draft. Add --mmproj with the projector file for images. On a 256 GB machine, download UD-IQ1_S/* instead, point --model at its first shard, and set the limit to 245760. The draft head still fits there. With it loaded, about 223,000 cache tokens stay free, so --ctx-size 131072 works.

The ggml-org route is shorter. The ggml-org card runs llama serve -hf ggml-org/Mistral-Large-4-1T-A52B-GGUF:IQ2_XXS through Llama App. That card stores the recommended sampling in the GGUF metadata and notes that reasoning cannot be turned off.

vLLM

No vLLM release wheel supports the model yet, so the recipe uses the vllm/vllm-openai:mistral-large-4 image. This is its NVFP4 command for one tray of four GB300 GPUs, with DSpark on:

docker run --gpus all \
  --privileged --ipc=host -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=6000 \
  vllm/vllm-openai:mistral-large-4 mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 4 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}'

The -p 127.0.0.1:8000:8000 flag publishes the port on this host only. The server has no authentication by default, so add --api-key with a secret before other machines can reach the port.

For the FP8 checkpoint on eight GPUs, the model card uses mistralai/Mistral-Large-4-1T-A52B with --tensor-parallel-size 8. The engine uses its FP8 latent cache by default, so you pass no cache flags. The recipe validated contexts up to 262,144 tokens, although the checkpoint allows 1,048,576. When the cache is the limit, add --decode-context-parallel-size equal to the tensor-parallel size. That splits the cache across the GPUs, for more tokens at a higher time per token.

SGLang

SGLang also has a release image, lmsysorg/sglang:mistral-large-4. This is the model card's command for eight GPUs, with one change. The card binds to 0.0.0.0, and this command binds to 127.0.0.1, as the vLLM command does:

sglang serve \
  --model-path mistralai/Mistral-Large-4-1T-A52B \
  --tp-size 8 \
  --mem-fraction-static 0.85 \
  --speculative-algorithm DSPARK \
  --max-running-requests 64 \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --host 127.0.0.1 \
  --port 30000

Use 0.0.0.0 only behind an auth proxy or with --api-key set. The SGLang cookbook loads the draft from the checkpoint, so do not pass a draft path or a draft length. With DSpark on, it caps running requests at 48 unless you set the flag. The cache is FP8 on Blackwell and BF16 on Hopper. The cookbook measured 3.71 accepted tokens per four-token verify step on eight B300 GPUs. It also offers decode context parallelism with --dcp-size when you need more cache.

Effort, sampling, tools

Effort. Reasoning is always on. Pass reasoning_effort as low, medium, or high, and high is the default. Mistral says low replaces the old none mode. Use low for chat, medium for daily work, and high for agents.

Sampling. Mistral recommends temperature=1.0 and top_p=0.95 at every effort. In llama.cpp, add --top-k 0 --min-p 0.0 so that no other filter changes those values.

Budget. On vLLM, max_completion_tokens counts the reasoning tokens as well as the answer. Set it high for high effort, or the answer gets cut off inside the reasoning.

Parsers. On vLLM, pass --reasoning-parser mistral and --tool-call-parser mistral. On SGLang, auto picks the same parsers. Without them, the reasoning stays in the content inside [THINK] tags and tool calls come back as [TOOL_CALLS] text.

Failure modes and fixes

• vLLM does not know the architecture. The release wheels lack support. Use the vllm/vllm-openai:mistral-large-4 image, or the ROCm image on AMD.
• The tokenizer fails to load. mistral-common older than 1.13.0 cannot read tekken.json. Upgrade it, or use a release image.
• Raw [THINK] or [TOOL_CALLS] text in replies. The server has no parser. Add the reasoning and tool-call parser flags.
• Out of memory on eight H200 GPUs. FP8 weights need about 133 GB per GPU. Use 16 H200 across two nodes, as the vLLM recipe does.
• NVFP4 fails on Hopper. It needs Blackwell. Use the FP8 build on Hopper.
• A Mac stops at load. The GPU memory limit is too low for a 300 GB file. Raise iogpu.wired_limit_mb, or pick a smaller quant.
• Long prompts take minutes on a Mac. Prefill is about 90 tokens per second at Q2_K. Reuse the prompt cache across turns and keep tool output short.

The license

The model card names the Mistral Ecosystem Open-Weight License. It calls it "a modified version of the MIT license, for both commercial and non-commercial use". Section 2 of the license adds one condition. A licensee that "operates a Model as a Service business" must "enter into a separate commercial license with Mistral" before commercial use. The license defines that business as third-party access to the model for inference or post-training. The third party must have control over the inputs, parameters, or training data. Internal use is exempt. So is an end-user product that embeds the model inside its own features.

FAQ

What is the smallest machine that runs it?

A machine with 256 GB of unified memory or RAM, such as a 256 GB Mac Studio, with the 233 GB UD-IQ1_S GGUF in llama.cpp. A 256 GB Mac Studio with the GPU limit at 240 GiB keeps about 250,000 cache tokens free after the weights. That is one 128K session on paper. A 256 GB server that keeps 32 GiB for the system has room for only about 42,000. A 512 GB Mac Studio runs the 2-bit builds with room for the full cache.

How much cache does it use per token?

82,688 bytes in BF16, from config.json. At 131,072 tokens that is 10.84 GB, and at 1,048,576 tokens it is 86.70 GB. An FP8 cache is about 47 KB per token.

What do the shared indexer and the KV sharing save?

The 28 shared indexer layers store no index key, which saves 7,168 bytes per token. The nine readers of layer 35 store no latent, which saves 9,216 bytes. Together that is 16,384 bytes per token, or 17.18 GB at the full window.

Can I turn off reasoning?

No. Set reasoning_effort to low, medium, or high. High is the default.

Is it free for commercial use?

Yes, with one condition. A company that operates a Model-as-a-Service business needs a separate commercial license from Mistral before commercial use.