How to Run DeepSeek V4 Flash Locally
DeepSeek-V4-Flash-0731, released July 31, is the most runnable frontier-class model yet: 284B parameters with only 13B active, a 1M-token context window, and, because DeepSeek trained it quantization-aware, a Q8 GGUF that is bit-for-bit lossless at 162 GB. This guide is interactive: set your machine's memory once, pick a quant, and every command on the page rewrites itself for your choice.
The GPU card holds attention, the router and the shared expert. The routed experts sit in system RAM, as the -ot flag in this guide puts them. The row shows the first 16 of the 43 layers. Decode a token and watch it cross to RAM for its 6 picks.
Every layer keeps a 128-token window. Past that, the layers alternate between a 4:1 compressed cache with sparse picks and a 128:1 compressed cache.
Lit squares are the 6 picks for one example token plus 1 shared expert that always runs.
hidden 4,096 · 64 query heads · 1 KV head of width 512 · window 128 · context 1,048,576
BF16 to the smallest usable Unsloth dynamic quant. The Q8 at 162 GB is bit-for-bit lossless because the model was trained quantization-aware; the 84 GB TQ1_0 is the "it actually starts on my machine" end. Pick your point on that line below.
Pick your quant
Slide to your machine's total memory (RAM + VRAM, or unified), then click a row. The verdict column updates, and so does every command below.
The odd shape of this table is the QAT story: because the experts are already 4-bit at the source, "4-bit" saves almost nothing over lossless Q8 (155 vs 162 GB, both quantizing only the non-expert 4 percent). The real size wins come from the sub-4-bit tiers that re-quantize the experts, at a measured quality cost. The 3-bit at 103 GB is the recommended sweet spot: it fits 110 to 128 GB machines that the lossless tiers cannot reach.
Why Q8 is truly lossless
DeepSeek trained V4-Flash quantization-aware: the routed experts, 96 percent of all weights, are stored natively in MXFP4, and the remainder in FP8 or BF16. A GGUF that repacks those experts bit-for-bit and dequantizes FP8 to BF16 reproduces the official checkpoint exactly. Unsloth's measurement across all 1,328 tensors: 100 percent bit-identical, KL divergence approximately zero, 100 percent top-token agreement. Conversions that re-quantize the experts into a different 4-bit grid (Q4_K) round nearly every weight for a file that is not even smaller. Practical rule: for this model, prefer quants that keep the native MXFP4 experts, and treat third-party conversions with re-quantized experts as strictly worse.
Run with llama.cpp
Build once (CUDA on, or -DGGML_CUDA=OFF for CPU and Apple):
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
--target llama-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp/Download your chosen quant (UD-IQ3_XXS) manually for reliability, then run. Both blocks track your selection above:
pip install huggingface_hub
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
--local-dir DeepSeek-V4-Flash-0731-GGUF \
--include "*UD-IQ3_XXS*"./llama.cpp/llama-server \
-hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ3_XXS \
--jinja \
-ngl 99 \
-ot ".ffn_.*_exps.=CPU" \
--temp 1.0 --top-p 1.0 --min-p 0.0 \
--cache-type-k q8_0 -fa on \
-c 65536 --host 127.0.0.1 --port 8080The load-bearing flags: -ot ".ffn_.*_exps.=CPU" keeps the expert tensors in system RAM so attention runs on whatever GPU you have (drop it on a big unified-memory Mac); --jinja applies the model's chat template, which carries the reasoning modes; the sampler is DeepSeek's official one, temperature 1.0 and top-p 1.0, with top-p 0.95 only for agentic runs on 0731. If you have spare VRAM, trade offload for speed progressively: ".ffn_(up|down)_exps.=CPU" keeps gate experts on GPU, ".ffn_(up)_exps.=CPU" keeps more still.
Speed: the 13B-active advantage
Here is why this 284B model is pleasant where the old 671B was patient-only. Per token, a MoE reads its active parameters: 13B. At roughly 4 effective bits that is ~6.5 GB per token from wherever the experts sit:
experts in DDR5 (80 GB/s): 80 / 6.5 ≈ 12 tok/s ceiling
experts in M-Ultra (800): 800 / 6.5 ≈ 120 tok/s ceiling
671B-class, same DDR5: 80 / 18 ≈ 4 tok/s (37B active)Real throughput lands at half to three quarters of ceiling, but the ranking holds: V4-Flash on a plain 128 GB desktop is a usable daily driver, not a demo. The full arithmetic is in tokens per second is a memory bandwidth number.
Reasoning modes
V4 thinks by default at High; there are three modes, switched through the chat template, and llama.cpp now has a direct flag:
# reasoning effort
--chat-template-kwargs '{"reasoning_effort":"max"}'
--chat-template-kwargs '{"reasoning_effort":"high"}'
# disable thinking entirely
--chat-template-kwargs '{"enable_thinking":false}'
# or simply
--reasoning off| Mode | Use for | Context advice |
|---|---|---|
| Non-think | Chat, fast lookups | any |
| Think High (default) | Coding, agents | 64K+ |
| Think Max | Hard reasoning | 384K minimum: the traces are enormous |
Two template subtleties worth knowing because they bite tool use: the correct template prepends DeepSeek's system prompt, and it must retain reasoning_content across tool-call turns. Early community templates dropped it; the Unsloth jinja adds it back, tested against 4,000 conversations. This is the same preserved-thinking lesson as Kimi K3 and OpenAI's ARC post: reasoning models degrade when the harness discards their thoughts.
Serve + wire an agent
The llama-server command above already exposes /v1/chat/completions. Point any coding agent at it:
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":"one-line hello"}]}'
# agents: base_url = http://127.0.0.1:8080/v1, api_key = anything
# agentic sampler: add "top_p": 0.95 per DeepSeek's 0731 guidance127.0.0.1. Never 0.0.0.0 without an authenticating proxy in front; a 1M-context frontier model on an open LAN port is a gift to whoever finds it.On a Mac
Unified memory makes the tiers simple: a 128 GB MacBook Pro or Studio runs the 3-bit (UD-IQ3_XXS, needs ~110 GB); a 192 GB Ultra runs lossless Q8 with room for real context; drop the -ot flag since there is nothing to offload past. Build llama.cpp with -DGGML_CUDA=OFF (Metal is on by default) and use the same commands. LM Studio will list the same GGUF repo for the no-terminal path once the quants finish uploading; the model is a day old, so check.
Troubleshooting
- Multi-part GGUF. Q8 ships as 5 shards; point llama.cpp at shard 00001, it loads the rest. Merge with
llama-gguf-split --mergeonly if another tool demands one file. - Gibberish or broken tool calls. Wrong template. Use
--jinja, and a template that retainsreasoning_content; a day-one third-party GGUF may not. - OOM at load. Drop a tier in the picker above, lower
-c, keep--cache-type-k q8_0, offload all experts. - Think Max rambles past context. By design; it wants 384K+. Use High for daily work.
- Slow downloads. Use the manual
hf downloadpath with--includerather than letting llama.cpp pull.
FAQ
How much RAM? By quant: ~92 GB (1-bit), ~102 (2-bit), ~110-135 (3-bit), ~162 (4-bit), ~169 (lossless Q8), counting RAM + VRAM together. The 3-bit on a 110-128 GB box is the sweet spot.
Why is Q8 lossless? QAT: 96 percent of weights ship natively in MXFP4; repack them bit-for-bit, keep the rest BF16, and you have the official checkpoint. Measured: ~0 KLD, 100 percent top-token match.
How fast on offload? 13B active ≈ 6.5 GB reads per token, so ~10-12 tok/s ceiling on dual-channel DDR5, several times the 671B-class experience.
Keep reading