Local model guide · updated July 31, 2026 · 15 min read

How to Run DeepSeek V4 Flash Locally

DeepSeek-V4-Flash-0731, released July 31, is the most runnable frontier-class model yet: 284B parameters with only 13B active, a 1M-token context window, and, because DeepSeek trained it quantization-aware, a Q8 GGUF that is bit-for-bit lossless at 162 GB. This guide is interactive: set your machine's memory once, pick a quant, and every command on the page rewrites itself for your choice.

Pick your quant

Slide to your machine's total memory (RAM + VRAM, or unified), then click a row. The verdict column updates, and so does every command below.

The odd shape of this table is the QAT story: because the experts are already 4-bit at the source, "4-bit" saves almost nothing over lossless Q8 (155 vs 162 GB, both quantizing only the non-expert 4 percent). The real size wins come from the sub-4-bit tiers that re-quantize the experts, at a measured quality cost. The 3-bit at 103 GB is the recommended sweet spot: it fits 110 to 128 GB machines that the lossless tiers cannot reach.

Why Q8 is truly lossless

DeepSeek trained V4-Flash quantization-aware: the routed experts, 96 percent of all weights, are stored natively in MXFP4, and the remainder in FP8 or BF16. A GGUF that repacks those experts bit-for-bit and dequantizes FP8 to BF16 reproduces the official checkpoint exactly. Unsloth's measurement across all 1,328 tensors: 100 percent bit-identical, KL divergence approximately zero, 100 percent top-token agreement. Conversions that re-quantize the experts into a different 4-bit grid (Q4_K) round nearly every weight for a file that is not even smaller. Practical rule: for this model, prefer quants that keep the native MXFP4 experts, and treat third-party conversions with re-quantized experts as strictly worse.

Run with llama.cpp

Build once (CUDA on, or -DGGML_CUDA=OFF for CPU and Apple):

git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
  --target llama-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp/

Download your chosen quant (UD-IQ3_XXS) manually for reliability, then run. Both blocks track your selection above:

pip install huggingface_hub
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
  --local-dir DeepSeek-V4-Flash-0731-GGUF \
  --include "*UD-IQ3_XXS*"

./llama.cpp/llama-server \
  -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ3_XXS \
  --jinja \
  -ngl 99 \
  -ot ".ffn_.*_exps.=CPU" \
  --temp 1.0 --top-p 1.0 --min-p 0.0 \
  --cache-type-k q8_0 -fa on \
  -c 65536 --host 127.0.0.1 --port 8080

The load-bearing flags: -ot ".ffn_.*_exps.=CPU" keeps the expert tensors in system RAM so attention lives on whatever GPU you have (drop it on a big unified-memory Mac); --jinja applies the model's chat template, which carries the reasoning modes; the sampler is DeepSeek's official one, temperature 1.0 and top-p 1.0, with top-p 0.95 only for agentic runs on 0731. If you have spare VRAM, trade offload for speed progressively: ".ffn_(up|down)_exps.=CPU" keeps gate experts on GPU, ".ffn_(up)_exps.=CPU" keeps more still.

Speed: the 13B-active advantage

Here is why this 284B model is pleasant where the old 671B was patient-only. Per token, a MoE reads its active parameters: 13B. At roughly 4 effective bits that is ~6.5 GB per token from wherever the experts live:

experts in DDR5 (80 GB/s):   80 / 6.5  ≈ 12 tok/s ceiling
experts in M-Ultra (800):   800 / 6.5  ≈ 120 tok/s ceiling
671B-class, same DDR5:       80 / 18   ≈  4 tok/s   (37B active)

Real throughput lands at half to three quarters of ceiling, but the ranking holds: V4-Flash on a plain 128 GB desktop is a usable daily driver, not a demo. The full arithmetic is in tokens per second is a memory bandwidth number.

Reasoning modes

V4 thinks by default at High; there are three modes, switched through the chat template, and llama.cpp now has a direct flag:

# reasoning effort
--chat-template-kwargs '{"reasoning_effort":"max"}'
--chat-template-kwargs '{"reasoning_effort":"high"}'
# disable thinking entirely
--chat-template-kwargs '{"enable_thinking":false}'
# or simply
--reasoning off

Two template subtleties worth knowing because they bite tool use: the correct template prepends DeepSeek's system prompt, and it must retain reasoning_content across tool-call turns. Early community templates dropped it; the Unsloth jinja adds it back, tested against 4,000 conversations. This is the same preserved-thinking lesson as Kimi K3 and OpenAI's ARC post: reasoning models degrade when the harness discards their thoughts.

Serve + wire an agent

The llama-server command above already exposes /v1/chat/completions. Point any coding agent at it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"one-line hello"}]}'
# agents: base_url = http://127.0.0.1:8080/v1, api_key = anything
# agentic sampler: add "top_p": 0.95 per DeepSeek's 0731 guidance

On a Mac

Unified memory makes the tiers simple: a 128 GB MacBook Pro or Studio runs the 3-bit (UD-IQ3_XXS, needs ~110 GB); a 192 GB Ultra runs lossless Q8 with room for real context; drop the -ot flag since there is nothing to offload past. Build llama.cpp with -DGGML_CUDA=OFF (Metal is on by default) and use the same commands. LM Studio will list the same GGUF repo for the no-terminal path once the quants finish uploading; the model is a day old, so check.

Troubleshooting

• Multi-part GGUF. Q8 ships as 5 shards; point llama.cpp at shard 00001, it loads the rest. Merge with llama-gguf-split --merge only if another tool demands one file.
• Gibberish or broken tool calls. Wrong template. Use --jinja, and a template that retains reasoning_content; a day-one third-party GGUF may not.
• OOM at load. Drop a tier in the picker above, lower -c, keep --cache-type-k q8_0, offload all experts.
• Think Max rambles past context. By design; it wants 384K+. Use High for daily work.
• Slow downloads. Use the manual hf download path with --include rather than letting llama.cpp pull.

FAQ

How much RAM? By quant: ~92 GB (1-bit), ~102 (2-bit), ~110-135 (3-bit), ~162 (4-bit), ~169 (lossless Q8), counting RAM + VRAM together. The 3-bit on a 110-128 GB box is the sweet spot.

Why is Q8 lossless? QAT: 96 percent of weights ship natively in MXFP4; repack them bit-for-bit, keep the rest BF16, and you have the official checkpoint. Measured: ~0 KLD, 100 percent top-token match.

How fast on offload? 13B active ≈ 6.5 GB reads per token, so ~10-12 tok/s ceiling on dual-channel DDR5, several times the 671B-class experience.