Model guide · September 11, 2026 · engine status updated September 24 · 14 min read

How to Run DeepSeek-V4.1-Flash: 510 GB, and 203 GB of It Is a Lookup Table

On September 10 DeepSeek put DeepSeek-V4.1-Flash on Hugging Face under MIT: a 552B-parameter backbone that activates 8B parameters per token in prefill and 16B in decode, a million tokens of context, and a global KV cache the model card puts at 890 bytes per token. The download is 510 GB. I read the header of every one of its 48 safetensors shards to see where those bytes sit, and the answer changes how you should plan hardware: 289 GB are routed experts, and 203 GB are two hash tables that the model reads 48 rows from per token and never computes over. That table does not need a GPU. This guide covers the architecture, where each byte can live, and the three engine paths that run it today.

What changed from V4-Flash

The model it replaces, DeepSeek-V4-Flash-0731, was a 284B mixture of experts with 13B active that fit on a large Mac as a lossless Q8 file. V4.1-Flash is bigger in every storage number and smaller in every per-token number. The model card's base-model table lists it at 552B backbone parameters against 284B, and at 8B active in prefill and 16B in decode against 13B. It is natively multimodal, with a 32-layer vision encoder trained from scratch, and DeepSeek says it was pretrained on 45T tokens with sparse attention at 64K and extended to 1M at the 34T mark.

The headline in the card title is Pushing the Limits of KV Cache Compression, and the numbers back it: 890 bytes of global KV per token, which the card describes as roughly a quarter of V4-Flash and 437 times smaller than DeepSeek V1. At that rate a full million-token context needs about 0.93 GB of global KV. The catch sits on the other side of the ledger. The native checkpoint is 510.3 GB, nearly twice V4-Flash, so this is not a laptop model and it is not a single-GPU model. It is a model you serve on four Blackwell GPUs or eight Hoppers, or run slowly on one big unified-memory box through a community branch.

The architecture

Five pieces are new or reworked compared with V4. All of them show up in config.json and the card.

Causal Encoder-Decoder. The 40 layers split into a 20-layer causal encoder and a 20-layer decoder. The decoder's global KV is projected from the encoder's final hidden states instead of from each decoder layer. The card credits this design for the split between 8B active in prefill and 16B in decode. It does not spell out the mechanics, but SGLang's open work on "encoder-only prefill" for disaggregated serving reads the same way: prompt tokens need the half of the network that produces the shared KV, and generated tokens run the whole stack. DeepSeek pitches this at input-heavy agent workloads, where prompts are long and replies are short.

CSA2 sparse attention. Every layer keeps a 128-token sliding window. Global attention reads a compressed latent that only a few source layers write: kv_source_layer_ids is [2, 8, 14, 20], layers 2 to 19 pool two positions into one, layers 20 to 39 keep one row per position, and eight index layers pick a top-512 candidate set per query from those keys. The card names three static per-layer modes, Full, Reindex, and Reuse, that share the main KV and index keys across layers.

SWA bounded replay. The sliding-window state is not persisted. When a cached prefix comes back, the engine replays the last 128 tokens to rebuild it, which the card says cuts the persistent KV footprint to about an eighth of V4-Flash's.

Engram. Two layers, 1 and 14, add rows looked up from n-gram hash tables into the residual stream through a learned gate. This is the part that reshapes deployment, and it gets its own section below.

DSpark. A bundled speculative drafter: three extra layers with their own 128-expert MoE that draft blocks of five tokens. The card also mentions a revised four-copy residual stream it calls Single-Pass mHC.

Where the 510 GB goes

A safetensors file starts with an 8-byte length and a JSON header listing every tensor's dtype, shape, and byte range. Reading those 48 headers with HTTP range requests, without downloading the weights, gives an exact byte count per tensor class:

Two numbers explain most of the confusion in early threads. Hugging Face's parameter counter reports 763B for this repo, because it counts the two Engram tables, 2 x ~384M rows x 256 = 196.6B parameters, on top of the 552B backbone plus the drafter and vision tower. The card's 552B leaves them out, and that is fair: they hold parameters that are looked up, never multiplied. The second number is 18.4 GB. That is everything outside the experts and the tables, the part every token touches on every layer.

So the placement question is three questions. Where do 289 GB of experts go, which need fast memory because six are read per token per layer? Where do 203 GB of tables go, which need only a few kilobytes per token? And how much HBM is left for the KV cache once the weights land? The figure below answers the third one for the engine defaults.

The pattern that falls out: on four B200s the SGLang default, which shards the tables across the GPUs, leaves about 16 GB per GPU for KV, and moving the tables to host memory multiplies that headroom several times over. On GB300s the difference is smaller in ratio and larger in absolute sessions. On eight H200s the model fits either way with room to spare. One more thing the figure assumes: with plain tensor parallelism a single-latent KV cache cannot be split by head, so each rank holds the whole latent for its sequences. That is why the session count comes from one GPU's headroom and not the sum.

Engram: the table that is never multiplied

Engram comes from the January paper Conditional Memory via Scalable Lookup, and vLLM's Engram page explains the runtime side clearly. At each position the engine normalizes the last few token ids (NFKC, accent stripping, lowercasing, whitespace collapsing, so " The", "the", and "THE" collapse to one id in a compressed vocabulary of 99,092), hashes the suffix n-grams of order 2, 3, and 4 with 8 independent hash heads each, and reads one row per (order, head) pair from a prime-sized bucket range. That is 24 rows per Engram layer and 48 per token. A learned gate compares each retrieved row with the hidden state and suppresses it when it contradicts the context.

The property that matters for hardware: the row indices depend only on token ids, so they are known before the forward pass starts. An engine can issue the reads early from host memory and have the rows waiting by the time layer 1 needs them. Each row is 256 bytes of FP8 plus 8 bytes of scales, so a token costs 12,672 bytes of reads regardless of context length.

At interactive decode rates the host traffic is noise: 2,000 tokens per second across all users is 25 MB/s. Prefill is where it shows up, because a 128K-token prompt needs about 1.7 GB of table reads. vLLM's prefetch change measured exactly that: on four GB200s, moving the lookup onto a side stream took 0.4 to 0.9 percent off time to first token at 8K and 16K prompts and changed nothing in decode, which held at about 7.5 ms per token for a single user.

This is why the three engines disagree on defaults. vLLM puts the tables in pinned host memory by default and reads them over unified virtual addressing, so the 203 GB never touch HBM. SGLang row-shards them across the tensor-parallel GPUs by default, which costs an all-reduce per Engram layer, and offers SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 to move them to one shared host copy. The community llama.cpp branch maps them with mmap and pages rows in on demand. All three are correct. They trade HBM for host RAM, and host RAM is the cheaper of the two by a wide margin.

890 bytes per token

The card's KV figure is the sum of the design choices above: one 512-wide latent per stored position, only four layers that store it, half of them pooling two positions into one, the main KV kept in FP4 (E2M1 with one E4M3 scale per 16 channels), and a sliding window that is rebuilt instead of stored. Engines choose their own storage format, so the number you get depends on the engine: vLLM's release notes say its SM100 path keeps the whole KV in MXFP8, which trades some of that compactness for kernel speed. Treat 890 bytes as the floor.

To see what the floor buys, compare it with the generations behind it. The rows below use the card's ratios for V4-Flash and V1, plus a counterfactual: the same 40 layers with a conventional grouped-query cache of 8 KV heads of width 128 in BF16, which works out to 163,840 bytes per token.

Engine status

The guide is dated to the release week; this table reflects what I could verify on September 24.

vLLM

vLLM 0.30.0 and later serve it with no model-specific flags; Engram is detected from the checkpoint and offloaded to pinned host memory by default. Plan for about 203 GB of pinned host RAM per data-parallel replica on top of the 307 GB of weights spread across the GPUs.

pip install -U "vllm>=0.30.0"

vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --tensor-parallel-size 4 \
  --reasoning-parser deepseek_v41 \
  --tool-call-parser deepseek_v41 --enable-auto-tool-choice

The parser names come from vLLM's reasoning and tool-parser registries. Two variants from the Engram docs are worth knowing. If host RAM is scarce and HBM is not, keep the tables on the GPU:

vllm serve deepseek-ai/DeepSeek-V4.1-Flash --tensor-parallel-size 4 \
  --engram-config.cpu_offload false

If you run several data-parallel replicas on one node, let them share one host copy of each table through /dev/shm, which vLLM enables automatically when it can. In a container that needs --ipc=host or a large --shm-size, and it falls back with a warning when /dev/shm is too small. To spread one table copy across all TP x DP ranks instead:

vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
  --tensor-parallel-size 2 --data-parallel-size 4 \
  --engram-config '{"embedding_across_dp": true}'

Engram in vLLM does not support dual-batch overlap or microbatching, so leave --enable-dbo off. It needs a CUDA-class platform with UVA support and fails at startup otherwise.

SGLang

SGLang's DeepSeek-V4.1 cookbook is the most detailed operating manual for this model anywhere, and it verifies recipes for GB300 (the reference platform), B200, B300, H200, and MI350X. The GB300 low-latency recipe, with DSpark on:

docker run --gpus all --shm-size 32g --ipc=host -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=<your-hf-token>" \
  lmsysorg/sglang:dev-dsv41 \
  sglang serve --trust-remote-code \
    --model-path deepseek-ai/DeepSeek-V4.1-Flash \
    --tp 4 --ep-size 4 --mem-fraction-static 0.8 \
    --speculative-algorithm DSPARK --speculative-dspark-block-size 5 \
    --cuda-graph-max-bs-decode 64 \
    --reasoning-parser auto --tool-call-parser auto \
    --host 0.0.0.0 --port 30000

On 8x H200 the recipe uses --tp 8 --ep-size 8, pins --attention-backend dsv4 --moe-runner-backend flashinfer_mxfp4, and adds --enable-decoder-swa-bounded-replay. For high throughput, drop the two DSpark flags and add --max-running-requests 256: the cookbook explains that a DSpark step costs a fixed amount more than a plain decode step, so it pays at batch 1 and stops paying at large batch. Its advice on backends is blunt: on Blackwell, do not pass them. Overriding them leaves the 32-wide FP8 blocks on a Triton fallback that eats most of the batch-1 throughput.

Two opt-ins trade exactness for speed or memory. SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 moves the tables to one shared host copy with bitwise-identical output. --enable-decoder-swa-bounded-replay makes prefill faster but is not numerically equivalent to full prefill and refuses prompt logprobs.

llama.cpp on a branch

This is the only path to a single box, and it is early. Victor Cruz, who opened PR #28696, publishes GGUF files and a DGX Spark recipe. The PR description is worth reading for its failure modes alone: V4's sparse-attention builder silently drops the long-range half of attention on V4.1's compress ratios while still producing fluent text, and V4's FP8 dequantization assumes 128-wide blocks where V4.1 uses 32, which rescales every weight without an error.

git clone https://github.com/vcruz305/llama.cpp
cd llama.cpp && git checkout runtime/deepseek41
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121
cmake --build build --config Release -j 12

./build/bin/llama-cli -m DeepSeek-V4.1-Flash-Q2_K-00001-of-00007.gguf \
  -lm mmap -ngl 99 -cmoe -ot "engram_embd.weight=CPU" \
  -fa on -c 2048 -b 2048 -ub 1024 -t 20

The recipe measured 2.8 tokens per second on one DGX Spark with a CUDA build and 2.3 with a CPU build, because 246 GiB against 121.7 GiB of unified memory is twice oversubscribed and paging sets the pace. -lm mmap is required on that machine; without it the loader tries to allocate the whole file up front and fails. In the PR thread, a contributor who ported the DSpark drafter to the branch reported 17.7 to 22.8 tokens per second on a Threadripper 5975WX with two GPUs and the experts mostly on CPU. That is one user's report, not a benchmark, but it shows the shape of a workstation build: experts and tables in system RAM, dense layers on the GPUs. As of the recipe's September 11 status, the two-level candidate mask was the one piece of sparse attention not implemented, the MTP head and vision were not mapped, and nobody had measured quality against the reference.

On a Mac, nothing is verified. A 512 GB Mac Studio could hold the Q3_K_M file in unified memory, but the branch targets CUDA and CPU and I found no report of it running on Metal.

Effort, sampling, prompts

Reasoning effort is a continuous integer from 1 to 100, and the card's instruct results all use 100. SGLang maps the OpenAI-style tiers low, high, xhigh, and max onto that range, accepts a float between 0 and 0.99, and defaults to high when thinking is on; thinking itself is off unless a request sends a reasoning effort. DeepSeek recommends temperature=1.0, top_p of 0.95 or 1.0, and max_tokens of at least 256K for reasoning runs.

There is no Jinja chat template in the repo. The prompt format lives in encoding/encoding.py, and DeepSeek also released deepseek-recipe, Rust libraries with Python bindings that turn Messages, Chat Completions, and Responses requests into V4.1 prompts and parse the output back. Tool calls use spaced DSML tags that the V4 parser does not read, which is why both vLLM and SGLang ship a separate V4.1 tool parser.

Benchmarks, honestly

All numbers here are DeepSeek's own, at maximum effort. Against its predecessor the agentic gains are large: Terminal-Bench 2.1 goes from 82.7 to 90.6 and DeepSWE v1.1 from 54.4 to 74.2. Against the closed models in the same table it leads on some rows and trails clearly on the harder ones: Terminal-Bench 4.0 is 31.2 against 51.8 for Claude Opus 5. The base-model table is mixed: SimpleQA-Verified rises from 30.1 to 42.3 over V4-Flash, while MGSM drops from 85.7 to 80.2 and LongBench-V2 barely moves, 44.7 to 45.2.

The table I find most useful is the one comparing agent harnesses. On DeepSWE v1.1 the same model resolves 74.2 with the minimal mini-SWE scaffold and 69.8 inside Claude Code, a 4.4-point spread from the harness alone. That is the argument of the harness comparison measured by a lab on its own model.

Rough edges

• Chat template in the GGUFs. A reviewer on the llama.cpp PR found that the template embedded in the published Q2_K files is the V4 prompt format, not V4.1. Format prompts with encoding.py or deepseek-recipe until that is fixed.
• Old GGUF headers. Files converted before September 10 say deepseek4 and need redoing; files from before September 11 had wrong Engram keys. The recipe repo ships a KV-only fix script.
• Not bitwise stable in SGLang. Output changes with batch composition today, and --enable-deterministic-inference is refused on this backend.
• Compressed vocab must match. vLLM builds the normalization map from the tokenizer at startup and fails if its size differs from engram_compressed_vocab_size, so do not swap tokenizers.
• Pinned memory limits. A CUDA build of the llama.cpp branch pins host memory for CPU-resident weights and fails above physical RAM; the recipe uses the CPU build for anything over about 100 GB.
• PD disaggregation and speculation cannot be combined in SGLang.

FAQ

Can I run it on one 24 GB or 32 GB GPU?

Not usefully. Even the Q2_K file is 264.5 GB, so a single consumer GPU box needs roughly 256 GB or more of system RAM for experts and tables, and the only runtime is an unmerged branch. V4-Flash-0731 remains the DeepSeek to run on a workstation or a Mac.

Should I move a V4-Flash deployment to V4.1?

If you serve long prompts to agents on Blackwell or Hopper nodes, the prefill economics and the KV size are the reasons to test it: 8B active in prefill and a quarter of the KV. Budget host RAM for the tables and use vLLM 0.30 or the SGLang preview image.

Why is it called Flash if it is 510 GB?

DeepSeek uses Flash for the serving-cost tier, not the download size. Per-token compute and KV are what got smaller.

Should I wait?

For a tagged llama.cpp, Ollama, or MLX build, yes. For a GPU cluster, vLLM 0.30 serves it today.