Rohit Ghumare · November 2, 2026 · 13 min read

A 763B Model on a 12 GB Card: The Biggest Tensors Are Lookup Tables

DeepSeek-V4.1-Flash-1017 has 763.2 billion parameters, and since late October it runs on one 12 GB graphics card. Low bit widths explain only part of that. The two largest tensors in the model are lookup tables. A token reads a few small rows from them, so they can stay on the SSD. This essay follows those rows and works their SSD bytes per token against drive bandwidth. It also shows how the bit budget is spent and what stays on the GPU.

Several releases arrived inside eleven days. On October 23, IST Austria's DASLab published GSQ-RCO GGUFs of the model at 1.75, 2.00, and 2.25 bits per weight. On October 25, llama.cpp release b11870 added native DeepSeek-V4.1 support. On October 27, the GGUFs moved to that layout. On November 2, Strata 0.2.10 came out, a local engine that runs these files on one 12 GB NVIDIA GPU. NVIDIA and DeepSeek published their own serving notes in the same days, and both keep the tables out of GPU memory.

The DeepSeek-V4.1-Flash guide explains how Engram hashes n-grams into table rows. This essay does not repeat that. It follows the bytes. I did not run these builds. The byte counts come from the GGUF headers, which I read with HTTP range requests on November 3. Speeds come from the projects, and I label them as theirs.

A weight is multiplied, a table is indexed

A matrix in a transformer layer does work on every token that reaches it. In a dense layer the GPU reads the whole matrix for each token. In a mixture-of-experts layer it reads the whole matrix of each expert that the router picks. DeepSeek-V4.1-Flash-1017 picks 6 of 384 routed experts in each of its 40 layers. That is 240 expert reads per token.

An Engram table does no arithmetic of its own. Layers 1 and 14 hash the last few token ids into row numbers, and they add the rows they find to the residual stream. Each layer hashes three n-gram orders with eight hash heads, so it reads 24 rows. A token reads 48 rows across the two tables. The ISTA model card says the tables are read "one row per token". That is true for each hash head.

The parameter counts show the split. The NVIDIA NVFP4 card lists a "552B backbone in total, 8B activated during prefill and 16B during decode, plus 196B Engram conditional memory". The ISTA card counts 551.9B backbone parameters and 196.6B in the tables. In the GGUF, llama.cpp keeps both tables in one tensor, per_layer_token_embd.weight, with 768,022,850 rows of 256 values.

The table file uses IQ4_NL, a GGUF type that packs 32 values into 18 bytes. A row of 256 values is therefore 144 bytes. The file is 110.6 GB, and the bit search leaves it alone. It stays at 4.5 bits per weight in every build. The card gives the reason: the tables are "a lookup, not a matmul weight".

Bytes per token, against the drive

Forty-eight rows of 144 bytes make 6,912 bytes per token. An SSD does not return 144 bytes, though. It returns pages. The llama.cpp loader maps the GGUF into memory and marks the table range with POSIX_MADV_RANDOM. The kernel then reads only the page that holds a row, with no read-ahead. On Linux x86 a page is 4 KiB.

Most rows sit inside one page. Every row starts at a multiple of 16 bytes, because 144 is a multiple of 16 and GGUF aligns tensor data to 32 bytes. A row crosses a page edge only when it starts in the last 128 bytes of a page. That is 8 of the 256 possible start positions, or about 3.1% of rows. The expected cost is 48 × 1.031 × 4,096 bytes, about 203 KB per token. That is 29 times the bytes the model uses.

Now set that against a drive. At 10 tokens per second the tables need 2.0 MB/s and about 500 random reads per second. Strata's RTX 5070 test box has a 3.6 GB/s NVMe drive, and 2.0 MB/s is 0.06% of it. The figure below runs the same arithmetic for your own drive.

The row numbers depend only on the token ids. An engine knows them before the forward pass starts, so it can issue the reads early. Each page also lands in the page cache. A frequent n-gram can cost a disk read only the first time it appears.

The experts show the scale. At 2.00 bpw one routed expert is 8.6 MB, so the 240 picks touch 2.06 GB per token. The table reads move about 10,000 times fewer bytes. That ratio is why the 110.6 GB file sits on the SSD while the expert weights get the RAM.

The bit budget is spent per tensor

The base checkpoint stores its dense weights in FP8 and its routed experts in FP4. The 2.00 build fits the backbone in 138.24 GB. ISTA used two methods from DASLab for that. The first is GSQ, which quantizes one tensor to a scalar GGUF type. It learns the grid choice of each value and the scale of each group through a Gumbel-Softmax relaxation. ISTA ran it for every candidate type of every tensor, which gives a database of quantized variants.

The second is RCO, which picks one type per tensor. The size target is a hard constraint. RCO turns it into a smooth Riemannian manifold in logit space. Gradient descent on the task loss then moves along that manifold, so the result meets the byte target exactly. The 2.00 build measures 2.0038 bits per weight over the backbone.

GGUF stores one type per tensor, and the 384 experts of a layer share one tensor for each projection. So the search chooses a type per layer, never per expert. Dense tensors choose from Q5_K, Q6_K, and Q8_0. Routed experts choose from IQ1_S up to IQ3_S.

The repository publishes the allocation files, and they show where the bits went. In all three builds the dense path is the same 6.16 GB of Q5_K, Q8_0, BF16, and F32 tensors. Only the routed experts change, from 114.56 GB to 132.07 GB to 149.06 GB. In the 2.00 build the first seven layers keep IQ1_M on all three projections. The last layer gets IQ2_XS on all three.

The ruler is the reason for the search. A uniform build must pick one type for every expert tensor. All IQ1_M gives a 125.07 GB backbone, and all IQ2_XXS gives 146.30 GB. The 2.00 target sits between them, and only a mix of types reaches it.

ISTA reports that the 2.00 build keeps 100.0% of the base model's task average on AIME25, GPQA-Diamond, and LiveCodeBench v6, measured at reasoning effort 50. Those are the quantizer's own numbers on three benchmarks. Strata measured the same files against the original model on 4,995 positions. It found a KL divergence of 0.170 and a perplexity 15.5% higher. Equal benchmark scores do not mean equal outputs.

What stays on the GPU

Every token runs attention, the shared expert, the routers, and the output head. That dense path is 6.16 GB in every build. It holds 3.82 GB of attention and indexer weights, 0.97 GB of shared experts, and 0.91 GB of embeddings and output head. The last 0.45 GB holds the Engram projections, the routers, and the norms.

On a 12 GB card the dense path fits with room left for the KV cache, the buffers, and a few GB of experts. Strata prints a plan for its pruned Coder build on a 12 GB card with 64 GB of RAM. It keeps "about 3 GB of the experts" on the card. The rest come from RAM and the SSD. On cards with 16 GiB or less, Strata also runs 5 of the 6 picked experts by default, which it calls Faster mode.

Experts that are not on the card run on the CPU. Strata computes them "read in place from the model files", and llama.cpp does the same with --cpu-moe. Either way the expert pages live in the page cache. Experts that do not fit in RAM come from the same SSD as the tables, and those reads are large.

Worked example: one desktop

Strata's DeepSeek notes describe an RTX 5070 with 12 GB, 64 GB of RAM, and a 3.6 GB/s NVMe drive. Take the 2.00 build and assume the OS keeps 8 GB. Then about 3 GB of experts sit on the card and 56 GB sit in RAM. That is 59 of 132.07 GB, or 44.7%.

With five picks per layer, a token touches 200 experts, or 1.72 GB. If the picks spread evenly, 55.3% of those bytes miss the cache. That is 0.95 GB per token from the SSD, which caps the box at 3.8 tokens per second. The table reads add 0.2 MB to that, about 0.02%.

Strata reports 4 to 6 tokens per second for this box in its README, and 4.7 in its DeepSeek notes. These are self-reported numbers, measured on Strata 0.2.5. They sit above the even-spread ceiling. That fits routing that picks some experts more often than others, and Strata ships a hot-expert profile for that reason.

The same README lists an RTX 3060 with 12 GB and 46 GB of RAM at 9 to 11 tokens per second, also self-reported. With less RAM, the even-spread estimate is 1.19 GB per token from the SSD. At 10 tokens per second, that needs about 12 GB/s from the drive. The notes do not give that drive's speed, and they do not explain why the 3060 box beats the 5070 box. I cannot explain it either.

The datacenter keeps the same split

DeepSeek serves the model with DeepInfer. Its deployment Q&A reached v0.2 on November 3. It gives Engram its own parallel group, ENP, sharded along the vocabulary dimension. Each GPU rank holds 1/ENP of the tables in local host memory, registered for RDMA.

At the start of each forward pass, a kernel issues "one-sided RDMA reads". The network card then copies the rows into GPU memory in the background. Only the small scales stay in GPU memory. The design works for the same reason the desktop works. The row numbers are known before the pass, and the rows are small.

NVIDIA's NVFP4 checkpoint of October 30 converts the experts to NVFP4 and the tables from FP8 to MXFP4, a conversion its card calls lossy. Its vLLM command passes --engram-config with cpu_offload set to true. In all three setups the tables stay out of GPU memory.

Run it in llama.cpp

You need llama.cpp b11870 or newer, about 249 GB of free disk for the 2.00 build, and an NVMe drive. Download one build and the shared Engram file. Then link the Engram file as shard 6, as the ISTA card shows.

hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "*IQ2_XXS*" "*Engram*" --local-dir .
ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf \
  DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00006-of-00006.gguf

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
  -lm mmap -lzm on -ngl 99 -cmoe -c 8192 \
  --reasoning on --reasoning-effort 50

The -lm mmap flag maps the files into memory. The -lzm on flag reads the table rows on demand. Its default, auto, does the same for any tensor over 4 GiB. The -ngl 99 flag puts every layer on the GPU, and -cmoe keeps the routed experts in host memory. The card then holds the 6.16 GB dense path and the KV cache. If you leave out -ngl and -cmoe, the default --fit pass picks a split for the free device memory.

At start, look for the loader line that names per_layer_token_embd.weight and ends in lazy read enabled. That line confirms that the tables stay on disk. I have not measured llama.cpp's speed on a 12 GB card. Its expert placement is simpler than Strata's, so do not use Strata's numbers as a llama.cpp target.

Size your own SSD

Two numbers decide whether a drive keeps up. Random 4 KiB reads per second serve the tables. Sequential bandwidth serves the expert misses. Measure the first on the Engram file itself, read only. Then watch the drive while the model writes.

fio --name=engram --filename=DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf --readonly \
  --rw=randread --bs=4k --direct=1 --ioengine=libaio --iodepth=1 \
  --runtime=20 --time_based

iostat -x 1

Run iostat in a second terminal while llama-server answers a prompt. Divide its r/s column by the tokens per second that the server reports. Near 48 reads per token, the reads are mostly table pages. Far above 48, expert pages also come from the drive, and more RAM is the fix.

The sizing rule from Fig. 1 is short. Table reads per second equal 48 × 1.031 × tokens per second × (1 − page-cache hit share). Your fio result should be many times that number. On Apple silicon the page is 16 KiB, so each row moves at least 16 KiB. The bytes per token then grow about fourfold. macOS has no libaio, so run fio there with --ioengine=posixaio and --bs=16k.

When it goes wrong

• Old files do not load. Downloads from before October 27 have four shards per build and an older layout. Run convert_ds41_upstream.py from the repository on the folder. It rewrites the files and fetches only 80 KB.
• llama.cpp cannot find shard 6. Make the link again with ln -s, or with mklink /H on Windows.
• The tables load into RAM. The lazy read line is missing from the log. The build is older than b11870, -lzm off is set, or the platform has no mmap. Without mmap, llama.cpp warns that it loads the tensor in full. Update llama.cpp and remove -lzm off.
• Every token waits on the disk. The table file sits on a hard disk or a USB drive. Move it to an SSD, as the ISTA card says. Strata advises an NVMe drive.
• Tool calls break. Strata's 0.2.10 notes report one bad tool call in five runs with about 13% of the model in memory. Add RAM, or try the pruned Coder build, which keeps 192 of the 384 experts per layer.

Place it in this order

1. Put the Engram file on an NVMe drive and map it. It needs about 203 KB of reads per token and no reserved RAM.
2. Put the 6.16 GB dense path and the KV cache on the GPU.
3. Give every spare GB of the card and of RAM to the routed experts. They touch about 2 GB per token.
4. Choose the build by how much RAM you have. Each step from 1.75 to 2.25 bpw adds about 17 GB of experts, and the card holds the same dense path in all three.

At 10 tokens per second, the 196.6B table parameters use less than 0.1% of a 3.6 GB/s drive. The rest of the drive bandwidth goes to the expert misses, and RAM decides how many of those there are.