How to Run Nemotron 3.5 Bolt Locally
NVIDIA released Nemotron 3.5 Bolt 74B-A9B 24GB on October 15, 2026, under the OpenMDW-1.1 license. It is a vision-language model with 74 billion parameters, 9 billion active per token, and a 256K context. The number in its name is a memory budget. NVIDIA started from a 120B-A12B parent, cut experts layer by layer under a 24 GB limit, then compressed every expert to 2.0625 bits. The text model is a 22.02 GB GGUF. The vision projector adds 1.73 GB and the speculative drafter adds 1.00 GB. The text model fits a 24 GB card, but the projector and the drafter do not both fit beside it. The sections below follow the compression that set those sizes, then the drafter, then the llama.cpp commands for each card.
Load the 22.02 GB text GGUF onto a 24 GB card, then the 1.00 GB LiLiCorr drafter and the 1.73 GB vision projector, as llama-server does at startup. Each of the 22 cells stands for four of the 88 blocks. Only the 8 cells that hold an attention block write to the cache, at 8,192 bytes per token. The bench at the front measures every loaded byte against 24 GB. The projector is the file that does not fit, so it goes back to host RAM, as in the 24 GB command below. The draft result, 4 tokens kept and 1 added, is an example near NVIDIA's reported average.
A hybrid of 88 blocks. Mamba-2 and Latent MoE blocks alternate, and one attention block replaces a MoE block eight times. Only those 8 attention blocks keep a per-token cache.
Attention at blocks 7, 16, 25, 36, 47, 58, 69, and 78. Each MoE block picks 9 of 240 to 496 routed experts plus one shared expert.
hidden 4,096 · 32 query heads · 2 KV heads · head dim 128 · latent 1,024 · vocabulary 131,072 · context 262,144
KV cache: 8 layers × 2 KV heads × 128 × 2 × 2 bytes = 8,192 bytes per token in f16
Source: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB config.json
What NVIDIA released
On October 15, 2026, NVIDIA published two models in one announcement. Nemotron 3.5 Super VL is a 120B-A12B vision-language model for servers. Nemotron 3.5 Bolt 74B-A9B 24GB is the same model compressed for one GPU. A third model, Bolt E4B 4GB, followed on October 23. Its card says NVIDIA post-trained it from Gemma-4-E4B-it and pairs it with its own DFlash drafter. So the Star Elastic and 2-bit steps in this guide apply only to the 74B model. The research page states the design target in one line. Fit in 24 GB of memory, decode at 80 tokens per second or more, and keep as much accuracy as possible. The page also links a technical report with the full method.
Bolt comes as five Hugging Face repositories. The main repository holds the quantized model as 90 safetensors shards, 24.08 GB in total, for vLLM. The GGUF repository holds the 22.02 GB text model and a 1.73 GB vision projector for llama.cpp. The LiLiCorr GGUF repository holds the 1.00 GB speculative drafter. The LiLiCorr repository holds the same drafter as safetensors for vLLM. A BF16 checkpoint of 149.77 GB exists for further training. The repositories were created between October 7 and October 12 and opened on October 15. All file sizes in this guide come from the Hugging Face file listings, in decimal gigabytes.
The license is OpenMDW-1.1 on every repository, and the card says the model is ready for commercial use. The model dates run from December 2025 to May 2026, with a post-training data cutoff of May 2026. Inputs are text, images, and video clips of up to two minutes. The output is text. The research page says the quantized model is tuned for llama.cpp on RTX Spark and for vLLM on DGX Spark. The vLLM card also lists A100, H100, and Blackwell GPUs.
The architecture
The config.json describes 88 blocks of three kinds. Forty are Mamba-2 blocks, forty are Latent MoE blocks, and eight are attention blocks. Mamba-2 and MoE blocks alternate. Every ten or eleven blocks, an attention block takes the place of a MoE block. The eight attention blocks sit at positions 7, 16, 25, 36, 47, 58, 69, and 78. The hidden size is 4,096 and the vocabulary has 131,072 tokens.
The three block kinds cost memory in three different ways. An attention block has 32 query heads and 2 KV heads of 128 values. It stores a key and a value for every token, so 8 blocks cost 8,192 bytes per token in f16. A Mamba-2 block has 128 heads of 64 values with a state size of 128. It keeps one fixed state of 1,048,576 values per sequence, 4 MB in f32, and the state does not grow with context. Forty of them hold 0.17 GB. A Latent MoE block first projects the 4,096-wide hidden vector down to a 1,024-wide latent. The routed experts live in that latent space, with an up matrix of 1,024 by the expert width and a down matrix back. One shared expert runs at the full 4,096 width with an inner size of 5,376. A sigmoid router picks 9 routed experts per token. The expert counts and widths differ from block to block, which is the subject of the next section.
The vision side is the C-RADIO v4-H encoder from the parent model. It has 32 blocks of width 1,280, reads 16-pixel patches from 512-pixel tiles, and feeds an MLP projector into the 4,096-wide text stream. The GGUF projector file holds the encoder and the projector together, 842 million parameters, almost all in BF16, which is why it weighs 1.73 GB.
Compressed backwards from 24 GB
The parent, Nemotron 3.5 Super VL 120B-A12B, has the same 88-block skeleton. Its config.json gives every MoE block 512 routed experts of width 2,688, and picks 22 of them per token. That is 20,480 experts and 112.7 billion routed parameters. At 2.0625 bits those experts alone would take 29.1 GB, over the budget before any other tensor is counted. So the compression had to remove experts, not only shrink them.
NVIDIA used Star Elastic, which the model card calls a router-guided architecture search. A budget-conditioned router chooses, for every MoE block, how many routed experts to keep and how wide they are. The top-K is fixed at 9 instead of 22. The search targets a footprint below 24 GB at batch size 1 and a 128K context. The budget already counts the precision each tensor will get later, such as 2-bit expert weights and 16-bit SSM states. The objective is the lowest knowledge-distillation loss that satisfies the budget, with a model-size penalty. The router converges in the first 8 billion tokens at an 8K context. Recovery then continues to 72 billion tokens at 8K and 48 billion more at 128K, 120 billion tokens in all.
The result is visible in the child's config.json. The 40 MoE blocks keep between 240 and 496 routed experts each, at widths of 1,536, 2,048, 2,304, or 2,560. The early blocks keep the fewest experts. Block 1 keeps 240 of width 2,048 and block 3 keeps 240 of width 1,536. The late blocks keep the most. Blocks 48, 79, and 85 keep 496 of width 2,560. In total 13,824 experts survive out of 20,480, and the routed parameters fall from 112.7 to 66.1 billion. Each token now runs 1.69 billion routed parameters instead of 4.84 billion.
Forty columns, one per Latent MoE block, from both config.json files. The dashed line is the parent, which keeps 512 experts of width 2,688 in every block. Each column is what the budget router kept in the child. Darker columns are wider experts. Move the slider to read one block.
Expert counts and widths come from block_configs in each config.json. Bytes per block = experts × 2 matrices × 1,024 latent × width × 2.0625 bits ÷ 8. The parent is priced at the same 2 bits to show that fewer experts were required, not only smaller ones.
After the search, NVIDIA attached the parent's vision encoder back onto the compressed text model. It retrained only the projector, on about 6.6 million image samples per epoch. A vision-heavy fine-tuning stage followed, then checkpoint merging, agentic reinforcement learning, and RLHF with the Nemotron 3 Ultra reward model. The GGUF card lists every stage. The point for a local user is the order. The shape was fixed first, under the byte budget, and every later stage worked inside that shape.
Two bits for the experts
The budget router assumed 2-bit experts. The quantization stage made that true. NVIDIA used quantization-aware distillation, QAD, in Model Optimizer. The student trains with the quantizer in the loop and learns from the full-precision teacher, so the 2-bit weights are fitted, not rounded. The card cites ParetoQ and Quest for the rule that a layer twice the size at 2 bits beats the original size at 4 bits. NVIDIA reports the same result for this model.
The GGUF header shows exactly which tensor got which format. I read it with a short parser. The file has 843 tensors and 74,042,720,848 parameters. Every routed and shared expert matrix is IQ2_XXS, which stores 256 weights in 66 bytes, 2.0625 bits per weight. The Mamba-2 input and output projections are NVFP4, a new ggml type that stores 64 weights in 36 bytes, 4.5 bits per weight. Attention, the latent projections, the token embeddings, and the output head are Q8_0 at 8.5 bits. The routers, the norms, and the small Mamba-2 constants stay in F32. In bytes the file splits like this:
- Routed experts, IQ2_XXS: 66.14 billion parameters, 17.05 GB.
- Shared experts, IQ2_XXS: 1.76 billion parameters, 0.45 GB.
- Mamba-2 projections, NVFP4: 4.38 billion parameters, 2.47 GB.
- Token embeddings and output head, Q8_0: 1.07 billion parameters, 1.14 GB.
- Latent projections and attention, Q8_0: 0.62 billion parameters, 0.66 GB.
- Routers, norms, and constants, F32: 59 million parameters, 0.24 GB.
- Header and alignment padding: 8 MB.
The experts are 92 percent of the parameters and 80 percent of the bytes. That ratio explains the 2-bit choice. Set the experts to 4.5 bits in the ledger below and the file grows to 42.7 GB. Set everything to BF16 and the ledger reads 148.1 GB, close to the 149.77 GB checkpoint. The small gap confirms the parameter counts.
Eight tensor classes with their real parameter counts from the GGUF header. Give each class a format. The bars are bytes on one ruler, and the red tick is 24 GB.
Bits per weight: IQ2_XXS 2.0625, NVFP4 4.5, Q8_0 8.5, BF16 16, F32 32, from the ggml block sizes. The as-shipped total is 22.01 GB plus an 8 MB header. The real BF16 checkpoint is 149.77 GB.
Every byte on one card
Now add the three files. The text GGUF is 22.02 GB, the projector is 1.73 GB, and the drafter is 1.00 GB. Together they are 24.75 GB. The 24 GB in the name is the text model's budget. The text model alone fits a 24 GB card with 1.98 GB to spare. The projector and the drafter do not both fit beside it. NVIDIA's own table says the same in other words. It gives a 28 GB recipe for 32 GB cards and the 64 GB RTX Spark. A 32 GB recipe covers 48 GB cards and the 128 GB RTX Spark. There is no 24 GB recipe.
The runtime memory on top of the files is small, because the architecture keeps almost nothing per token. The cache costs 8,192 bytes per token in f16 and 4,352 bytes in q8_0. At 65,536 tokens that is 0.54 GB or 0.29 GB. At the full 262,144 tokens it is 2.15 GB or 1.14 GB. The Mamba-2 state is 0.17 GB per sequence at any context. The server allocates the whole cache at startup for the context you set. So plan with the number at your --ctx-size. Compute buffers add about 1 GB in the planner below, which is an assumption and not a measurement. I have not run this model on a 24 GB card. The arithmetic comes from the file sizes and the config.
On a 24 GB card, three options follow from the arithmetic. Keep the projector on the CPU with --no-mmproj-offload, which costs image encoding speed and nothing else. Keep the drafter on the CPU with --spec-draft-ngl 0, which slows every draft block. Or move the routed experts of the first few blocks to system RAM with --n-cpu-moe. Block 1 holds 0.26 GB of experts and block 3 holds 0.19 GB, so a small number buys the room for the drafter. The planner prices each choice and writes the command.
Place the text GGUF, the projector, and the drafter on the GPU or the host, pick a card, a cache type, and a context, and read the llama-server command that results. The GPU map must stay under the card's usable memory.
GPU
Host RAM
File sizes from Hugging Face. Cache = 8 attention blocks × 2 KV heads × 128 × 2 × bytes per value × tokens, with q8_0 at 34/64 of f16. Mamba-2 state is 40 blocks × 4 MB in f32. Routed expert bytes per block come from block_configs at 2.0625 bits. Buffers are an assumed 1 GB. The reserve is yours to set.
What the drafter does
Each decoded token reads about 4.8 GB of weights. That is the 9 routed experts, the shared expert, the Mamba-2 projections, the attention, the latent projections, the F32 routers, and the output head. On a DGX Spark, whose specification gives 273 GB/s of memory bandwidth, that read alone caps plain decoding near 57 tokens per second. NVIDIA's target was 80 or more. Speculative decoding closes the gap. A small drafter proposes several tokens, and the big model checks them all in one pass. Every position in that pass shares about 4.4 GB of the read. The routed experts are the exception, because each position picks its own 9. At 8 positions a block reads at most 72 experts, so one pass reads between 4.82 and 7.88 GB and can return several tokens.
The drafter is LiLiCorr, from an NVIDIA paper of August 2026. Its GGUF header describes a 5-block dense transformer of 1.11 billion parameters. It reads the outputs of target blocks 1, 18, 35, 51, 68, and 85, counted from 0 as in the drafter config. Inside a block of 8 positions it attends in both directions, with a 1,024-token sliding window and attention sinks. One pass of the drafter fills all 7 draft slots at once. That is the DFlash idea, and it is why a 7-token draft costs little more than a 1-token draft. The weakness of any such drafter is that each slot is trained on its own. The best token at slot 2 and the best token at slot 3 are each likely on their own. Together they can form a sequence that the target rejects at the first mismatch.
LiLiCorr repairs that with a second, smaller network. It keeps the top 8 candidates at each of the 7 slots, a lattice of 56 tokens. A two-layer transformer of 37 million parameters reads the whole lattice in one pass. For every candidate it produces an out vector and an in vector of 1,024 values. Two neighbors fit when the earlier candidate's out vector points the same way as the later candidate's in vector, measured by cosine similarity. With 7 slots and 8 candidates, the first slot needs 8 scores and each of the next 6 needs 64. That is 392 cosine scores, computed as one batch of small matrix products. The only sequential step is a walk from left to right, one argmax per slot over scores that already exist. The DFlash 2 essay animates the same walk for its drafter.
The target then verifies the 7 tokens in one pass, keeps the longest correct prefix, and adds one token of its own. NVIDIA's drafter card reports an acceptance length of 4.30 tokens per pass on SPEED-Bench at draft length 7. It ranges from 5.26 on coding down to 3.41 on writing. In vLLM at concurrency 64, NVIDIA reports 4.68 with greedy decoding and 4.25 with sampling for the NVFP4 drafter. On an RTX Spark laptop with 128 GB and llama.cpp, the research page reports 83.99 tokens per second on the AIPerf SPEED-Bench coding set. That page does not give the RTX Spark memory bandwidth, so the DGX Spark ceiling of 57 is only a reference point. The arithmetic is the same on both machines. A verify pass reads at most about 1.63 times the bytes of one plain token. It returns about 4 tokens, so decode gets faster even when memory bandwidth is the limit.
The same table also shows the cost of that choice. On the same laptop, Qwen3.6 35B-A3B reaches 96.63 tokens per second and Gemma 4 26B-A4B reaches 100.56. Both run as UD-Q4_K_M GGUFs with MTP drafting. Bolt is slower than both, and NVIDIA argues for it on accuracy, which the benchmark section covers.
Run it in llama.cpp
Support arrived in two pull requests on October 15, 2026. Pull request 30520 added the NemotronH_Omni_Reasoning_V3 architecture, the vision graph for it, and the converter for both Super VL and Bolt. Pull request 30506 added LiLiCorr drafting on top of the existing DFlash path. Build b11669 was tagged thirteen minutes after the second merge, so use that build or newer. The drafter still runs under --spec-type draft-dflash, because LiLiCorr is a scoring step inside the DFlash loop and not a new type.
Download the three files with the Hugging Face client. The commands below are the Linux form of NVIDIA's recipe.
export MODEL_DIR="$PWD/models/nemotron-bolt"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF \
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
--local-dir "$MODEL_DIR"
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF \
NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf \
--local-dir "$MODEL_DIR"NVIDIA's 28 GB recipe runs on a 32 GB card or a 64 GB RTX Spark. It sets a 65,536-token context, a q8_0 cache, one sequence, and six draft tokens.
llama-server \
--model "$MODEL_DIR/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf" \
--mmproj "$MODEL_DIR/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf" \
--batch-size 2048 --ubatch-size 512 \
--ctx-size 65536 --parallel 1 -ngl 999 \
--spec-type draft-dflash \
--spec-draft-model "$MODEL_DIR/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf" \
--spec-draft-n-max 6 --spec-draft-ngl 999 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--temp 1.0 --top-p 0.95 --flash-attn on --jinjaThe 32 GB recipe is for a 48 GB card or a 128 GB RTX Spark. It raises the context to 262,144 and the draft to 7 tokens. It uses a 4,096 batch with a 2,048 micro-batch, and it drops the q8_0 cache flags. For a 24 GB card there is no NVIDIA recipe. The command below follows from the planner above, with its assumed 1 GB of buffers and 2 GB kept free. The projector stays on the host, and the context is 32,768 with a q8_0 cache. The drafter stays on the GPU, so --n-cpu-moe 20 moves the routed experts of the first nine MoE blocks, 2.59 GB, to system RAM. The GPU then holds 21.73 GB of its 22 GB usable. It is derived from the flags and the file sizes, not measured.
llama-server \
--model "$MODEL_DIR/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf" \
--mmproj "$MODEL_DIR/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf" \
--no-mmproj-offload \
--ctx-size 32768 --parallel 1 -ngl 999 --n-cpu-moe 20 \
--spec-type draft-dflash \
--spec-draft-model "$MODEL_DIR/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf" \
--spec-draft-n-max 7 --spec-draft-ngl 999 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--temp 1.0 --top-p 0.95 --flash-attn on --jinjaImages and video go through the same server. The pull request documents llama-mtmd-cli for one image and a /video command in the same tool. On the server, --media-path allows a directory of video files. The chat template reads enable_thinking from chat_template_kwargs, with thinking on by default. The card recommends temperature 1.0 and top_p 0.95, and sets 16,384 output tokens in its example, because reasoning traces are long. Over the OpenAI-compatible API, send:
{"model": "nemotron-3.5-bolt",
"messages": [{"role": "user", "content": "Explain why binary search takes logarithmic time."}],
"temperature": 1.0, "top_p": 0.95, "max_tokens": 16384,
"chat_template_kwargs": {"enable_thinking": true}}Ollama and vLLM
Ollama lists nemotron-3.5-bolt as a 25 GB download with a 256K window and text plus image input. The tags 74b-a9b and 74b-a9b-iq2 point at the same quantization. Since October 23 the same library entry also holds the Bolt E4B tags, so pin the 74b-a9b tag. Ollama allocates the cache for the context you set, so set it on purpose, as with every model in this series. I did not check how Ollama handles the drafter. NVIDIA's blog also names LM Studio and Unsloth builds.
ollama run nemotron-3.5-bolt:74b-a9b
OLLAMA_CONTEXT_LENGTH=65536 ollama serveFor vLLM the main repository holds the packed 24.08 GB checkpoint. The card's recipe uses a nightly vLLM image, the b12x linear and MoE backends, and FlashInfer attention. It sets an fp8 KV cache, a 262,144-token context, and one sequence. A dflash speculative config loads the LiLiCorr drafter with 7 speculative tokens. It also needs the ultra_v3_reasoning_parser.py file from the repository and the qwen3_coder tool parser. The card lists H100, H200, GB200, RTX 5090, DGX Spark, and RTX Spark as test hardware. Copy that recipe from the card rather than from here, because the nightly image tag changes.
Benchmarks, vendor-reported
These are NVIDIA's numbers from the GGUF card, measured on the IQ2 model. I found no independent evaluation by November 3, 2026. NVIDIA picked two MoE models that fit in 24 GB at 4 bits, Qwen3.6 35B-A3B and Gemma 4 26B-A4B. The card scores both of them in BF16. It also compares with its own Nemotron 3.5 Lightning 30B-A3B.
| Benchmark | Bolt 74B-A9B 24GB | Best of the three comparison models |
|---|---|---|
| GDPVal-AA-V2 | 28.47 | 25.75 |
| Terminal-Bench 2.1 | 50.98 | 44.38 |
| GPQA Diamond | 86.43 | 83.40 |
| BrowseComp | 66.75 | 48.74 |
| WebArena (vision) | 50.86 | 38.2 |
| PinchBench | 87.82 | 88.07 |
| AA-Omniscience accuracy | 16.90 | 22.17 |
| AA-Omniscience non-hallucination | 61.27 | 68.81 |
Two readings matter for a local choice. The gains are largest on agent tasks that use tools, a browser, or a terminal, which is what the reinforcement learning stages targeted. The losses sit on knowledge recall and on refusing to guess, where the 2-bit experts and the smaller expert pool cost something. The research page adds that Bolt is first among small open models on PinchBench's own leaderboard and second on WebArena and BrowseComp. Treat all of it as vendor numbers until others reproduce them.
Failure modes and fixes
- Out of memory at startup on a 24 GB card. The three files are 24.75 GB together. Add
--no-mmproj-offload, or--spec-draft-ngl 0, or--n-cpu-moe 4, and lower--ctx-size. - Unknown architecture at load. The build is older than b11669. Update llama.cpp to a build from October 15, 2026 or later.
- The drafter gives no speed-up. The drafter does not match the target. The LiLiCorr GGUF reads hidden states from Bolt blocks 1, 18, 35, 51, 68, and 85, so it works only with the Bolt 74B target. The Bolt E4B DFlash drafter does not work here.
- Out of memory at 256K context. The f16 cache needs 2.15 GB there. Add
--cache-type-k q8_0 --cache-type-v q8_0with--flash-attn on, or lower the context. - Thinking text appears inside the answer. The template did not run. Pass
--jinja, and useenable_thinkinginchat_template_kwargsto switch it off. - Images are ignored. The server started without
--mmproj. Video over the API also needs--media-path. - Decoding slows after
--n-cpu-moe. The CPU now runs the picked experts of those blocks from system RAM. Keep the number small and move the projector to the host first. - Answers stop inside the thinking block. The output budget is too small. The card's example uses 16,384 tokens.
FAQ
Does it fit on a 24 GB GPU?
The 22.02 GB text GGUF fits with 1.98 GB to spare. The 1.73 GB projector and the 1.00 GB drafter together need 2.73 GB more, so all three do not fit at once. Keep one of them on the host. NVIDIA publishes a 28 GB recipe for 32 GB cards and a 32 GB recipe for 48 GB cards.
Why are the experts at 2 bits?
The routed experts hold 66.1 of the 74.0 billion parameters. At 2.0625 bits they take 17.05 GB. At 4.5 bits they would take 37.2 GB on their own. NVIDIA used quantization-aware distillation to recover accuracy. It reports that a larger model at 2 bits beats a smaller one at 4 bits for the same bytes.
What does the LiLiCorr drafter do?
It proposes 7 tokens in one pass and keeps the top 8 candidates per slot. It scores neighboring pairs by the cosine similarity of an out vector and an in vector, then walks the best path. The target verifies the 7 tokens in one pass. NVIDIA reports an acceptance length of 4.30 tokens per pass on SPEED-Bench.
Which llama.cpp build do I need?
Build b11669 of October 15, 2026 or newer. Pass --spec-type draft-dflash for the drafter, --jinja for the template, and --mmproj for images.
How much cache does it use?
8,192 bytes per token in f16, from the 8 attention blocks. That is 0.54 GB at 65,536 tokens and 2.15 GB at 262,144. The Mamba-2 state is a fixed 0.17 GB per sequence.
Keep reading