Why does the output head matter so much in Flint?

Each token runs about 2.2B body parameters, and the lm_head is 307 million more, read in full for every token. At one uniform precision the head is 12 percent of the weight bytes. With a 4-bit body and a 16-bit head, the common recipe, the head is 33 percent. That is why Recursive quantized the head to 4 bits.

Can I turn off thinking in Flint?

No. The model decides per turn whether to reason inside think tags, and there is no switch or budget. The serving kit says enable_thinking in chat_template_kwargs has no effect. Give hard problems at least 32,768 output tokens, because reasoning counts toward max_tokens.

Local model guide · October 23, 2026 · 17 min read

How to Run Flint Locally

Q: How much memory does Flint need?

The 4-bit builds are 22 to 26 GB of weights. Unsloth's UD-Q4_K_XL GGUF is 25.1 GB, Recursive's MLX int4 build is 22.6 GB, and the mlx-community 3-bit build is 17.8 GB. Add the cache: 28,672 bytes per token on the 14 full-attention layers, plus a fixed 36.8 MB for the 35 window layers. A 32 GB machine runs the 4-bit build at short context. A 24 GB machine needs a 3-bit build.

Q: Which llama.cpp build runs Flint?

The Flint architecture merged into llama.cpp on October 21, 2026 in pull request 30928. Release b11803, from the same day, is the first that carries it. Always pass --jinja so the chat template and its tool format apply.

Q: Is Flint free for commercial use?

The model card states that Flint is released under the Apache 2.0 license, and that hosted API terms and data practices are separate from the weight license. Recursive's quantized repositories and the Unsloth, ggml-org, and community MLX builds list the same license.

Recursive released Flint on October 21, 2026: a 39.8B mixture-of-experts coding model that activates about 2.5B parameters per token, under Apache 2.0. The 4-bit builds are 22 to 26 GB, so the model fits a 32 GB machine with room for context. The unusual part is where each decode step spends its bytes. The active body is 2.2B parameters, and the output head alone is 0.3B. Keep that head in 16 bits next to a 4-bit body and it is a third of the weight bytes read per token. The sections below follow one decode step from config.json to the vLLM kernels, then give the commands for llama.cpp, MLX, vLLM, and SGLang.

Machine · Flint, one decode step and its output head

Load the BF16 checkpoint, prefill a 4,096-token prompt, decode one token, and grow the context. Then quantize the body and the head in turn. The 26 cells follow config.json: two dense full-attention layers, then eleven groups of three window layers and one full layer, and a last group of two and one. Only the 14 full layers feed the KV pool, and every token also crosses the output head before it returns to the tray. The byte counts in the verdict are computed from the tensor shapes.

Architecture map · Flintconfig.json

Two dense layers, then 47 MoE layers. Fourteen layers attend to the whole context with no positional encoding. The other 35 see a 513-token window with RoPE.

49 layers, first to last, one cell each

14 full attention layers, no RoPE35 sliding window layers (513 tokens)

256 routed experts per MoE layer, 6 picked per token, plus one shared expert

Lit squares are the 6 picks for one example token. The dark square is the shared expert that runs for every token.

hidden 2,048 · 32 query heads · 4 KV heads · head dim 128 · expert size 512 · shared expert 1,024 · context 262,144

KV cache: 14 full layers × 4 KV heads × 128 × 2 × 2 bytes = 28,672 bytes per token in BF16, plus 36.8 MB for the 35 windows

Source: RecursiveSI/flint config.json

39.8B total · 2.5B active context 262,144 license Apache-2.0 llama.cpp b11803 or newer

What Recursive released

Recursive published Flint on Hugging Face on October 21, 2026. The repository holds the BF16 checkpoint in 15 safetensors shards, 79.6 GB in total, with the tokenizer, the chat template, and an SGLang serving kit. The model card gives the license in one sentence: "Flint is released under the Apache 2.0 license." It adds that hosted API terms and data practices are separate from the weight license.

The card describes a decoder-only mixture-of-experts transformer for coding agents. It takes text in and gives text out, reasons before it answers, and calls tools across many turns. Recursive reports 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified. Those are the developer's own numbers. I found no independent run as of November 3, 2026.

The announcement says a small team built the model from scratch in under four months. Its automated research system chose much of the design. The search started from a baseline with 35 billion total and 3 billion active parameters. It then tried expert counts and widths and kept the configuration it projected to give the best mix of speed and coding score. The result activates 2.5 billion parameters per token. Transformers modeling code is still not included. The runtimes that work on day one are llama.cpp, vLLM, SGLang, and MLX, and the sections below cover each one.

The architecture, from config.json

The values below come from the repository's config.json and from the tensor shapes in the safetensors index. The parameter total comes from the Hugging Face safetensors metadata.

Field	Flint
Architecture	`FlintForCausalLM`, model type `flint`
Parameters (BF16)	39,785,287,456
Layers	49: layers 0 and 1 dense, then 47 MoE layers
Attention layers	14 full attention, 35 sliding window of 513 tokens
Positional encoding	RoPE with theta 10,000 on the window layers, none on the 14 full layers
Hidden size	2,048
Query heads / KV heads	32 / 4, head dimension 128, per-head QK RMSNorm
Dense MLP size	6,144 (layers 0 and 1 only)
Routed experts	256 per MoE layer, 6 per token, size 512, sigmoid router with normalized top-k
Shared expert	1 per MoE layer, size 1,024, runs for every token
Attention sinks	`softmax_type` learnable, one `softmax_offset` value per query head
max_position_embeddings	262,144
Vocabulary	150,144, input and output embeddings not tied
BF16 checkpoint	79.6 GB in 15 shards

Read the layer list and a pattern appears. Layers 0 and 1 are full attention with a dense MLP. From layer 2 on, three window layers are followed by one full layer, eleven times. Layers 46 and 47 are window layers, and layer 48 is full. The 14 full layers are the ones in no_rope_layer_indices, so position enters only through the window layers. A full layer with no positional signal has no rotation angle that runs out of range as the context grows. The announcement says Flint supports up to 512K tokens and recommends 256K or less. The config and every serving kit use 262,144.

The MoE block is small per expert. Each routed expert is three matrices of 2,048 by 512, so 3.1 million parameters. A layer holds 256 of them, 805 million parameters, and 47 layers hold 37.8 billion. That is 95% of the model. The router is a sigmoid score over 256 experts with the top 6 kept and their weights normalized. A shared expert of size 1,024 runs beside them for every token. The two dense layers use a 6,144-wide MLP instead.

One decode step in bytes

At batch size one, a decode step reads every weight it uses once. So the question is how many parameters one token touches. The tensor shapes give the answer. Attention is the same in all 49 layers. The query projection is 2,048 by 4,096. The key and value projections are 2,048 by 512, and the output projection is 4,096 by 2,048. That is 18.9 million per layer and 924.8 million in total. The two dense MLPs add 75.5 million. Six routed experts per layer for 47 layers add 887.1 million. The shared experts add 295.7 million and the routers 24.6 million. The norms and sinks are 0.2 million. The active body is 2,207,993,632 parameters, which the card rounds to 2.2B.

Then comes the output head. The tensor lm_head.weight is 150,144 by 2,048, which is 307,494,912 parameters. Every token reads all of it to score the vocabulary. Add it and the active count is 2,515,488,544, the 2.5B on the card. The input embedding has the same shape, but a token reads one row of it, so it does not count. At one uniform precision the head is 12.2% of the bytes a step reads. Now apply the usual recipe for a local build: experts and projections at 4 bits, the head kept at 16 bits for accuracy. The body falls to 1.24 GB and the head stays at 0.61 GB. The head is now 33% of the weight bytes per token.

Recursive's announcement says its research system went against that practice and quantized the head to 4 bits. The reason it gives is the one above. With 2.5 billion active parameters, the output layer is a large share of the weights read per token. To recover the lost accuracy, it added an offline distillation stage with the 16-bit model as teacher, trained on the quantized model's own pre-sampled responses. Recursive reports that the NVFP4 model lands within 0.7 points of the 16-bit model across its benchmarks. With the head at 4 bits the step reads 1.41 GB of weights, and the head's share is back to 12%.

The cache follows the layer list. The 14 full layers store keys and values for every token. Multiply 14 layers, 4 KV heads, 128 values, two tensors, and two bytes, and you get 28,672 bytes per token. At 262,144 tokens that is 7.5 GB. The 35 window layers store only the last 513 tokens each, 36.8 MB in total, at any context. A model with all 49 layers full would need 26.3 GB at the same context, 3.5 times more. The cache also costs read bandwidth. At 100,000 tokens a decode step reads 2.9 GB of cache from the full layers, twice the quantized weights. The window layers keep that number from being 10 GB.

Fig. 1 · the head-share board

Every tensor group one decode step reads, with its parameter count from the safetensors index. Set the body and the head to different widths, grow the context, and watch the head's share of the bytes move. The chips divide a published memory bandwidth by the bytes per token to give a decode ceiling.

build

body

head

cache

context = 4,096 tokens

Parameter counts are the tensor shapes from model.safetensors.index.json multiplied out. Bits per weight: 16 for BF16, 8.5 for Q8_0, 4.5 for group-64 int4 or NVFP4 with their scales. Cache = 28,672 bytes per token on the 14 full layers plus 36.8 MB of windows, with q8_0 at 34/64 of BF16. Bandwidths: M4 Pro 273 GB/s and M5 Pro 307 GB/s and M5 Max with the 40-core GPU 614 GB/s from Apple, DGX Spark 273 GB/s from NVIDIA. Real decode lands below the ceiling.

Read the board with the released build selected. On an M4 Pro at 273 GB/s, 1.41 GB of weights plus a 4,096-token cache sets a ceiling of 174 tokens per second. Recursive claims more than 125 tokens per second on an M4 Pro MacBook Pro with its own engine. That is 72% of the ceiling. Switch the head back to 16 bits and the ceiling falls to 136, a fifth lower. The experts do not change in that comparison.

Sinks and the 513-token window

A window layer sees the current token and the 512 before it, so its cache never grows past 513 entries. The full layers carry the long-range work and have no positional encoding at all. The card lists one more part of the attention: learned per-head attention sinks. In config.json that is softmax_type: "learnable", and in the checkpoint it is a tensor named softmax_offset with 32 values per layer, one per query head.

A sink is one extra slot in the softmax. The slot has a fixed logit, the learned offset, and no value vector. When no token in the window matches the query well, the slot takes the attention mass and the head adds nothing to the residual stream. Without it, a head must spread its mass over tokens it does not want. That is why older models put unwanted attention on the first token. I read all 1,568 offsets from the last shard of the checkpoint. They run from -9.0 to 12.4 with a mean of 3.9, and they differ by layer and by head. Layer 0 holds the widest spread. The figure below shows every value and lets you test one head against a window.

Fig. 2 · the learned-sink board

All 1,568 learned sink offsets, read from the checkpoint: 49 layers across, 32 query heads down. Pick a head, then set how many tokens sit in its window and how well the best of them matches. The arithmetic gives the share of attention the sink keeps.

offset -9.0offset 12.4below zero

layer 0head 0 at the top, head 31 at the bottomlayer 48

layer 24

head 3

tokens in the window 64

logit of every token 2.0

sink

tokens

Offsets are the BF16 values of model.layers.N.self_attn.softmax_offset in model-00015-of-00015.safetensors, read on October 23, 2026. The window test is a toy: n tokens tie at one logit s, and the sink keeps 1 / (1 + n e^(s - o)). Real heads see different logits per token, so the sink's share in practice sits between the two extremes this board can show.

What vLLM changed

The vLLM day-0 post, written with Recursive, reports a result that the byte count alone does not predict. The first implementation used vLLM's stock layers and was slow at small batch sizes. A single decode step ran 1,096 kernels, many of them tuned for much larger batches. On one GB200 that step is far from any bandwidth limit, so the time went to launching and finishing small kernels, not to reading memory. The team then replaced the stock layers with Flint-specific kernels one change at a time and checked each against Recursive's reference implementation.

The post lists six changes with their effect on the NVFP4 forward: fusing the Q/K norms, RoPE, and the KV-cache write into one kernel, 8%. Small-batch MoE kernels that read only the six selected experts, 27%, also the biggest prefill gain. A small-batch dense GEMV on the same packed NVFP4 weights, 25%. Norm and SiLU kernels that follow the reference's rounding, 10%. A decode attention kernel with sinks and windows, written in CuTe DSL, 5%. And a low-latency decode path, 14%. On that path, for up to eight requests, each MoE layer runs as seven kernels. The Q/K norm, RoPE, routing, and the expert combine are folded into the neighboring GEMV kernels. Each kernel is launched with programmatic dependent launch, so it loads its weights while the previous one still runs.

The post reports the results. NVFP4 decode went from 4.81 to 1.78 ms, and BF16 decode from 3.53 to 2.34 ms. An 8K-token prefill went from 207 to 145 ms. A forward now runs 346 kernels instead of 1,096. On the low-latency path the 47 MoE layers run seven kernels each, which is 329 of the 346. Served end to end, batch-1 decode at 100K context takes 2.06 ms per token, about 485 tokens per second. The post says that figure is the forward plus the LM head and sampling. So on this hardware the head and sampling take about 0.28 ms of a 2.06 ms step, or 14%. The byte count predicts 12% for the head alone.

Fig. 3 · the kernel-launch strip

One batch-1 decode step on a GB200, as vLLM published it. Each change multiplies the NVFP4 forward by its reported factor. With all six on, the 1,096 launches fold to 346: 47 MoE layers of seven kernels, which is 329, plus 17 other launches that the post does not break down. One MoE layer shows its seven kernels launched while the previous one still runs.

NVFP4 decode forward published end state 1.78 ms4.81 ms

BF16 decode forward only the two ends are published3.53 ms

Factors, kernel counts, and the end-to-end 2.06 ms are from the vLLM post of October 21, 2026. The per-change kernel counts are not published, so the strip folds only when all six changes are on. The stock count of 22 kernels per MoE layer is the post's own figure. The post measured the changes in one order, so a partial set here multiplies the factors and is an estimate. The seven-kernel path applies to up to eight requests on one Blackwell GPU.

Two more facts from the post matter for a local run. The NVFP4 checkpoint is a GGUF whose blocks are NVIDIA's NVFP4 format in a different byte order. The engine repacks it losslessly at load time and runs it weight-only, with BF16 activations and a BF16 cache, as the reference does. And the Flint kernels target a single Blackwell GPU with tensor parallel size one. Any other configuration falls back to vLLM's standard layers, which means the 1,096-kernel path. NVIDIA's GTC Berlin post makes a related claim for RTX Spark: Recursive's specialized kernels raised generation speed by 55% without changing the weights. That number is NVIDIA's, and I could not test it.

Every build and its size

File sizes come from the Hugging Face tree API on October 23, 2026, in decimal gigabytes. Unsloth published its dynamic GGUFs on October 20, a day before the model card went public. LM Studio followed the same day, and the ggml-org conversion and the community MLX build came on October 22. Recursive's own quantized files appeared with the card.

Build	Size	Repository
`UD-IQ2_XXS`	11.9 GB	unsloth/flint-GGUF
`UD-Q3_K_XL`	18.5 GB	unsloth/flint-GGUF
`UD-Q4_K_XL`	25.1 GB	unsloth/flint-GGUF
`UD-Q6_K_XL`	36.3 GB	unsloth/flint-GGUF
`Q8_0`	42.3 GB	unsloth/flint-GGUF
`Q4_K_M`	23.4 GB	ggml-org/Flint-GGUF, non-expert tensors at Q8_0
`Q4_K_M` / `Q6_K` / `Q8_0`	24.2 / 32.7 / 42.4 GB	lmstudio-community/flint-GGUF
`flint-int4-q4_1.gguf`	26.0 GB	RecursiveSI/flint-int4-gguf, for Metal
`flint-nvfp4.gguf`	22.9 GB	RecursiveSI/flint-nvfp4-gguf, for CUDA and vLLM
MLX int4, group 64	22.6 GB	RecursiveSI/flint-int4
MLX mixed 3-bit and 4-bit	17.8 GB	mlx-community/flint-mixed-3-4bit
`BF16`	79.6 GB	RecursiveSI/flint, unsloth, ggml-org

Three of these carry Recursive's trained 4-bit values rather than a post-hoc rounding. The MLX int4 build quantizes projections and experts to group-64 affine int4 and keeps the embeddings at int8. The Q4_1 GGUF stores the same int4 codes with FP16 scales and offsets. It keeps the floating tensors in F32, which is why it is 3.4 GB larger. The NVFP4 GGUF is the checkpoint the vLLM post benchmarks. The community MLX build starts from Recursive's int4 and cuts only the routed experts to 3 bits. The attention, embeddings, and output head keep the trained values. Unsloth's guide gives the memory need per build, counting RAM plus VRAM or unified memory. It lists 13 to 15 GB for 2-bit, 16 to 20 GB for 3-bit, and 26 GB for 4-bit. The 6-bit and 8-bit builds need 38 and 44 GB.

llama.cpp

The flint architecture merged into llama.cpp on October 21, 2026, in pull request 30928. Release b11803, from the same day, is the first that carries it. An older build stops at load with an unknown architecture error. Always pass --jinja, because the chat template defines the reasoning and tool-call format. Keep --swa-full unset so the window layers keep their compact 513-token cache.

# 4-bit on a 32 GB GPU: 25.1 GB of weights, 32K of context, f16 cache
llama-server -hf unsloth/flint-GGUF:UD-Q4_K_XL \
  --alias flint --jinja -ngl 99 -fa on -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --host 127.0.0.1 --port 8080

# 3-bit on a 24 GB machine
llama-server -hf unsloth/flint-GGUF:UD-Q3_K_XL \
  --alias flint --jinja -ngl 99 -fa on -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

The sampling flags match generation_config.json. The context flag is the lever on memory. At 32,768 tokens the full-layer cache is 0.94 GB, and at the full 262,144 it is 7.5 GB, both in f16. Recursive's own GGUF cards use the full context and recommend 48 GB of memory. Their commands are worth copying for the NVFP4 file on NVIDIA and the Q4_1 file on Metal:

# NVIDIA, tested by Recursive on B300, RTX PRO 6000 Blackwell, RTX Spark, and DGX Spark
hf download RecursiveSI/flint-nvfp4-gguf flint-nvfp4.gguf pi-models.json opencode.json \
  --local-dir flint-nvfp4-gguf
llama-server -m flint-nvfp4-gguf/flint-nvfp4.gguf \
  --alias flint -ngl all -c 262144 -np 1 \
  -fa on -ctk f16 -ctv f16 --jinja \
  --temp 1 --top-p 0.95 --top-k 20 --min-p 0 \
  --host 127.0.0.1 --port 8080

# Apple Silicon with 48 GB or more
hf download RecursiveSI/flint-int4-gguf flint-int4-q4_1.gguf pi-models.json opencode.json \
  --local-dir flint-int4-gguf
llama-server -m flint-int4-gguf/flint-int4-q4_1.gguf \
  --alias flint -ngl all -c 262144 -np 1 --cache-ram 0 \
  -b 256 -ub 128 -t 8 -fa on -ctk f16 -ctv f16 --jinja \
  --temp 1 --top-p 0.95 --top-k 20 --min-p 0 \
  --host 127.0.0.1 --port 8080

On a 32 GB Mac the 4-bit files are a tight fit. By default macOS lets the GPU use only part of unified memory, so a 25 GB file plus cache does not load. Raise the limit with sudo sysctl iogpu.wired_limit_mb=28672, or use the 3-bit build. The limit resets at the next restart.

MLX on a Mac

There are three ways to run the MLX weights, and they fit different machines. Recursive's own engine, rec, is the fastest path on an M4 Pro, M4 Max, M5 Pro, or M5 Max with 48 GB or more. It downloads the int4 build, starts a server on port 11500 with OpenAI and Anthropic-compatible endpoints, and opens the pi or OpenCode harness against it. Recursive's README reports 1.4 to 2.6 times the throughput of llama.cpp on an M4 Pro, which is its own measurement.

# Recursive's engine, 48 GB or more
curl -fsSL https://raw.githubusercontent.com/recursive-oss/rec/main/install.sh | bash
rec launch pi          # or: rec launch opencode
rec serve              # server only: http://127.0.0.1:11500/v1, key from rec auth token

# mlx-lm from the main branch: Flint merged on October 24, after release 0.32.2
pip install "mlx-lm @ git+https://github.com/ml-explore/mlx-lm"
mlx_lm.server --model RecursiveSI/flint-int4 --port 8080
mlx_lm.generate --model RecursiveSI/flint-int4 \
  --temp 1.0 --top-p 0.95 --top-k 20 --max-tokens 32768 \
  --prompt "Write a Python function that merges overlapping intervals."

# 24 GB or 32 GB Mac: the 17.8 GB community build on mlx-vlm 0.7.10 or newer
pip install -U mlx-vlm
sudo sysctl iogpu.wired_limit_mb=20480     # 24 GB Macs only, until restart
mlx_vlm.server --model mlx-community/flint-mixed-3-4bit --port 8080

The mlx-lm support landed in pull request 1998 on October 24. The latest release on November 3 is still 0.32.2, from October 16, so install from the main branch until the next one. The community build's card reports 76 to 119 tokens per second on an M5 Max and a peak of about 19.5 GB in use. It also reports that the model sometimes uses wrong argument names in tool calls on mlx-vlm 0.7.10. The fix, pull request 2571, is still in review on November 3, and release 0.7.11 does not include it. Pass the sampling values on every request, because a local server without them decodes with different settings than the ones Flint was tuned on.

vLLM

Flint merged into vLLM on October 22 in pull request 62938. vLLM 0.33.0, released on November 2, is the first release that carries it. The day-0 image vllm/vllm-openai:flint also has the support built in, for x86_64 and arm64 on CUDA 13. The NVFP4 checkpoint is a GGUF, so the config and tokenizer come from the BF16 repository. Flint writes tool calls and thinking blocks in the GLM-4.7 style, which is why the parsers are named glm47.

# NVFP4, one Blackwell GPU
docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:flint \
  RecursiveSI/flint-nvfp4-gguf \
  --hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --reasoning-parser glm47 --exclude-tools-when-tool-choice-none

# BF16, 79.6 GB of weights
pip install -U "vllm>=0.33.0"
vllm serve RecursiveSI/flint \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --reasoning-parser glm47 --exclude-tools-when-tool-choice-none

The fast kernels need one Blackwell GPU, tensor parallel size one, and a BF16 cache. Any other setup runs the standard layers and loses the 2.7 times decode gain. The flag --exclude-tools-when-tool-choice-none drops the tool definitions from the prompt when a request sets tool_choice to none, which matches the reference behavior.

SGLang

The serving kit in the repository's sglang/ folder targets SGLang 0.5.16. It holds a wheel with the model code, a Dockerfile, and an install script for an existing container. It also carries a patched FlashAttention 4 kernel for the learned sinks and two example clients. Its serve scripts cover one B300, one H200, or eight replicas on an 8-GPU node. The kit's README asks for one GPU with 141 GB or more per replica, so a B300, B200, or H200. It says 80 GB GPUs are not supported, because the BF16 weights alone are 79.6 GB.

pip install -U huggingface_hub openai
hf download RecursiveSI/flint --local-dir flint && cd flint      # about 80 GB
docker build -t flint-sglang sglang
docker run -d --name flint --gpus '"device=0"' --ipc=host -p 127.0.0.1:30000:30000 \
  -v "$PWD:/model" -e MODEL=/model flint-sglang bash /model/sglang/serve_b300.sh   # H200: serve_h200.sh
until curl -sf localhost:30000/health_generate >/dev/null; do sleep 15; done
python3 sglang/example_client.py --base-url http://localhost:30000/v1

The serve scripts set the flags the model refuses to start without: --tool-call-parser flint, --reasoning-parser flint, and --grammar-backend flint. They also set --enable-fp32-lm-head and the full 262,144 context. The B300 script uses FlashAttention 4 with the TensorRT-LLM routed MoE backend. The H200 script uses FlashAttention 3 with the Triton backend. Two flags size the cache. The flag --mem-fraction-static is the share of GPU memory for weights plus cache. The flag --swa-full-tokens-ratio sizes the window pool against the full-attention pool. The README puts the cache at 28 KiB per token on the 14 full layers. The 35 window layers add 70 KiB per token for the last 513 tokens only. That is the same arithmetic as above. The first start takes about four minutes on a B300.

Sampling, reasoning, tools

Sampling. Use temperature 1.0, top-p 0.95, and top-k 20. Those are the values in generation_config.json, the values every benchmark used, and the defaults the SGLang kit applies when a request omits them. Greedy decoding is not validated.

Reasoning. The model decides on each turn whether to reason inside <think> tags. There is no switch and no budget. The kit's README states that enable_thinking in chat_template_kwargs has no effect. Reasoning comes back in reasoning_content and counts toward max_tokens. On GPQA-Diamond the kit measured a median of 4,300 reasoning tokens with a 95th percentile of 33,000. So give hard problems at least 32,768 output tokens. A reply with finish_reason set to length was cut off, possibly before any answer or tool call. Send reasoning_content back on assistant messages when you replay the history, because the template writes it into the prompt.

Tools. Send the tool list with every request. The template renders each function as JSON inside a system message. The model answers with <tool_call>name<arg_key>k</arg_key><arg_value>v</arg_value></tool_call>. A parser turns that into OpenAI-style tool_calls. Send function.arguments back unchanged and one tool message per call, in order. The format has no escaping, so a string argument cannot contain the literal </tool_call>. Strip those tags from tool output before it goes back in. Generation stops at the end-of-text, user, observation, or developer tokens, and the prompt and the output share the 262,144-token window.

A worked example

Here is one path from a bare 32 GB GPU to a coding agent, with the arithmetic that makes it fit. The 4-bit file is 25.1 GB. At 32,768 tokens the full-layer cache is 0.94 GB and the windows are 0.04 GB. With about 1 GB of compute buffers the total is near 27 GB, inside 32 GB. Start the server with the command from the llama.cpp section, then send one request with the sampling values and a large output budget:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "flint",
       "messages": [{"role": "user", "content": "Write a Python function that merges overlapping intervals and include tests."}],
       "temperature": 1.0, "top_p": 0.95, "top_k": 20, "max_tokens": 32768}'

The reply carries the reasoning in reasoning_content and the code in content. Check finish_reason before you trust the answer. Then point a harness at the same server. Recursive's GGUF repositories include a pi-models.json with a provider named flint-local at http://127.0.0.1:8080/v1, a 262,144-token window, and a 65,536-token output budget:

hf download RecursiveSI/flint-nvfp4-gguf pi-models.json --local-dir flint-cfg
mkdir -p .pi-flint && cp flint-cfg/pi-models.json .pi-flint/models.json
PI_CODING_AGENT_DIR="$PWD/.pi-flint" pi --provider flint-local --model flint

If your server uses a smaller context, lower the window in that file to match. The harness sends the whole conversation on every turn, so a long session reaches the server's limit before the file's.

Benchmarks, vendor-reported

The card prints three numbers for the BF16 checkpoint: 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified. Terminal-Bench ran under Terminus 2, SWE-bench under mini-swe-agent, and the agentic scores average pass@1 over at least five attempts. Recursive counts Terminal-Bench solutions obtained through web retrieval as failures, which other reported scores do not always do. The announcement claims the best agentic coding scores among MoE models under 50 billion total parameters. It also claims parity with dense models of 25 to 40 billion at more than five times the speed. The NVFP4 build is reported within 0.7 points of BF16. All of these are Recursive's measurements, and the comparison scores on its chart are developer-reported. The stated knowledge cutoff is June 2026.

Failure modes and fixes

llama.cpp reports an unknown model architecture. The build is older than b11803. Update llama.cpp and rebuild.
The 4-bit file fails to load on a 32 GB Mac. The default GPU memory limit is below 25 GB plus cache. Raise it with sudo sysctl iogpu.wired_limit_mb=28672, or use the 3-bit build.
The reply ends with finish_reason length and no code. The output budget ran out inside the reasoning. Set max_tokens to 32,768 or more.
Raw <tool_call> text appears in the content. No tool parser is active. Use glm47 on vLLM and flint on SGLang, and keep --jinja on llama.cpp.
Thinking does not turn off. There is no switch. The model reasons when it decides to, and the tag has no effect on the kit.
SGLang refuses to start. The kit requires --grammar-backend flint, --reasoning-parser flint, and a MoE runner of flashinfer_trtllm_routed, triton, or auto. Use the serve scripts as they are.
The vLLM decode is slower than the post. The fast path needs one Blackwell GPU with tensor parallel size one and a BF16 cache. Any other setup runs the standard layers.
A turn ends with tool-call text inside the reasoning. An open llama.cpp issue reports this with the NVFP4 file in OpenCode, and the client never runs the call. A report in the same thread did not see it with Q4_K_M. Retry the turn, or try a Q4_K_M build.
Tool calls fail with wrong argument names on mlx-vlm. The community card reports this on 0.7.10, with a fix in review. Let the agent retry, or wait for the release and add --constrain-tool-calls.
mlx-lm reports an unknown model type. Release 0.32.2 predates the merge. Install mlx-lm from the main branch.
Decode slows down late in a long session. The 14 full layers read 28,672 bytes of cache per token of context on every step. At 100K tokens that is 2.9 GB per step. Start a new session or trim the history.

FAQ

How much memory does Flint need?

The 4-bit builds are 22 to 26 GB of weights. Add 28,672 bytes per token of context for the full layers and a fixed 36.8 MB for the windows. A 32 GB machine runs the 4-bit build at short context. A 24 GB machine needs the 3-bit build.

Why does the output head matter so much?

Each token runs about 2.2B body parameters, and the head is 307 million more, read in full for every token. At one uniform precision it is 12% of the weight bytes. With a 4-bit body and a 16-bit head it is 33%. That is why Recursive quantized the head to 4 bits.

Which llama.cpp build runs Flint?

Release b11803 from October 21, 2026, or newer. Pass --jinja every time.

Can I turn off thinking?

No. The model decides per turn, and the serving kit says enable_thinking has no effect. Budget at least 32,768 output tokens for hard problems.

Is Flint free for commercial use?

The card states that Flint is released under the Apache 2.0 license, with hosted API terms separate from the weight license. Recursive's quantized repositories and the Unsloth, ggml-org, and community MLX builds list the same license.

Keep reading

gpt-ossMXFP4 experts, a 128-token window, and attention sinks on a 16 GB card. Tokens per second is memory bandwidthWhy bytes read per token set the decode ceiling.

AI Engineering from Scratch free course The curriculum behind these pages: every core AI algorithm built by hand, 503 lessons across 20 phases, from the math to the tokenizer to the agent loop. Open source, learn at your own pace. Start at aiengineeringfromscratch.com →

Rohit Ghumare

CNCF Ambassador and Google Developer Expert. I build agent infrastructure and write about the fundamentals underneath the AI stack. Config values, tensor shapes, sink offsets, and file sizes come from the Hugging Face repositories of RecursiveSI, unsloth, ggml-org, lmstudio-community, and mlx-community. I read them on October 23 and checked them again on November 3, 2026. Kernel counts and latencies are from the vLLM day-0 post. Speed claims are Recursive's and NVIDIA's own.

gpt-oss · Hardware guide · More guides · X