DSpine: The Draft Conditions Through Depth, and a 24 GB Card Pays in Context
On November 7, Tencent YouTu Lab published a speculative drafter for Qwen3.8-27B. One day later, a community build ran it in llama.cpp on an RTX 3090. By its author's measurement, mixed prompts went from 28.2 to 90.1 tokens a second, and code reached 128.7. The same build cut the largest context on the card from 176,128 tokens to 65,536. Both numbers come from one design choice. The drafter passes each position's prediction to its neighbour inside every layer, and it drafts 16 tokens at a time. This essay reads that choice, prices it, and shows how to measure it on your own prompts.
This is the third page on this site about block drafters. DFlash 2 explained how a parallel drafter stutters and how a path selector repairs the picks after the backbone. The Qwen3.8-27B guide covers the target: 48 Gated DeltaNet layers, 16 attention layers, and a recurrent state that cannot be rewound. The Ripple-65B and Nemotron 3.5 Bolt guides do the arithmetic of a drafter against memory bandwidth. None of that is repeated here. The sources are the paper, the project page, the model card with its config.json, and the Tencent/DSpine repository. The community GGUF card with its RTX 3090 numbers, the open llama.cpp pull request, and two MLX packages complete the list. I read all of them on November 9 and 10. Every speed and acceptance number below is somebody else's measurement, and I say whose.
The parts and the block of 16 come from config.json and the paper. The tower of 15 state copies, 157 MB each, and the 34 KiB per token of cache come from Qwen3.8-27B's config.json. The 59 ms pass and the 28.2 tokens a second of plain decoding are derived from the community RTX 3090 card. The knob is the per-position acceptance rate, the chance that the target accepts a draft position when it accepted the ones before it. The accepted count the lamps show is the rounded expectation at that rate, not a draw. The tank shows the cache memory free at that moment, and the line across it is the ceiling with 15 copies reserved for every pass, both from this page's model of the card. The verdict names the measured value beside it.
Where the predecessor enters
A parallel drafter predicts every position of a block in one pass, before any position has a token. The DFlash 2 essay showed what that costs. Position two does not know that position one took the verb. A block of likely tokens can still be inconsistent, and the target rejects it at the first mismatch. DFlash 2 repaired the picks after the backbone with a pairwise selector over the top sixteen candidates. Inside the backbone it mixed neighbouring hidden states with a two-tap convolution. Domino and DSpark repaired the picks with a sequential head after the backbone. In all four designs, the paper argues, the predecessor's information arrives late, or it arrives mixed with everything else in a hidden state.
Two measurements on the official Qwen3-8B DFlash drafter set up the design. First, a layer-wise readout recovers the first positions' tokens from shallow layers while later positions are still forming. Second, with the earlier history correct, replacing only the adjacent predecessor's features with its draft prediction lowers the successor's accuracy by 31.8 points. Supplying the same correct predecessor features earlier rather than later gains 14.8 points. So the predecessor matters more than the rest of the history, and it matters more when it arrives early.
DSpine acts on both. After every one of its five layers, each position receives one message, from its immediate predecessor only. The message is the predecessor's current prediction, projected into a fixed space built from the target's output embeddings. It does not carry the predecessor's whole hidden state. The paper takes the rows of the LM head, normalizes them, whitens them, and keeps a few hundred directions. In that space, a direction names a token. A gate computed from the receiver's own feature and the predecessor's feature decides how much of the message enters the residual stream. The anchor, the last verified token, enters the same space as its embedding. Attention inside the block is causal, so a position reads the target context and the positions before it, never the ones after. Every position still updates in parallel within one forward pass, and the chain of conditioning unfolds over depth. What position three received from position two at layer one shapes what it sends to position four at layer two. The injection adds about 33 million parameters to the Qwen3-8B drafter, roughly 3 percent. In SGLang it adds at most 0.15 ms per round across all five layers.
The released drafter for Qwen3.8-27B follows the same shape, per its config.json. Five layers at the target's width of 5,120, with 32 attention heads and 8 key-value heads. A sliding window of 8,192 tokens, and a block of 16. It reads the target's hidden states at layers 1, 16, 31, 46, and 61. Its config.json gives the transfer code a size of 1,024 and the projection a rank of 512. The file is 2.03 billion parameters in 4.05 GB of BF16, under Apache 2.0, and it serves this one target. The official DFlash2 drafter for the same target reads layers 5, 19, 33, 47, and 61. It keeps a 2,048-token window and drafts a block of 8.
Resolve the block from a table
After the last layer, each position proposes its top 16 candidates through the shared LM head. Then the drafter decides the block from left to right, and it does so without a sequential matrix product. For position t, the final-layer write is redone with each of the 16 candidates of position t minus 1 as the exact sender. The candidate's embedding replaces the predicted feature, the gate and the message change, and the LM head scores position t's 16 candidates again. That gives a 16 by 16 table per position: the score of candidate j at t, given candidate i at t minus 1. The paper calls it the transition cache. All 15 tables are computed in one parallel pass. Selection then walks the tables from the anchor. Pick the best successor of the anchor at position one, the best successor of that pick at position two, and so on. Each step is a lookup. Under sampling, the walk draws from the same scores and rejection sampling restores the target's distribution. The GGUF card calls this step serial last-write refinement, and config.json names it: head_topk 16, mode serial_table.
On Qwen3-8B in SGLang, this decoding step costs 0.36 ms per round, against 0.97 ms for Domino's head and 0.88 ms for DSpark's. Verification takes 72 to 76 percent of a round for every method, and a DSpine round costs 7.14 ms against 7.19 ms for DFlash2. The paper's ablation says what each part is worth. Without last-write refinement the acceptance length drops 6.6 to 7.1 percent. Without the injection as well, it drops 8.8 to 10.7 percent.
The numbers, and what the block length did
The model card measures the 27B drafter in SGLang 0.5.22 on BF16 with FlashAttention 3 at concurrency 8, with reasoning at xhigh and greedy decoding. GSM8K accepts 7.20 tokens per pass against 5.69 for the official DFlash2 drafter. MATH-500 accepts 6.49 against 5.49, HumanEval 5.71 against 4.87, MBPP 6.14 against 5.04, and MT-Bench 4.48 against 4.11. On replayed agent trajectories at low reasoning, WorkBuddy Office goes from 210 tokens a second autoregressive to 665 with DFlash2 and 806 with DSpine, 3.84x. SWE-bench Pro goes from 242 to 658 and 713. The WorkBuddy benches are Tencent's own, and all of it is the vendor's measurement.
Read the block sizes before the ratios. DSpine drafts 16 tokens per pass and DFlash2 drafts 8. A pass with 7 drafted tokens can never return more than 8, so part of the 7.20 against 5.69 is the longer block. The simplest model separates the two. Assume each drafted position is accepted with the same probability p, given that the ones before it were. Then the expected accepted length of n drafted tokens is p(1 − pn)/(1 − p). The acceptance length is that plus one for the target's own token. The model is crude, because the real rate falls along the block, but it can be inverted. DFlash2's 5.69 at 7 drafted tokens implies p = 0.900. DSpine's 7.20 at 15 implies p = 0.879. Projected to 15 drafted tokens, DFlash2's rate gives 1 + 0.9 × (1 − 0.915)/0.1 = 8.13. The same inversion gives DFlash2 the higher implied rate on all five tasks. Under this model, the longer block explains the card's gap on its own.
The same-block evidence is on Qwen3-8B. In the paper, every drafter is retrained on the same ShareGPT data with five layers and block 16. DSpine reaches 4.82 against 4.39 for DFlash2 and 3.77 for DFlash at temperature zero, a 10 percent lead over DFlash2. In the repository's table, trained on Open-PerfectBlend with the same budget, the lead is 7.83 against 7.55, 3.7 percent. Against the official DFlash checkpoint it is 7.83 to 5.92. The repository's absolute numbers are about 1.6 to 1.7 times the paper's. The training data differs, and both tables are Tencent's runs. So the method's own share is 4 to 10 percent over DFlash2 and 28 to 32 percent over DFlash. The rest of the 27B gap is the extra 8 slots, which a DFlash2 drafter could also have been trained with. Figure 1 runs the inversion on every row.
Measured acceptance lengths from the model card, the paper, and the repository. Each bar is inverted to an implied per-position rate under the constant-rate model, then projected to the block you choose. A dashed outline is the published bar when the solid bar is a projection. Tap a row for its arithmetic. Vendor numbers throughout.
Passes per second times accepted length
Tokens per second is passes per second times tokens per pass. The community GGUF card gives both for an RTX 3090 with a Q4_K_M target and a q8_0 cache. Code runs at 128.7 tokens a second with 7.6 tokens per verify step, so the card completes 16.9 passes a second, 59 ms each. Prose runs at 41.6 with 2.5 per step: 16.6 passes a second, 60 ms each. Math and summarization land at 59 and 58 ms. The pass time is almost constant, and the accepted length sets the speed. Plain decoding on the same card produces 28.2 tokens a second, 35.5 ms per token, so one block pass costs 1.67 plain tokens. That is the break-even. The drafter pays once it returns more than 1.67 tokens per pass. Prose at 2.5 clears it by a little. Code at 7.6 returns 4.5x.
The 1.67 is a property of the 3090 and the CUDA build. A different GPU gives a different number. On an Apple GPU the ratio is worse. The community MLX package for the 27B target timed a 4-bit forward pass on an M5 Max. One token took 34 ms, five took 42 ms, and sixteen took 89 ms. A 16-token block there costs 2.6 plain tokens. So on quantized targets the package verifies a block of 5 by default, and the 4-bit target goes from 31.9 to 64.7 tokens a second. It uses the full block of 16 only on the BF16 target, which goes from 9.1 to 39.1 tokens a second, a 4.31x gain. On a DGX Spark with a Q8_0 target, the author of the upstream llama.cpp pull request measured 29.5 tokens a second with DSpine. DFlash2 gave 29.6. The author wrote that the 27B draft step costs more and offsets its higher acceptance. On an RX 7800 XT over Vulkan, 15 drafted tokens ran at 19.1 tokens a second, below plain decoding at 30.9. The best setting there was 2 drafted tokens, at 47.4. Measure the pass time on your GPU before you believe any ratio.
The context it displaces
Qwen3.8-27B is a hybrid. Its 48 Gated DeltaNet layers hold a recurrent state instead of a key-value cache, and the guide explains why that state cannot be rewound. During verification the target runs 16 positions in one batch. If it rejects position 5, it needs its state as it was after position 4. So llama.cpp keeps a copy of the recurrent state per drafted token during the verify pass, as the GGUF card describes. The SGLang path does the same job with ReplaySSM and an extra Mamba cache buffer. That is what --enable-linear-replayssm-spec and --mamba-radix-cache-strategy extra_buffer in the launch command are for. The size of a copy comes from the target's config.json. Each DeltaNet layer holds 48 value heads of 128 by 128 values. In float32 that is 48 × 128 × 128 × 4 bytes, 3.15 MB per layer and 151 MB across 48 layers. The short convolution adds 3 steps of 2 × 16 × 128 + 48 × 128 values, 5.9 MB more. One copy is 157 MB. Fifteen copies are 2.35 GB.
That memory comes out of the context budget. The key-value cache of the 16 attention layers costs 16 × 2 × 4 × 256 values per token, 32,768 values. That is 64 KiB in f16 and 34 KiB in q8_0, at 34 bytes per 32 values. Eight extra copies, the difference between an 8-token block and a 16-token block, are 1.26 GB, or 36,000 tokens of q8_0 cache. The card measured the drop: 102,400 tokens with DFlash2 at 7 drafted, 65,536 with DSpine at 15, a difference of 36,864 tokens. With no drafter the same card holds 176,128 tokens. With DSpine at 7 drafted tokens it holds 98,304, and with the Q8_0 drafter at 15 it holds 40,960. Two slots at 15 do not fit, and the card says to use 7. Figure 2 prices every setting from the file sizes and the config, and shows where the model lands against the measured values.
A 24 GB card as named segments. The target is the 17.1 GB Q4_K_M file the card used. The reserve is calibrated once from the no-drafter measurement: 24 GB minus the target minus 176,128 tokens of q8_0 cache. The drafter file sizes are from the GGUF listing, the drafter's own 8,192-token window cache and each 157 MB state copy from the two config.json files. What is left is cache at 34 KiB per token. Ticks mark the four measured ceilings.
Running it
Three paths exist on November 9, and only one is the vendor's. The SGLang path is Tencent's out-of-tree plugin, pinned to SGLang 0.5.22 with Python 3.11 and CUDA. The launch command from the card asks for FlashAttention 3 on both target and drafter, a kernel written for Hopper GPUs. It also sets the Triton linear-attention backend, ReplaySSM, a float32 state, 131,072 tokens of context, and 8 running requests. The repository's SPS tuning script sweeps verification widths of 4, 8, 12, and 16 over your own prefixes and writes an sps.json that --speculative-dspark-sps-table-path loads.
# official path: SGLang 0.5.22 plus Tencent's plugin
git clone https://github.com/Tencent/DSpine.git && cd DSpine && pip install -e .
sglang serve --model-path Qwen/Qwen3.8-27B \
--speculative-algorithm DSPINE \
--speculative-draft-model-path tencent/Qwen3.8-27B-DSpine-b16 \
--speculative-num-draft-tokens 16 --dtype bfloat16 \
--attention-backend fa3 --speculative-draft-attention-backend fa3 \
--linear-attn-backend triton --enable-linear-replayssm-spec \
--mamba-ssm-dtype float32 --mamba-radix-cache-strategy extra_buffer \
--max-mamba-cache-size 64 --mem-fraction-static 0.80 \
--context-length 131072 --max-running-requests 8 \
--cuda-graph-max-bs-decode 8 --min-free-slots-delay 1 \
--sampling-backend flashinfer --reasoning-parser qwen3 \
--tool-call-parser qwen3_coder --trust-remote-code
From the model card. Add --speculative-dspark-sps-table-path sps.json after the tuning run.
The llama.cpp path is a community fork. Upstream does not run it yet. The dspine branch of Ajay9o9/llama.cpp is master as of November 8 plus DSpine support. The GGUF repository holds the drafter at Q4_K_M, 1.26 GB, with the bridge tensors kept in BF16. Stock llama.cpp refuses to load the files. A separate upstream pull request opened on November 9 as a draft by another contributor. A contributor who is refactoring the DFlash code asked to hold it until that refactor lands. Until one of those merges, the build is:
# community fork, RTX 3090 settings from the GGUF card
git clone -b dspine https://github.com/Ajay9o9/llama.cpp.git && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j
./build/bin/llama-server \
-m Qwen3.8-27B-Q4_K_M.gguf \
-md Qwen3.8-27B-DSpine-b16-Q4_K_M.gguf \
--spec-type draft-dflash --spec-draft-n-max 15 \
-ngl 999 --spec-draft-ngl 999 -fa on --jinja -c 32768 -np 1 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--spec-draft-type-k q8_0 --spec-draft-type-v q8_0
Two flags matter. --spec-draft-n-max defaults to 3, which wastes most of the block, so set 15 on CUDA and sweep lower on other backends. Set -c yourself, because auto-fit does not reserve room for the drafter and the state copies. If the log shows failed to allocate ... using host copies, lower it.
The MLX path splits in two. Tencent's own dspine_mlx package covers full-attention Qwen3 targets only, so Qwen3-8B. On an M4 Pro it takes the 8B model from 15.37 to 38.52 tokens a second at block 16, with 6.53 tokens accepted per round. For the hybrid 27B target there is a community package, dspine-hybrid-mlx, published on November 9, with a matching copy of the drafter. It adds rollback of the DeltaNet state and the drafter's window, and it runs with any MLX build of the target. Its numbers above come from one MacBook Pro M5 Max with 128 GB. I have not run it.
# community package for the hybrid 27B target on Apple Silicon
pip install "git+https://github.com/internet-projects/dspine-hybrid-mlx.git"
dspine-hybrid-generate \
--target mlx-community/Qwen3.8-27B-4bit \
--draft netprojects/Qwen3.8-27B-DSpine-b16-MLX \
--prompt "Write a Python function that checks whether a number is prime." --baseline
vLLM has no DSpine support as of November 9. A community quantization card ships an experimental vLLM plugin beside INT4, FP8, and NVFP4 copies of the drafter, which I did not test.
Measure acceptance on your prompts
Every number above is a mean over somebody's prompt set. The knob in the machine stands for a rate you can read from your own server. The llama-server log prints one line per finished request, in the form draft acceptance = 0.40800 (612 accepted / 1500 generated), mean len = 7.12. The mean len is the acceptance length, one plus accepted drafts per verify step. At --verbosity 4 it also prints acc per pos = (0.912, 0.840, ...). That is the share of verify steps that accepted each draft position, and a step counts position two only if position one also passed. Divide each share by the one before it to get the per-position rate the machine's knob stands for. The denominator is every verify step, so the ratios hold when each step drafts the full block. Or move the knob until the machine's tokens per pass matches your mean len. SGLang prints accept len and accept rate in its decode batch log line and exports them as metrics. The procedure from the GGUF card is the right one. Take ten prompts from your real traffic. Run them with --spec-draft-n-max at 15, 7, and 3, and with no drafter. Compare tokens per second beside mean len. If mean len on your prompts is under 2 at 15 drafted tokens, stop. You pay 2.35 GB of context for a pass that returns about what it costs.
When it does not pay
- Your prompts accept few tokens. Prose and summaries accept 2.5 to 3.6 per step on the 3090, and 15 drafted tokens then cost more context than they return. Try 3, or DFlash2, or the target's MTP head.
- The pass is expensive on your GPU. On an Apple GPU a 16-token pass costs 2.6 plain tokens, and on one Vulkan card 15 drafted tokens ran slower than plain decoding. Sweep
--spec-draft-n-maxfrom 2 upward. - The context does not fit. Auto-fit fails, or the log reports host copies. Lower
-c, lower--spec-draft-n-max, or use the Q4_K_M drafter instead of Q8_0. - Two slots on 24 GB. Use
--spec-draft-n-max 7, or one slot. - The output differs at temperature 0. Batched verification rounds differently from one token at a time, so near-tied tokens flip. The pull request measured every flip at a top-2 gap of at most 0.015 on BF16 and 0.46 on Q4_K_M.
- The drafter does not match the target. It reads layers 1, 16, 31, 46, and 61 of Qwen3.8-27B and serves no other checkpoint. A fine-tune needs a retrained drafter. SpecForge has DSpine training, and a fix to its training objective is still an open pull request.
- The engine version is wrong. The plugin pins SGLang 0.5.22, stock llama.cpp refuses the GGUF, and Tencent's MLX package needs mlx-lm 0.32 or newer.
- Concurrency is high. At 8 requests the card still reports 2.7x to 3.8x on agent replays. A full batch leaves less idle compute for a drafter to use, as the DFlash 2 essay showed at concurrency 32.
Three numbers before you deploy
The design choice was where the predecessor enters, and it changed two prices at once. The draft got better: 4 to 10 percent over DFlash2 at the same block, 9 to 27 percent on the card with the longer block. The verify pass got longer, and on this target every drafted token holds 157 MB of state while the target checks it. Three measurements decide whether it pays on your machine. Read the per-position acceptance on your prompts from the server log. Time one pass on your GPU, with the block you intend to use, against one plain token. And count the context your sessions need, because 15 drafted tokens cost 110,592 tokens of it on a 24 GB card. Multiply the first two for the speed. Subtract the copies from the third. If the speed holds and the context still fits, run it.
Keep reading