Revolut's Self-Hosted GLM: Where $0.30 per Million Tokens Comes From
Revolut made a self-hosted GLM model on 56 B200 GPUs the default coding model for about 2,000 engineers, in place of metered coding-agent APIs. Its infrastructure lead reports a blended cost of $0.30 per million tokens, a tenth of the old invoice. This essay rebuilds that number from the hardware, the model's config.json, and four serving levers. Then it prices a cluster of your own size.
David Abitbol gave the talk at the AI Engineer conference in New York in October 2026 (video, 12 minutes). He gave a 15 minute version at GTC Berlin with Together AI (NVIDIA On-Demand). The Register reported it on November 3. Every Revolut number below comes from the two talks, and I say so where the two tellings differ. Abitbol named his levers but did not publish his settings, his traffic, or his turn shape. So the arithmetic is mine, and every assumption in it is labeled and adjustable. It shows what the levers can do on a model of this cluster, not what Revolut runs.
Engineer terminals feed a gateway and a row of 8-GPU nodes. Each node is cut open to show its weight slab and its KV cache tank. The dark base of each tank is the weight slab. The knobs change the fleet, the price, the quantization, the tensor parallel size, the prefill share, the context cap, the demand, and the target hit rate. The prefill share starts at zero, one mixed engine per replica. Model sizes come from the Hugging Face file lists and config.json. The B200 numbers come from the DGX B200 specification. The 40% prefill efficiency, the 60% of peak bandwidth, the 16K floor, and the two minute gap between turns are assumptions. The 400 output tokens are the median of NVIDIA's agentic trace. The verdict prices one day two ways. It gives the prefill load as a share of capacity, and the first-token wait and the decode speed as multiples of their values at 1 request a second. It shows no absolute times, because it leaves out routing, reasoning output, tool calls, and real kernel speed. The model locates the knee, where prefill demand meets prefill capacity. The absolute times come from Revolut's measurement only. The 70% line is a sizing margin I chose. The model does not compute it.
Why they moved
Revolut rolled AI coding tools out in stages. Gemini came first in January 2025. Claude Code followed, with a budget per engineer that Abitbol put at $300 a month in the GTC version of the talk. In his words, "we were seeing engineers spend $1,000 a month, $10,000 a month with not so much ROI." A budget that small ran out mid-month, so engineers saved tokens for the hardest task of the quarter.
In April 2026 the company started serving open weights on its own GPUs, first GLM-5.1 and later GLM-5.3. The promise was unlimited tokens. The chart in the talk covers a sample of 2,000 engineers. After that date it shows coding time down about 60% and lead time down about 70%. Lead time fell more than coding time. Abitbol reads that as engineers pointing the model at Jira, Confluence, and their inbox once tokens stopped being scarce. The chart is observational. It does not separate the model change from the budget change, and the talk does not claim that it does.
The service targets came from the same experience. "What you want to aim for is a sub-3 second time to first token. ... As soon as you exceed 5 seconds, my Slack inbox starts to blow up." Throughput matters less. The cluster serves about 200 tokens a second per stream in the working day and about 800 after 5 pm. People run several tasks at once and check back, so the stream is rarely watched.
What the cluster is
The fleet is 56 B200 GPUs, the count of seven 8-GPU nodes. The DGX B200 specification gives 1,440 GB of HBM3e and 64 TB/s per node. So each GPU holds 180 GB and reads it at 8 TB/s. Dense FP8 compute is 36 PFLOPS per node, half the sparse figure on the sheet. Abitbol does not say whether Revolut owns the nodes or rents them. The GTC session was presented with Together AI, which sells dedicated GPU capacity.
The model is GLM-5.3 from z.ai. Its config.json describes 78 layers and a hidden size of 6,144. Each layer has 256 routed experts, 8 of them active per token, plus one shared expert. The window is 1,048,576 tokens. The GLM-5.3 guide counts 743 billion parameters in the main stack, 753 billion with the draft head, and about 39 billion active per token. Every layer keeps a compressed attention cache of 576 values per token, and a sparse indexer picks the 2,048 cached tokens each step reads. In FP8 that cache costs 44,928 bytes per token: 11.8 GB for a 256K session and 47.1 GB for a 1M session.
Two checkpoints matter here. The default zai-org/GLM-5.3 repository is FP8 and its safetensors files sum to 755.63 GB. Inferact/GLM-5.3-NVFP4 sums to 464.82 GB. Its quantization config lists what it left alone. That list holds the embeddings, the output head, the three dense layers, every attention projection, every indexer, every router, and every shared expert. Only the routed experts are 4-bit. Both sizes are sums of the Hugging Face file lists, read on November 6, 2026.
Where a dollar per million tokens comes from
A rented cluster costs the same every hour. The price of a token is that rent divided by the tokens the cluster gets through. The rent is GPUs times the hourly price times 24. The tokens are the peak demand, times the average load as a share of the peak, times the tokens per turn.
Take the machine's defaults. Price aggregators put the median on-demand B200 at $8.11 an hour and the median reserved rate at $6.62 on November 6, 2026 (GetDeploying). Nebius lists $8.50 on its own page. At $6.62, 56 GPUs cost $8,897 a day. The talk puts the peak at about 8 requests a second. Engineers work about a quarter of the hours in a week, and load inside the day is uneven. So the machine assumes an average of 35% of the peak. That is 241,920 turns a day and $0.037 a turn.
Tokens per turn decide the rest, and here the accounting splits. A coding harness replays the whole conversation on every turn. Suppose sessions grow until the harness compacts them near a 256K window. Then the mean context per turn is about 126,000 tokens, plus 400 output tokens. Count every one of those tokens, the way an API bills, and the day comes to $0.29 per million. Count only the tokens the GPUs compute, the 10% that miss the prefix cache plus the output, and the same day costs $2.83 per million. The 30 cents reproduces under the first accounting. That is the right accounting for a comparison with an API invoice. The old invoice counted every token too, at a discount for cache reads. Neither talk says which count sits under the 30 cents. The machine shows both.
At $0.30 per million, "a tenth of the invoice" puts the old blended price near $3 per million tokens. That is the gap between a frontier model's list price and an open model's. Self-hosting is the delivery mechanism, and the model choice is most of the saving. The last section prices the two apart.
Lever one: NVFP4 against FP8
Abitbol's rule for Blackwell cards is to run nothing above FP4, and "NVFP4 is best." He put the saving at roughly 40% of the GPU footprint, for "zero loss in precision." The file sizes say 38.5%: 755.63 GB against 464.82 GB. NVFP4 stores each weight in 4 bits and adds one FP8 scale per block of 16 (NVIDIA). So a routed expert costs 4.5 bits per weight instead of 8. NVIDIA reports 1% or less accuracy loss against FP8 on its own DeepSeek-R1 runs. Revolut published no evaluation of its own. The Inferact model card shows no comparison with FP8, so the "zero loss" claim rests on his word.
The saving goes to the cache. At TP8 the FP8 weights take 94.5 GB of each 180 GB card, and the NVFP4 weights take 58.1 GB. By default, vLLM reserves 8% of the card (gpu_memory_utilization 0.92). The machine sets aside another 6 GB per GPU for activations and graphs. That leaves a KV pool of 521 GB per FP8 replica and 812 GB per NVFP4 replica: 11.6 million against 18.1 million cached tokens. The 290 GB of weights you no longer hold become 6.5 million tokens of prefix cache per replica.
NVFP4 does not make a quiet cluster faster. At batch one, one decode step reads every active weight once. The FP8 build reads about 40 GB per step. The NVFP4 build reads about 48 GB, because the attention, the shared experts, and the dense layers stayed in BF16 at two bytes a weight. At batch 64, a step touches about 222 of the 256 experts in every layer, so the expert bytes dominate. That is 647 GB per step on FP8 against 390 GB on NVFP4. Tokens per second is a memory bandwidth number, so the 4-bit experts pay at load and not at night. The vLLM recipe's command for the build, with the context cap from lever three added:
VLLM_USE_RUST_FRONTEND=1 vllm serve Inferact/GLM-5.3-NVFP4 \
--tokenizer-mode glm5_next \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--kv-cache-dtype fp8_e4m3 \
--max-model-len 262144 \
--reasoning-parser glm47 \
--tool-call-parser glm47 \
--enable-auto-tool-choice \
--served-model-name glm-5.3-nvfp4From the vLLM recipe for zai-org/GLM-5.3, with --max-model-len added. Not run here.
Lever two: TP8 against TP4
GLM fits at TP4, he said, but then there is not enough KV cache, "and so maybe a TP8 works better." Tensor parallelism splits one replica across GPUs. A fleet at TP4 holds twice as many replicas as the same fleet at TP8, and twice as many copies of the weights. Fifty-six GPUs at TP4 hold 14 copies of the 465 GB NVFP4 weights, 6.5 TB of a 10 TB fleet. At TP8 they hold 7 copies, 3.3 TB. The difference is 3.3 TB of KV pool, 54 million cached tokens against 127 million.
The FP8 build makes the choice for you. At TP4 its weights come to 189 GB per GPU, 9 GB more than the card holds, and vLLM refuses to start. So "fits on TP4" was already a statement about NVFP4. The price of TP8 is that every request's prefill spreads over eight GPUs with an all-reduce after each layer. That makes one request faster and the cluster no faster, since half as many replicas each do twice the work. NVIDIA's Dynamo recipe for this model splits the difference: TP4 on the prefill workers and TP8 on the decode worker.
Lever three: the 256K cap
"Long context is a scam. ... Even though you can serve 1 million tokens, 256 is really what you need." An empty window costs nothing. Agents fill whatever window they get, because coding harnesses compact a session only near the limit. A session that compacts near 256K averages about 126K tokens of context per turn. A session that compacts near 1M averages about 480K.
Now multiply by the live sessions. At 8 requests a second, with a turn every two minutes per session, about 960 sessions are live at the peak. Their prefixes take 121 million tokens of cache at the 256K cap, which the TP8 NVFP4 fleet holds in full. At the 1M cap they take 461 million, and the fleet holds 27% of them. The rest get evicted between turns. The hit rate falls from 90% to about 25%, and each turn computes 360,000 tokens instead of 12,600. The machine's prefill rate is 183,000 tokens a second per TP8 replica. At that rate the 1M cap puts prefill demand at 78% of capacity at 4 requests a second and at 134% at 5. So the fleet falls behind above about 4 requests a second.
The engines agree about the cost. The vLLM recipe reaches the 1M window on 8 B200s only with --max-num-seqs 32, because the window and the concurrency share one pool. NVIDIA's B200 recipe stops at 500K and spills cache to host memory. Set the cap with --max-model-len 262144 and your harness will compact earlier. One caution: NVIDIA's agentic trace has a median context of 64K tokens. If your sessions look like that trace, they never reach either cap, and this lever moves nothing.
Lever four: prefill and decode apart
Coding agents are prefill heavy because of the replay. Every turn sends the full context back, and the prefix cache turns most of it into a lookup. The uncached tail still has to attend over the whole context. With GLM-5.3's indexer, every new token scans the index keys of the 21 layers that own an indexer. At 126K that is about 680 MB of BF16 keys per new token, before the 2,048 selected latents. Prefill is compute bound, and decode is bandwidth bound, so the two phases want different machines.
In one engine the two phases share a step. On a B200, vLLM gives each step a budget of 16,384 tokens (get_batch_defaults). It schedules the decode tokens first and fills the rest with prefill chunks (chunked prefill). At the machine's prefill rate a full chunk takes as long as about 70 decode steps at batch one. Every decode stream on that replica waits for it. With 90% hits the chunks are rare and the stall is noise. With a cold cache at 1M, almost every step carries a chunk. The machine then puts each stream at about 1% of its low-load decode speed.
Disaggregation moves prefill to its own replicas. The vLLM docs name the two reasons: tune time to first token and inter-token latency separately, and control the tail of inter-token latency. The same page says it "DOES NOT improve throughput." The cost is a KV transfer per turn, 5.7 GB for a 126K context, which a 400 Gb/s port moves in about 0.11 s. The prefix cache also has to live where prefill runs, so a 50% split leaves the working set half the pool it had. The machine shows the hit rate fall when you split. NVIDIA's recipe answers it with KV-aware routing and a host-memory cache behind the prefill workers. A DGX B200 node has 2 TB of host memory for that. The Register reports the split as implemented. In the talk, Abitbol lists it with speculative decoding among the things to check once the big levers are set. The SGLang launch, from its docs:
python -m sglang.launch_server --model-path Inferact/GLM-5.3-NVFP4 \
--disaggregation-mode prefill --tp-size 8 --port 30000 \
--disaggregation-transfer-backend mooncake
python -m sglang.launch_server --model-path Inferact/GLM-5.3-NVFP4 \
--disaggregation-mode decode --tp-size 8 --port 30001 \
--disaggregation-transfer-backend mooncake
python -m sglang_router.launch_router --pd-disaggregation \
--prefill http://prefill-host:30000 --decode http://decode-host:30001 \
--host 0.0.0.0 --port 8000Flags from the SGLang PD disaggregation docs, with this model's path. Not run here.
The queue, not the GPUs
The slide behind the talk follows the same 56 GPUs from the start of self-hosting to August. In the GTC telling, demand rose from about 4 requests a second to more than 8. Time to first token fell from above 10 s to below 1.8 s. The slide reads 14.26 s and 1.68 s. The New York telling says about 3 and more than 7 requests a second. Abitbol called it an 88% improvement for a 76% rise in demand in New York, and for a 78% rise at GTC. The GPU count never changed. Abitbol's reading is that the serving parameters are where the gains are, and "you really need to resist scaling."
The machine cannot reproduce those seconds. It holds only the weights, the cache, and the prefill queue, and it runs the kernels at fixed shares of peak. Routing, reasoning output, tool calls, and real kernel speed are not in it, so its absolute times come out far below Revolut's. For that reason the machine and the figures give no modeled seconds. They give the prefill load as a share of capacity, and the wait as a multiple of the low-load wait. The model locates the knee, where prefill demand meets prefill capacity and the queue stops draining. The absolute time comes from Revolut's measurement only: 14.26 s at the start and 1.68 s in August. Revolut also measured about 200 tokens a second per stream in the working day. What the model does show is the shape. Time to first token is a queue. Each turn needs its uncached tokens computed, and the prefill replicas compute a fixed number of tokens a second. When demand reaches that rate, the queue grows without bound, and the first token comes whenever the backlog clears. Every lever above moves one of the two sides. The cap and the hit rate set how many tokens a turn needs. The quantization and the tensor parallel size set how many prefixes survive, which sets the hit rate. The split sets how many replicas compute prefill. The figure replays the climb with the levers as switches.
Demand climbs from 4 to 8 requests a second, the range in the GTC talk, on 56 GPUs. Each switch sets one lever in the machine's model, and the curve is its prefill load at each step of the climb: the uncached tokens the turns need, as a share of what the prefill replicas compute. The label over each point is the first-token wait as a multiple of the low-load wait. The red dashed line is the knee at 100%, where the queue becomes unbounded. The green dashed line is a 70% sizing margin. I chose it, and the model does not compute it. The open circles on the top row are the two measured numbers on Revolut's slide. The levers start where a first attempt often starts: FP8 weights, the full window, one mixed engine, and 60% hits.
The curve is the machine's model. It holds only the weights, the cache, and the prefill queue, so it gives load and multiples, not seconds. The model locates the knee at 100% of capacity. The absolute time comes from Revolut's measurement only: 14.26 s at the first point and 1.68 s at the last.
When self-hosting does not pay
The fleet costs $8,897 a day at 35% load and the same at 100%. So the cost per token is a utilization number before it is a hardware number. At 100% of the peak the same fleet serves tokens at $0.10 per million. At 10% it serves them at $1.02. A team on one time zone, with evenings and weekends idle, lives near the low end unless it finds night work for the GPUs.
Then compare against the same model behind someone else's API. The z.ai price list has GLM-5.3 at $1.40 per million input tokens, $0.26 for cached input, and $4.40 for output (pricing). Under the machine's turn shape and 90% hits, that is $0.049 a turn, against $0.037 self-hosted at 35% load. Add two platform engineers at $250,000 a year loaded and the self-hosted turn costs $0.042. The saving against the same model's API is about 13% at this scale. Below about 1,400 engineers the API is cheaper at these inputs, apart from a few team sizes where the last node is exactly full. Above 1,400 self-hosting stays ahead. The order-of-magnitude saving in the talk is the saving from changing the model, and it needs no cluster to collect. Self-hosting then adds a flat bill, data residency, and the 800 tokens a second after hours.
The talk gives no number for the ops work, so the figure makes it a slider. Someone owns the engine upgrades, the GPU failures, the router, the sandbox rules, the telemetry, and the on-call rotation. That person is the first head in the ops line.
Team size runs along the bottom on a log scale. The fleet rent climbs in 8-GPU steps, because the prefix working set of the live sessions sets how many TP8 NVFP4 nodes a team needs at the 256K cap. The ops line adds a fixed amount per head. The API bill is a straight line at the blended price per million tokens. The default API price is z.ai's list price for the same model under this turn shape. Move it to about $3 to see an invoice ten times the self-hosted one.
Turns use the machine's shape: 126K tokens of context and 400 of output at a 256K cap, 90% hits. Nodes are the larger of two counts. One is the prefix working set plus the cache of the turns in flight, over one node's pool. The other is the prefill demand over 70% of its prefill rate. The z.ai list price gives $0.386 per million under this shape, rounded to $0.39 on the slider.
How to size your own
- Read the turn shape from your gateway logs: mean context per turn, output per turn, and the prefix hit rate you get today.
- Read how far sessions grow before the harness compacts them, and set the context cap there. Reserve the full window only for a named workload.
- Multiply the live sessions at the peak by the mean context. Pick the quantization and the tensor parallel size that hold that working set with the fewest copies of the weights.
- Compute the uncached tokens per second at the peak and divide by the prefill rate of the replicas. Keep that ratio under about 70%, because the queue grows fast above it.
- Run one mixed engine first. Split prefill from decode when the inter-token latency tail, not the first token, is what your engineers notice.
- Price the day at your measured load share, and put the same model's API price next to it before you sign for nodes.
Failure modes
The first token takes 10 seconds at low demand. The hit rate is below the target, often because the router spreads one session's turns over replicas that have never seen its prefix. Route by session or by prefix, and read the hit rate from the engine's metrics before you add GPUs.
The server refuses to start with "KV cache is needed". The weights left no room for the cache at that tensor parallel size. The vLLM recipe records exactly this for FP8 on 8 H200s at the 1M window and caps it at 256K. Lower the cap, raise the tensor parallel size, or take the NVFP4 build.
Decode stalls at regular intervals. Prefill chunks share the step with your stream. Lower --max-num-batched-tokens at the cost of first-token time, or split prefill from decode.
Tool calls come back as plain text. GLM-5.3 always thinks before it answers. Without --reasoning-parser glm47 and --tool-call-parser glm47, the harness receives the reasoning and the calls as one string and stops.
On this cluster, the price of a turn is the day's rent divided by the turns the engineers send. The levers move neither number. They decide whether the same nodes keep prefill demand safely below the knee, or whether you add nodes and raise the rent. Your own measurement tells you how many seconds that knee is worth. The cap and the hit rate set the uncached tokens in each turn. The quantization and the tensor parallel size set how many prefixes the pool keeps. Measure your own turn shape and hit rate, and then put them into the machine.
Keep reading