Engine internals guide · October 14, 2026 · 38 min read

Inside SGLang

Ask an agent five questions about the same codebase and it sends the same ten thousand tokens of context five times. A plain serving engine computes those tokens five times. SGLang keeps them in a tree and computes them once. This guide follows one request through SGLang v0.5.22, the release tagged on October 13, 2026, and stops at each part that decides speed or memory. Eight figures let you drive the real rules. Three of them replay data captured on a Mac, and one replays GPU runs. The numbers come from the source at that tag and the project docs. They also come from runs on an Apple silicon Mac, an NVIDIA L4, and a DGX Spark.

Step 1: Why SGLang exists

A language model answers one token at a time. At every layer, each new token reads the stored key and value vectors of all earlier tokens. An engine keeps those keys and values in GPU memory so it does not compute them again. That store is the KV cache. It grows with every token of every live request, and after the model weights it takes the most memory.

In 2023, most engines threw the KV cache away when a request finished. That was a waste for the programs people were starting to write. A few-shot prompt repeats the same examples on every call. A chat repeats the whole history on every turn. An agent repeats its system prompt, its tool list, and its notes on every step. Tree-of-thought search branches from one shared stem.

The SGLang paper by Zheng and colleagues (arXiv 2312.07104) made one design choice the center of the system. Keep the KV cache of finished requests in a radix tree keyed by token sequence. A new request walks the tree, takes the longest prefix that is already there, and computes only the rest. The authors named this RadixAttention. They also wrote a small frontend language, with gen, select, and fork, so that programs with branches would produce shared prefixes on purpose.

That frontend is still in python/sglang/lang/ at v0.5.22, but most users now call the OpenAI-style HTTP API or the in-process Engine. The frontend is a client library. It talks to the runtime through the same /generate route as everyone else. The idea that lasted is in the runtime: cached prefixes and running requests share one memory pool, and the scheduler knows about the tree.

The project grew from that idea into a standard piece of AI infrastructure. It started in 2023 at LMSYS, the research group behind Vicuna and Chatbot Arena, and TechCrunch traces it to Ion Stoica's lab at UC Berkeley. In March 2025 it joined the PyTorch ecosystem. On October 16, 2026, the GitHub API reported 37,209 stars, 9,615 forks, and 2,074 contributor entries, including anonymous commits.

Two habits explain much of that growth. The first is day-0 support. When a large open model launches, the SGLang team often publishes a serving recipe on the same day, and DeepSeek releases are the long-running example. The second is published results at scale. In May 2025 the team reported serving DeepSeek on 96 H100 GPUs at 52.3k input tokens and 22.3k output tokens each second on each node. That is the team's own claim, with the setup published for anyone to repeat.

In May 2026 the core maintainers started a company, RadixArk, with a $100 million seed round led by Accel. The launch release says SGLang runs on hundreds of thousands of GPUs and produces trillions of tokens a day. It names Google, Microsoft, NVIDIA, Oracle, AMD, LinkedIn, xAI, and others as users. Those are company claims. LinkedIn has written about its own production use in detail.

Popularity does not explain speed, though. The rest of this guide does. It follows one prompt from the HTTP socket to the last streamed token. On the way it stops at each mechanism that makes SGLang fast. The order is the CPU and GPU split, the attention math, prefix reuse, the memory tiers, the scheduler, decoding, and scale-out.

Keep one question in mind for the rest of this guide. Every few milliseconds the scheduler asks which tokens the GPU computes next. Each step below is one part of the answer.

Step 2: Follow one prompt through CPU and GPU

Start the server and send it one prompt. On a Mac, SGLang runs through MLX. This is the exact command behind every Mac number in this guide, with SGLang 0.5.22, MLX 0.32.4, and the 4-bit Qwen3-0.6B model:

SGLANG_USE_MLX=1 sglang serve --model-path mlx-community/Qwen3-0.6B-4bit \
  --disable-cuda-graph --host 127.0.0.1 --port 30000 \
  --enable-cache-report --enable-metrics --mem-fraction-static 0.4

curl -s localhost:30000/generate -H 'content-type: application/json' \
  -d '{"text":"The capital of France is","sampling_params":{"max_new_tokens":12,"temperature":0}}'

The process tree that answers has three parts, and the reason is Python. Tokenizing and detokenizing text is CPU work. The scheduler must start a GPU step every few milliseconds. If all three shared one Python interpreter, they would wait on one global lock. So the Engine starts them as separate processes and connects them with ZMQ sockets.

The main process runs the HTTP server and the TokenizerManager. It turns the request into token ids, records a ReqState with an asyncio event, and pushes a TokenizedGenerateReqInput message to the scheduler. The HTTP handler then waits on that event.

The scheduler process owns the GPU. Under tensor parallelism each GPU has a scheduler process, and only rank 0 reads the socket. Rank 0 copies every request to its peers, so all ranks build an identical batch. The scheduler builds a Req, places it in the waiting queue, and later runs it in a batch through the ModelRunner.

Each step, the scheduler sends new token ids to the detokenizer process. The detokenizer decodes them with two offsets per request. It holds back text that ends in half of a multi-byte character, and it sends complete text back to the main process. There, the event fires and the HTTP response streams the new text.

Stop conditions live in the scheduler, not in the detokenizer. The scheduler checks stop strings before end-of-sequence tokens, because speculative decoding can accept several tokens in one step. A streamed response also delays text at the end of the output if it matches the first characters of a stop string. If you see a few characters arrive late, that is the reason.

What runs on the CPU, and what runs on the GPU

Split the work of one request into two kinds. The CPU parses the HTTP body, applies the chat template, and turns text into token ids. In the scheduler process, the CPU also walks the radix tree, builds the batch, allocates KV slots, and fills the grammar mask. The detokenizer process turns ids back into text on the CPU.

The GPU does the arithmetic. It looks up the embedding row of each token and runs every layer's matrix multiplies and attention. Then it projects the last hidden state to logits over the vocabulary and samples the next token. For Qwen3-0.6B that is 28 layers and a vocabulary of about 152,000 entries.

ZMQ moves each message between processes over a local IPC socket. By default the message is a pickled Python object, because SGLANG_USE_PICKLE_IPC is true at v0.5.22. The hop is cheap, but it is not free, and it happens for every request and every step of output.

The figure below replays five real requests to the Mac server. Each one carries the timestamps that --enable-metrics adds to meta_info. Pick a request, then step through the stages and watch which process holds it.

Three readings stand out. Tokenizing and the handoff to the scheduler took well under one millisecond each, even for a 200-token prompt. The first real prefill took about 800 ms, because MLX compiled its kernels on that request. The same prompt a second time took 20 ms, with 9 of its 10 tokens served from the tree.

On a big GPU, the arithmetic per step is much faster, so the CPU share of each step grows. LinkedIn's write-up found that its first bottlenecks were CPU tokenization, ZMQ message splitting, Python garbage collection, and the global interpreter lock. That is the reason SGLang keeps adding Rust pieces at the edges, such as the gRPC server and the radix tree core.

The native response from the command above carries more than text. Its meta_info reported prompt_tokens: 5, cached_tokens: 2, num_retractions: 0, a queue time of 0.24 ms, and a first-token latency of 889 ms. That first request paid for warm-up.

The fields matter more than the values. The cached_tokens field tells you how much of the prompt came from the tree. The num_retractions field tells you whether the scheduler removed this request in the middle of its answer. Steps 4 and 8 explain both.

Step 3: Look inside the forward pass

The GPU spends its time in the forward stage of the last figure. Most of that time goes to two operations: matrix multiplies against the weights, and attention. Attention is also where the KV cache enters the math. So it helps to see the matrices with real numbers.

Take Qwen3-0.6B. Its hidden size is 1,024. Each layer projects every token into queries for 16 heads, and into keys and values for 8 heads. Each head is 128 numbers wide. For a prompt of n tokens, one layer makes a query matrix of n by 2,048. The key and value matrices are n by 1,024 each.

Two query heads share each key and value head. This is grouped-query attention. It halves the KV cache compared with one key and value head per query head. That is why the model stores 114,688 bytes per token and not twice that.

For one head, attention takes four steps. Multiply the query matrix by the transposed key matrix to get a score for every pair of tokens. Scale the scores by 1 over the square root of 128, which is 0.0884. Set every score above the diagonal to minus infinity, so no token can look at a later token. Then take a softmax along each row and multiply the result by the value matrix.

The figure runs those steps on real numbers. I loaded the 4-bit Qwen3-0.6B weights with MLX and recorded the queries, keys, and values of every layer for one short sentence. The sentence is "The cat sat on the mat because it was tired".

Look at layer 6, head 1. The row for " it" puts 0.939 of its weight on " cat". That head resolves the pronoun. Layer 27, head 0, puts 0.759 of the same row on the first token, "The". Many heads put spare weight on the first token when no other token matters. People call this pattern an attention sink.

Prefill and decode have different shapes

During prefill, every prompt token is a query at the same time. The score matrix for one head is n by n, and the GPU fills it with large matrix multiplies. That work is compute-bound, because each weight and each key is reused across many queries.

During decode, there is one new query per request. The score matrix is 1 by n. The new token must still read the key and value of every earlier token, in every layer. For this model at 32,768 tokens of context, that is about 3.76 GB of KV per step per request. Decode is therefore memory-bound. Its speed depends on how fast the GPU reads memory, and batching many requests shares the weight reads across them.

Paged KV and the kernels that read it

The keys and values do not sit in one block per request. SGLang stores them in a pool of slots, one slot per token by default. A table maps each request position to a slot. SGLang calls this table req_to_token. The attention kernel reads the slot indices for a request and gathers its keys and values from wherever they are.

That indirection is what makes prefix sharing possible. When two requests share a prefix, the leading entries of both rows hold identical slot numbers. No copy happens.

The kernels come from attention libraries that SGLang calls through one backend interface. FlashInfer handles both shapes well. For prefill it has a ragged kernel for the new tokens, stored back to back, and a paged kernel for the cached prefix. For decode it has a paged kernel that reads one query against many scattered slots. At v0.5.22 the default on Hopper GPUs is FlashAttention 3, and on Blackwell GPUs it is the TensorRT-LLM attention kernels through FlashInfer.

Decode has one more cost that is not math. Each step launches hundreds of small kernels for a few tokens, and launching them from Python takes time. SGLang records those launches once per batch size as a CUDA graph and then replays the graph with one call. Decode graphs are on by default, for batch sizes up to a limit set by GPU memory. That limit matters again in Step 14.

Step 4: Grow a prefix tree, then run out of room

The tree is the RadixCache class in mem_cache/radix_cache.py, and at v0.5.22 the default controller is UnifiedRadixCache with a Rust core. The rules are the same in both. Each node holds a run of tokens and, as its value, the KV pool slots where those tokens live. Children are keyed by their first token, so a lookup walks one branch.

A lookup is match_prefix. It walks down while the tokens agree. If the match ends in the middle of a node, the node splits in two, so the tree records the exact boundary. The walk also updates the access time of each node it passes. The result is the matched length and the slot indices along the path.

One rule surprises people. The scheduler caps a match at the prompt length minus one. Even a fully cached prompt computes its last token, because the engine needs that token's logits to sample the first output. Asking for prompt log-probabilities from position k also caps reuse at k.

When a request finishes, its prompt and its output join the tree as new nodes. If another request already added the same prefix, the duplicate slots go back to the free list. The finished request's tokens are now cache, and the next turn of the same chat can start from them.

The tree and the running requests share one pool. There is no separate cache size to set. Cache is any KV that no running request holds right now.

A running request locks the last node it matched, and the lock covers its whole path to the root. When the pool is short, evict frees unlocked leaves in least-recently-used order. A parent that becomes a leaf is next in line. So eviction removes a branch from its tip toward the root, and a shared stem lasts longest.

Drive it below. The pool holds a support bot's 2,048-token system prompt, a translator's prompt, and the questions and answers of a few users. Send requests in any order. Then make the pool smaller, or pin user A's path as if their request were still running, and watch what eviction is allowed to take.

Two things in that figure carry over to real traffic. First, the system prompt is the node that survives, because every user's path runs through it and it is touched the most. Second, a pinned path cannot be evicted. When pins fill the pool, a new request waits in the queue even though the tree holds plenty of old answers.

A Mac shows how the tree works. A GPU shows the size of the effect. During the SGLang 101 manual work we ran the same shared-prefix benchmark on an NVIDIA L4 24 GB with Qwen3-8B and SGLang 0.5.22. Each run had four groups of 16 prompts, a 128-token question, 64 output tokens, and at most eight requests at a time. We changed the shared system prompt from 0 to 4,096 tokens, with three seeds per point.

With a 4,096-token shared prompt, the median time to first token was 479 ms with the tree and 14,706 ms with --disable-radix-cache. Output was 88.87 tokens per second against 20.80. That is a 30.7 times shorter wait and 4.3 times the throughput. With no shared prompt, the median was 349 ms with the tree and 379 ms without it, and the throughput was the same. The tree costs almost nothing when it has nothing to reuse.

One caveat belongs with those numbers. The card ran hot during the longest run without the cache, and 62% of its samples carried a thermal throttle flag. Read the 30.7 times with that in mind. Figure 8 at the end of this guide replays every point of this sweep.

The scheduler can also use the tree to order work. The default --schedule-policy is fcfs. With lpm, longest prefix match, the waiting queue is sorted by how much of each prompt the tree already holds. The paper proves that a depth-first visit of the tree gives the best hit rate for a fixed batch, and lpm follows that order. The code turns lpm back into first-come-first-served when more than 128 requests wait, because the sort costs CPU time on every pass.

There is also a small trick for siblings that arrive together. Two waiting requests can share a prefix of 32 tokens or more that is not yet cached. Then the scheduler holds the second one back for one round. The first one writes the shared part, and the second one then reads it from the tree.

Step 5: Find where your prompt breaks the match

The tree only helps if your prompts start the same way. One different token near the start makes every token after it a miss, because a radix match stops at the first difference. Timestamps, request ids, and shuffled tool lists are the usual causes. This is a client problem, and no server flag repairs it.

I measured the effect on the Mac. One script sent five rounds of a chat request with a long system prompt and a 16-token answer. In the first case the prompt stayed the same. In the second case a fresh request id sat at the very start of the system message. In the third case the same id moved to the end, just before the user question.

The stable prompt had 2,999 tokens. From round 2 on, the tree served 2,998 of them, and the median time to first token was 45 ms. With the id at the start, the tree served about 5 tokens of chat template header. The median time to first token was 1,255 ms. With the id at the end, the tree served about 2,985 tokens and the median was 88 ms.

The fix is an order, and it costs nothing. Place fixed content at the top: system prompt, tool definitions in a stable order, and long reference text. Put content that changes per request last: ids, dates, and the user turn. Some agent clients add a status block after the last system instruction. Each update to that block moves the break point into the middle of a long prompt.

Sharing also shows up at scale. A second benchmark sent 64 requests in 4 groups of 16, eight at a time. Each group had its own 2,048-token system prompt and a 128-token question. The server reported a cache hit rate of 0.94 and finished in 48.8 seconds. With --disable-radix-cache, the same run took 129.7 seconds, and the mean first-token latency rose from 1.3 seconds to 7.1 seconds.

The first three rows show the other half of the story. Output throughput stopped growing after 8 concurrent requests, and the time to first token grew five times between 8 and 32. Past that point, more concurrency only adds waiting. The last row is the subject of the next step.

Step 6: Decide how much memory becomes KV cache

Every number in the last two steps depends on the pool size, so look at where it comes from. Start with the cost of one token. Take a model with grouped-query attention. One token costs the KV heads per GPU, times the key plus value width, times the layers, times the bytes per value.

Qwen3-0.6B has 28 layers, 8 KV heads, and a head width of 128. In 16-bit precision that is 8 times 256 times 28 times 2, or 114,688 bytes per token. The MLX log printed the same number as bytes_per_slot=114688. Llama-3.1-8B costs 128 KiB per token by the same formula, so a 100,000-token conversation holds 12.5 GiB. Models with multi-head latent attention, such as DeepSeek-V3, store one compressed latent per token. That is about 68.6 KiB, but every tensor-parallel rank keeps a full copy.

The flag that sets the pool is --mem-fraction-static. Its docstring defines it as model weights plus KV pool, divided by device memory. It is not the share of memory the server uses.

Activations, CUDA graph buffers, and the grammar and sampling buffers live outside it. On CUDA, SGLang picks a default from a reserve estimate. The estimate is 512 MB, plus 1.5 MB per token of chunk size, plus room for parallelism and CUDA graphs. GPUs above 60 GB keep at least 10 GB in reserve.

The MLX path on a Mac sizes the pool with its own formula in _compute_pool_size. The budget is the smaller of two values. One is the Metal working-set limit times the fraction, minus memory MLX already holds. The other is the system memory available at startup times the fraction. The pool size in tokens is the budget over the per-token cost.

That second value is where the Mac run failed. The resolved default fraction on this Mac was 0.872. System memory available at startup was 27.8, 38.3, and 49.1 GB on three starts, depending on what else was open. At 38.3 GB the pool took 33.4 GB. At 32 concurrent requests, Metal reported Insufficient Memory, and every SGLang process exited, not only the request that hit the limit. With --mem-fraction-static 0.4, every request finished.

The same rule applies on GPUs. The default assumes the device belongs to SGLang. On a shared machine, or next to another process on the same GPU, set the fraction yourself. Then read the startup log line, which prints max_total_num_tokens, chunked_prefill_size, max_running_requests, context_len, and available_gpu_mem. The tuning doc suggests 5 to 8 GB of free memory for activations on a GPU. Raise the fraction when you see more, and lower it when you see less.

The KV dtype is the other lever. With --kv-cache-dtype fp8_e4m3, each token costs half as many bytes, so the same memory holds twice the tokens. It needs an attention backend that reads FP8 without a separate dequantize pass. It also needs scales from the checkpoint, because the help text warns that a missing scale defaults to 1.0 and can cost accuracy.

On a small GPU the dtype decides how many requests run at all. We measured Qwen3-8B on the L4 in two forms, three seeds per point, with 512 input and 128 output tokens per request. With BF16 weights and KV, the pool held 24,094 tokens. With FP8 weights and an FP8 KV cache, it held 142,369, which is 5.9 times as many.

At 64 concurrent requests, BF16 output stopped at 251 tokens per second, and FP8 reached 736. The BF16 log showed the cause. The server ran at most 38 requests, with token usage at 0.99 and 26 requests in the queue. FP8 ran all 64 at token usage 0.29 with an empty queue.

FP8 was also faster per token at one request, 39.66 ms against 60.16 ms. Decode on an L4 reads every weight for each token, and FP8 weights are half the bytes. We also ran an open-loop sweep with a target of TTFT under 1 s and TPOT under 100 ms. FP8 served 3.18 requests per second inside the target at 4 offered per second. The best BF16 point served 0.96.

Step 7: Extend the cache to host memory and storage

GPU memory fills fast with long sessions. A coding agent that has run for an hour can hold a context of 100,000 tokens. For Qwen3-0.6B that is 11 GB of KV for one session, and for larger models it is many times more. Once the GPU pool has no free slots, LRU eviction drops old prefixes, and the next turn of that session must compute them again.

HiCache adds two more tiers under the GPU pool. Turn it on with --enable-hierarchical-cache. The second tier is pinned host memory, sized by --hicache-ratio, which defaults to 2.0 times the GPU pool. The third tier is optional storage, set by --hicache-storage-backend, with choices that include file, mooncake, hf3fs, and nixl.

The radix tree records where each node's KV lives. A node can be on the GPU, in host memory, or in both. The controller class is HybridCacheController, and the host pool for standard attention is MHATokenToKVPoolHost.

Writes go down the tiers under a write policy, set with --hicache-write-policy. Under write_through, the default, a node is copied to host memory when its hit count reaches 1. Under write_through_selective, the threshold is 2, so only reused nodes are copied. Under write_back, a node is copied only when it is evicted from the GPU.

Reads come back up. When a request matches a prefix that is only in host memory, the controller copies it back to fresh GPU slots. It does this layer by layer while the forward pass runs, so the copy for layer 2 overlaps the compute of layer 1. The default IO backend, kernel, uses GPU copy kernels for this.

Storage reads are prefetches, and they follow a policy set with --hicache-storage-prefetch-policy. Under best_effort, the prefetch stops as soon as prefill can run. Under wait_complete, prefill waits for all of it. Under timeout, the default, the prefetch stops after 1 second plus 0.25 seconds per 1,024 tokens. A storage hit shorter than 256 tokens is not prefetched at all, because recomputing it is cheaper.

The figure plays an agent workload through the three tiers. Six agents share one 2,048-token system prompt, and each turn adds 3,000 tokens of tool output and reply. Each node is real KV at 114,688 bytes per token. Change the tier sizes, the write policy, and the order in which agents speak.

Play the default, then make the GPU pool smaller. With host memory at ratio 0, every eviction means recompute. With ratio 2.0, most evicted prefixes come back from host memory, and the cost turns into bytes over the PCIe link. Switch to one agent at a time and the GPU alone holds almost everything, because each session finishes before the next one starts.

The write policy changes where the bytes go. write_through copies every new node down at once. Evictions then cost nothing, but the link carries every node. write_back copies only at eviction. It saves link traffic for nodes that stay on the GPU. write_through_selective sits between them.

The SGLang team's HiCache post claims up to 6 times higher throughput and up to 80% lower time to first token on multi-turn workloads. Those are team numbers. User reports also describe deadlocks and wrong answers from host-tier hits on hybrid models. Test HiCache on your own model before you rely on it.

Step 8: Admit requests on a forecast, retract when it fails

When the scheduler admits a request, it does not know how long the answer will be. It only knows max_new_tokens. If it reserved the full maximum for every running request, the pool would sit mostly empty, because most answers stop early. If it reserved nothing, a long answer could fill the pool in the middle of a decode step.

SGLang takes a middle path and keeps a number called new_token_ratio. The PrefillAdder charges each running request its remaining max_new_tokens times that ratio. A request that asks to join must fit its whole input plus its full remaining max_new_tokens in what is left. The ratio starts at 0.7 and falls by a fixed amount on each decode step. After 600 steps it reaches 0.098, so the scheduler grows bolder while nothing goes wrong.

One flag turns the guess off. A request with ignore_eos always runs to its limit, so the scheduler charges it at a ratio of 1.0 and the reservation is exact. Benchmarks set ignore_eos by default. That is why the small-pool run in the table never retracted. Each request needed 256 input tokens plus 512 reserved tokens, so only 5 fit in a 4,096-token pool. The log showed #running-req: 5 for most of the run, the queue reached 15, and the median time to first token was almost 28 seconds.

When the guess is wrong, decode runs out of slots. Before each decode step, check_decode_mem checks that the pool has a free slot for the next token of each running request. If not, retract_decode sorts the running requests by output length, then by longer input, and removes requests from the end of that order. The requests with the least progress go first. Their KV goes back to the pool, and they return to the waiting queue to prefill again later.

After a retraction the ratio jumps. The new value is the decoded tokens plus 20 per remaining request, divided by the sum of their max_new_tokens, capped at 1.0. The log line reads KV cache pool is full. Retract requests. with the old and new ratio. A retracted prompt often still sits in the tree, so its second prefill is cheaper than the first.

Play both modes at the same pool size. With ignore_eos the queue is long and nothing is wasted. With natural stops and many long answers, more requests start early, then some of them lose their work. The forecast trades queue time for the risk of recompute.

The project's tuning doc says one retraction a minute is acceptable. Frequent retraction is a configuration problem. Raise --schedule-conservativeness, which scales the starting ratio, or lower --max-running-requests. One user report on the project tracker describes a retraction after which decode dropped from roughly 429 tokens per second to less than 40. It never recovered. Treat the retraction log line as a performance alarm.

Priority adds a second way to remove a running request. With --enable-priority-scheduling, a new request can push out running ones if its priority beats theirs by 10 or more, the default threshold. With --retraction-policy priority, retraction also removes low-priority requests first.

Step 9: Keep the GPU busy between steps

Each scheduler loop does CPU work: read new requests, sort the queue, match prefixes, allocate slots, and process the last step's tokens. If the GPU waits for all of that, it sits idle for part of every step. With a small model, the idle part can be a large share.

The overlap scheduler, event_loop_overlap, runs one batch ahead. The GPU starts on batch N+1 first. The CPU then handles the output of batch N during that forward pass. Batch N+1 is built before the tokens of batch N exist on the CPU. So its input ids are placeholders, and a FutureMap fills them on the GPU from batch N's output. Since late 2025 this uses two CUDA streams instead of a helper thread.

The SGLang team reported a 1.1 times throughput gain from this change in its v0.4 release post of December 2024. Its profiles showed no idle GPU gaps across five decode batches. Overlap is on by default. The code turns it off where it cannot work, such as the Torch MPS path without MLX and speculative decoding on CPU.

Turning it off is for debugging, and on the Mac it did not even work. With --disable-overlap-schedule, the MLX server crashed on the first prefill that had a cache hit. The error was Expected all tensors to be on the same device, but found at least two devices, mps:0 and cpu. On MLX the default loop is a separate overlap version built on mx.async_eval. The plain loop is the path that broke.

One side effect of running ahead is in flight tokens. A request that finishes on its length limit in batch N is still part of batch N+1 when that batch is planned. The v0.5.22 scheduler tracks those requests and leaves them out of the next decode batch. The same code avoids a retraction when finishing requests are about to free KV. That case is behind crash reports on earlier releases.

Step 10: Split long prompts into chunks

A prompt of 30,000 tokens is one large forward pass. While it runs, every other user waits for their next token. Chunked prefill splits the prompt and runs one chunk per step. The other requests decode between chunks.

The PrefillAdder handles the split. If a request's remaining input is larger than the chunk budget, only the first chunk runs now. The request keeps its place as the chunked request, and the next prefill step continues it before anything else. Its finished chunks go into the tree as they complete, so another request with the same prefix can reuse them.

The default --chunked-prefill-size depends on GPU memory. It is 2,048 under 20 GB, 4,096 up to 60 GB, 8,192 up to 160 GB, and 16,384 above. The Mac run resolved it to 4,096. A separate budget, --max-prefill-tokens, caps one prefill batch at 16,384 tokens by default.

By default a step is either prefill or decode, so decode still pauses during each chunk. The flag --enable-mixed-chunk puts the decode rows into the same forward pass as the chunk. That smooths inter-token latency for other users, at the cost of a slower chunk. It is off by default, and it turns itself off when a request asks for log-probabilities.

A smaller chunk protects inter-token latency for running users and costs the long prompt more steps. A larger chunk finishes the long prompt sooner. If long prompts cause out-of-memory errors in prefill, lower the chunk size to 4,096 or 2,048 before anything else.

Step 11: Constrain every token with a grammar

Structured output in SGLang means the model cannot sample a token that breaks the format. A request asks for json_schema, regex, ebnf, or structural_tag. The default backend is xgrammar. Outlines and llguidance are the other choices.

Grammar compilation takes time, so it happens off the main path. The GrammarManager keys each grammar by its type and text and checks a cache. On a miss, the request waits in a grammar queue while a thread pool compiles it. Across tensor-parallel ranks, the scheduler admits a request only after all ranks finish compiling its grammar.

Each step, the grammar fills a bitmask over the whole vocabulary. Before sampling, ModelRunner applies that mask to the logits, so forbidden tokens have zero probability. After sampling, the grammar matcher accepts the token and moves its state. Under the overlap loop, the CPU builds the following mask during the forward pass, and sampling waits for it.

The idea behind fast structured output came early. In February 2024 the team described a compressed finite state machine. Some parts of a schema allow only one path, such as the text {"name":. There the engine can add the whole path in one step, not one token per step. The post claimed up to 2 times lower latency and 2.5 times higher throughput on JSON decoding.

The paper described a jump-forward mode that writes forced runs of text without a forward pass. The methods still exist on the grammar objects at v0.5.22, but nothing in the runtime calls them. Do not plan around jump-forward on this release.

Step 12: Guess several tokens, check them in one pass

Decode is slow because each step produces one token per request and reads all of the weights. Speculative decoding lets a small draft model guess a few tokens. The large target model then checks all of them in one forward pass. Every accepted guess is a step saved.

SGLang's main family is EAGLE, and DeepSeek's multi-token prediction runs as EAGLE with the draft layer inside the target checkpoint. One step in the v2 worker has three parts. The draft runs speculative_num_steps times and keeps topk candidates at each level, which grows a small tree. The target scores the best num_draft_tokens nodes of that tree in one pass, in a mode called TARGET_VERIFY. Last, the draft model runs over the accepted tokens to update its own cache and hidden state.

The gain is the acceptance length, the tokens committed per verify step, divided by the cost of drafting plus verifying. The defaults show the trade. Llama-family models get 5 steps, top-4, and 8 draft tokens. DeepSeek, GLM, and most others get 3 steps, top-1, and 4 draft tokens. When speculative decoding is on and you did not set --max-running-requests, SGLang sets it to 48, because the method wins at small batches.

Speculative decoding also uses KV slots for the draft tokens. One user report measured an agent replay with EAGLE on. The prefix hit rate fell from 97% to between 40% and 53%, and input throughput fell by more than three times. The draft reserve had pushed cached prefixes out of the pool. If your traffic depends on prefix reuse, measure speculative decoding and the radix cache together.

We measured both sides of the trade with EAGLE-3, three seeds per point. The first model was the mixture-of-experts Qwen3-30B-A3B-Instruct-2507 on a DGX Spark (GB10). The median time per output token was 1.60 times shorter at one request and 1.89 times shorter at 32. The acceptance length stayed between 2.71 and 2.74 tokens per step. Our reading of the growing gain is this. A mixture-of-experts step reads only the experts its tokens chose. The verify pass reuses those weights for the draft positions.

The dense Qwen3-8B on the L4 gave a different answer. Each token still came about twice as fast, 25.6 ms against 60.2 ms at one request. But the draft model took memory, and the KV pool fell from 24,094 to 9,951 tokens. Output was 2.37 times the baseline at one request, 1.00 times at 32, and 0.98 times at 64.

At 32 requests the median time to first token was 8,040 ms with EAGLE-3 and 2,970 ms without it. The requests waited for pool space. Speculative decoding spends memory that the batch needs. On a small card, measure it at the concurrency you serve.

Step 13: Spread one model across many GPUs

Everything so far ran in one world of GPUs. In SGLang, one model copy is a torch.distributed world of tensor parallel size times pipeline parallel size. Attention data parallelism, context parallelism, and expert parallelism are different ways to split that same set of GPUs for different layers. They do not add GPUs on top of it.

Tensor parallelism splits each layer's matrices across GPUs. Every rank runs the same scheduler loop on the same batch, which is why rank 0 broadcasts requests. For grouped-query models, more tensor-parallel ranks also split the KV heads, so each rank stores less per token.

Multi-head latent attention breaks that rule. DeepSeek-V3 has one latent per token, so tensor parallelism of 8 stores the same KV 8 times. Attention data parallelism fixes this. Each attention rank holds a separate set of requests with their KV, while the expert layers stay split. At v0.5.22 the flag is --attn-dp-size. The older --enable-dp-attention still works, is rewritten to the new flag, and is set to be removed after 2026.

Mixture-of-experts models add expert parallelism, where each GPU holds some experts and tokens travel to them by all-to-all. --moe-a2a-backend picks the transport, such as DeepEP. In its automatic mode, DeepEP uses a high-throughput mode for prefill and a low-latency mode, which works with CUDA graphs, for decode. EPLB moves hot experts between GPUs every 1,000 forward passes by default.

Prefill and decode want different hardware. Prefill is compute-bound on long inputs, and decode is memory-bound on many short steps. Prefill-decode disaggregation runs them on separate servers with --disaggregation-mode prefill and --disaggregation-mode decode.

The router chooses one server of each role and sends the request to both of them, with the same random 63-bit bootstrap_room id. First the decode server allocates KV slots and tells the prefill server where they are. The prefill server runs the forward pass and writes the KV straight into those slots with a one-sided transfer, by default through Mooncake. The decode server then starts from a prebuilt batch that skips prefill.

The transfer is also where a split fails without an error. Our two L4 cards have no RDMA, so we tried the split with --disaggregation-transfer-backend mooncake_tcp. Prefill ran on the first card and decode on the second, which also ran another team's pods. The decode pool held 2,925 tokens against 24,094 on prefill.

Decode could run only two requests at a time, and 19 more waited for decode memory. Prefill kept producing KV for them, and the TCP transfer queue filled. The log printed TCP lane queue-full rejection count: 4096. About three minutes into the sweep, the transfer session failed, and it did not come back.

Every later request still returned HTTP 200 through the router, with no text. The benchmark reported 240 successful requests out of 240, a mean TTFT of 0.00 ms, and zero retokenized output tokens. Its throughput line looked better than the FP8 server, so that number is not in this guide. Read Total generated tokens (retokenized) in every disaggregated benchmark before you read its throughput.

Those pieces are what the 96-GPU DeepSeek deployment combined. Prefill and decode ran on separate nodes. Attention ran data-parallel to stop KV duplication for MLA. Experts were spread with DeepEP, with GEMMs from DeepGEMM and two-batch overlap to hide the all-to-all time. The team reported up to 5 times the output throughput of plain tensor parallelism on the same hardware. That is the team's claim from its own published setup.

Requests for all of these servers arrive at sgl-model-gateway, a separate Rust binary. Its default policy is cache_aware. It keeps an approximate radix tree over the raw request text, without a tokenizer. Each node records the workers that served that prefix.

When loads are balanced and more than 30% of a request's text matches a worker's prefixes, the request goes to that worker. Loads count as uneven when the longest queue is 64 requests longer than the shortest and 1.5 times as long. In that case the least loaded worker gets the request. The tree is a guess built from routing history, and it can name a worker that already evicted the prefix.

The tuning doc prefers several replicas behind the gateway to --dp-size inside one server. The in-process data-parallel controller uses round robin by default, and it does not know which replica holds which prefix.

Round robin also does not know how big each replica is. On our L4 pair, the second card held two pods for another team, at 1,746 MiB each. With --dp-size 2, the copy on that card had 3.17 GB free after the weights. Its pool held 2,925 tokens, an eighth of the other copy. The controller still sent it every second request.

At 64 concurrent requests, 63 of them sat on the small copy, 4 running and 59 waiting. The large copy used 2 to 7 percent of its pool. Two copies produced 42 tokens per second, and one L4 alone produced 251. The p99 time to first token was 391 s against 33 s. In the open-loop sweep, two copies served 0.21 requests per second inside the latency target, and one card served 0.96.

--tp-size 2 refused the same pair at startup with The memory capacity is unbalanced. Some GPUs may be occupied by other processes. Every tensor-parallel rank holds the same share, so the smallest card sets the size for all of them, and the check stops the start. None of this says that SGLang is slow. It shows what happens when two parts of a deployment are not the same size. The flag --load-balance-method total_tokens sends each request to the copy with the fewest tokens. We have not measured it yet.

Step 14: Tune a server in a fixed order

With the parts in place, the order of tuning follows from them. Start with a launch line you can read. This one is for one node with two GPUs and a shared-prefix workload, and every flag on it exists in the v0.5.22 help:

sglang serve --model-path Qwen/Qwen3-8B --tp-size 2 \
  --mem-fraction-static 0.85 --chunked-prefill-size 4096 \
  --max-running-requests 256 --cuda-graph-max-bs 256 \
  --schedule-policy lpm --enable-cache-report --enable-metrics

Then work through the gauges in this order. Each one depends on the one before it.

1. Read the startup line. Check max_total_num_tokens and available_gpu_mem, and move the fraction until 5 to 8 GB stay free.
2. Match the concurrency limits. The default max_running_requests comes from KV capacity with a floor of 2,048, while the largest decode CUDA graph comes from a GPU memory table. A user on an L40 measured 4,096 against 32, and decode fell to eager mode past 32 requests. Set both to your target.
3. Load the server and read the decode lines. The tuning doc aims for token usage above 0.9 with #queue-req between 100 and 2,000 for offline work. Low usage with a queue means the forecast is too careful. Frequent retraction means it is too bold.
4. Balance first-token latency against inter-token latency with the chunk size and --enable-mixed-chunk.
5. Fix prompt order for prefix reuse, then try lpm if many requests share prefixes and the queue stays under 128.
6. Scale out behind the gateway with cache_aware routing.

A few failures repeat often enough to name. A KV pool that fills during decode used to retract in a loop and hang. At v0.5.22, the last request that still cannot fit is aborted with HTTP 500 instead. Hybrid models with Mamba or sliding-window layers can cap concurrency below your setting and only say so in a log line. The /metrics endpoint, turned on with --enable-metrics, exposes sglang:cache_hit_rate, sglang:token_usage, and sglang:num_retracted_reqs, and those three catch most of these problems early.

On a Mac, use the MLX path with SGLANG_USE_MLX=1, set the fraction yourself, and keep the overlap loop on. The Torch MPS path without MLX turns overlap off and rejects quantized checkpoints. MLX runs the same Python scheduler, radix tree rules, and log lines as a GPU server. That makes a Mac a good place to learn the scheduler. It is not a place to measure GPU kernels, CUDA graphs, or attention backends.

Step 15: When inference becomes part of training

Reinforcement learning on language models needs rollouts, and rollouts are inference. Miles, an RL framework from RadixArk, runs real SGLang servers as its rollout engines and talks to them over HTTP. After each training step it pauses generation, sends new weights, and resumes. With trainer and server on the same GPUs, weights move as CUDA IPC handles through /update_weights_from_tensor. Otherwise they move by NCCL broadcast through /update_weights_from_distributed.

Read the version note before you copy a setup. Miles does not run on stock v0.5.22. It needs a patched SGLang from a sglang-miles branch on a v0.5.21 base, and some routes it calls do not exist upstream.

Two SGLang details matter more in RL than in serving. The log-probabilities SGLang returns for a sampled batch come from the temperature-scaled distribution, unless SGLANG_RETURN_ORIGINAL_LOGPROB=1 is set. Those values enter the policy ratio of the loss. The flag --enable-deterministic-inference swaps several matrix and reduction kernels for batch-invariant versions. Then a rollout and its replay agree.

Determinism needs that flag even in serving.

On the Mac, I sent one greedy prompt twice. The first run computed all 10 prompt tokens. The second run took 9 from the tree and computed 1. The first three output tokens matched, then the outputs split: " of the dog's tail" against " of the dog. The dog was tired". Different kernel shapes give slightly different float sums, and a near tie can flip.

When to choose SGLang

Why it is fast, in one place

Now the speed story can be told in order, one prompt at a time. Each mechanism removes a specific kind of waste, and each one appeared in a step above.

The process split removes waiting on Python's global lock. Tokenizing and detokenizing run in their own processes, so the scheduler can start a GPU step every few milliseconds. Rust pieces now take over the busiest CPU edges.

The radix tree removes repeated prefill. A prompt that shares a prefix with an earlier request computes only its new tokens. On the Mac, a repeated prompt needed 20 ms from scheduler entry to its first token. The first run needed 800 ms, partly for kernel warm-up.

The shared pool removes reserved but unused memory. Cached prefixes and running requests use the same slots, so there is no fixed cache size to tune. The admission forecast then fills the pool on a guess about answer length, and retraction corrects the bad guesses.

HiCache removes recompute for long sessions that do not fit in GPU memory. It turns that recompute into copies from host memory or storage, which cost bytes on a link and not GPU time.

The overlap loop removes idle GPU time between steps. The CPU prepares batch N+1 during the forward pass of batch N.

Chunked prefill removes the stall that one long prompt causes for everyone else. Decode for other users keeps going between chunks.

CUDA graphs remove kernel launch time in decode. The paged attention kernels read scattered KV slots without copying them into one block first.

The grammar backend removes invalid output without a retry. The CPU builds each mask during the forward pass, in parallel with the GPU.

Speculative decoding removes some of the memory reads of decode. One verify pass checks several draft tokens and reads the weights once for all of them.

At scale, attention data parallelism removes duplicate MLA caches, expert parallelism spreads the experts, and disaggregation gives prefill and decode separate hardware. The cache-aware gateway keeps each conversation on the worker that already holds its prefix.

The GPU runs add one rule to that list. Every large gain we measured came from a mechanism that removes repeated work. Examples are the prefill of a shared prompt and the bytes of each cached token. Every loss came from a feature that spends memory the batch needs, or from two parts of a deployment that are not the same size. None of the losses printed an error, and two of them looked like success in the benchmark summary.

None of these is unique to SGLang alone today. The combination, the defaults, and the speed of day-0 releases are what users name when they explain their choice.

Choose SGLang when your traffic shares prefixes. Agents, multi-turn chat, few-shot templates, and retrieval with a common instruction block all fit. The tree turns repeated context into cache hits. On the Mac, that meant a 45 ms first token instead of more than a second. A shared-prefix run took 38% of the time.

Choose it for DeepSeek-class mixture-of-experts models across many GPUs. Attention data parallelism, expert parallelism with DeepEP, EPLB, and prefill-decode disaggregation are first-class features with flags, metrics, and docs.

Be more careful with unique prompts and a single dense model. There, field reports find small differences between engines, and the deciding factors are model support, startup time, and your hardware. Spheron reported 1,920 output tokens per second for SGLang 0.5.9 and 1,850 for vLLM 0.18.0 at 50 users. The test used Llama 3.3 70B FP8 on one H100 with unique prompts. The often repeated 29 percent gap between the two engines has a weak base. The test behind it paired SGLang v0.2.3 with a release of the other engine about two years newer, as DeepInfra pointed out. For the other popular engine, the vLLM guide on this site walks through its block pool and scheduler in the same depth.

Whatever you choose, keep the habits from this guide. Put stable content first in every prompt. Read the startup line. Watch the retraction count. Set the memory fraction yourself on any machine you share.