The Inference Engine Underneath
The model gets the headlines. The serving engine gets the bill. Take one set of weights and one GPU, then route the same requests through two different engines, and throughput can move by a factor of two while your tail latency moves further. The weights did not change. The scheduler around them did. Five engines do this one job, and each makes a different bet about how to keep an expensive chip busy: vLLM, SGLang, TensorRT-LLM, LMDeploy, and TGI.
One job, five bets
An inference engine sits between an HTTP request and a matrix multiply. Its job is to take a stream of prompts, none of which arrive at the same time or run for the same length, and keep a GPU that costs a few dollars an hour as close to fully used as it can. Everything the five engines argue about reduces to that: how to pack uneven work onto a fixed machine, and how to avoid computing anything twice.
Three mechanisms do most of the work, so it helps to see each one move before naming who implements it best.
Keep the GPU full
The first mechanism is the floor every modern engine stands on. Old serving batched requests the way a bus leaves on a schedule: fill the seats, then wait for the slowest passenger before anyone moves again. A short request finishes early and its slot sits idle until the whole batch drains. Continuous batching, the technique vLLM made the default, evicts a finished sequence the moment it ends and admits a waiting one in the same step, so the batch never empties out. Toggle the two below and watch the idle gaps appear and vanish.
Never compute a prefix twice
The second mechanism is reuse. Most production prompts share a long head: the same system prompt, the same tool definitions, the same retrieved document in front of every question. The key and value tensors for that shared head are identical every time, so an engine that keeps them can skip recomputing them and start the new request partway through. vLLM does this with a hash over fixed KV blocks (automatic prefix caching). SGLang does it with RadixAttention, a tree keyed by the token sequence, which reuses a shared trunk even when requests branch at different points, and needs no tuning to do it. The figure shows the principle both share: compute the trunk once, and each branch pays only for its own tail.
Compile or load
The third mechanism is where the engines split hardest. TensorRT-LLM can compile a model into a kernel graph tuned for one GPU, one batch shape, and one sequence length. That build takes on the order of half an hour, and the result is the fastest tokens on the list. Its newer PyTorch backend, the default since the 1.0 release, skips the build and loads weights in a minute or two at a small throughput cost. The compiled path only pays off once you serve enough requests on that fixed configuration to amortize the build. Drag the slider to find the point where it does.
Shapes are illustrative. The build cost (about half an hour) and the eager cold start (a minute or two) are real; the exact crossover depends on your model, GPU, and how often you redeploy.
The numbers, with the asterisk they deserve
Head-to-head throughput numbers exist, and they are worth reading with discipline. The cleanest recent three-way set is from Spheron, run on one H100 SXM5 with Llama 3.3 70B in FP8, at fixed input and output lengths. Pick a concurrency level and read the shape, not the third digit.
Source: Spheron, H100 SXM5, Llama 3.3 70B FP8, unique prompts, 512 in / 256 out. vLLM v0.18.0, TensorRT-LLM v1.2.0, SGLang v0.5.9. A vendor benchmark at one version snapshot; some sub-tables in the source are labeled illustrative. Directional, not law. All cross-engine numbers move with every release.
The shape holds up across the concurrency range: on a fixed model, TensorRT-LLM leads throughput and time-to-first-token by roughly 8 to 13 percent, and vLLM and SGLang sit within a few percent of each other on unique prompts. The picture changes when prompts share a prefix. On an 80 percent shared prefix, Spheron's two-engine run put SGLang's median time-to-first-token about 37 percent below vLLM's, though turning on vLLM's own prefix caching closes most of that gap on a flat shared prompt. Measure your prefix overlap before you trust any single ranking.
The five, in one line each
vLLM is the safe default. Apache-2.0, born at UC Berkeley, now the community standard and the engine under Red Hat's llm-d and NVIDIA Dynamo. Its 2025 V1 rewrite folded prefix caching, chunked prefill, and speculative decoding into one scheduler and made prefix caching free at zero hit rate. Broadest model and hardware coverage, simplest to deploy, hard to regret. (V1 notes.)
SGLang is the prefix specialist. Apache-2.0, from the LMSYS community, with RadixAttention on by default, aggressive grammar-cache reuse for repeated JSON schemas, strong prefill-decode disaggregation, and day-zero support for new DeepSeek and MoE models. Pick it when your traffic reuses context heavily. (DeepSeek-V4 day-0.)
TensorRT-LLM is peak throughput on NVIDIA, at the price of a compile. Apache-2.0, and as of March 2025 fully developed in the open. The 1.0 release promoted the PyTorch backend to the default, so you can start eager and compile later. NVIDIA-only, and the base of NVIDIA NIM. (release notes.)
LMDeploy is the quantization workhorse. Apache-2.0, from the InternLM team, built on the TurboMind C++/CUDA engine. Its signature is weight-only 4-bit serving (AWQ and GPTQ) with the project reporting 4-bit inference around 2.4 times faster than FP16, plus online INT8 and INT4 KV-cache quantization. Reach for it when memory is tight. (repo.)
TGI, Hugging Face's serving toolkit, is the one to know about rather than reach for. It pioneered chunked prefill and long-prompt prefix caching in its v3 line and is back on Apache-2.0. It also entered maintenance mode on December 11, 2025, taking only minor bug fixes while Hugging Face steers serving toward vLLM. For a new build in 2026, start elsewhere.
The same techniques, side by side
| Technique | vLLM | SGLang | TensorRT-LLM | LMDeploy | TGI |
|---|---|---|---|---|---|
| Continuous batching | yes | yes | yes | yes | yes |
| Prefix caching | hash (APC) | radix tree | KV reuse | yes | v3 long-prompt |
| Chunked prefill | yes | yes | yes | split & fuse | yes |
| Speculative decoding | Eagle3 | V2 + DSpark | yes, MTP | partial | limited |
| Prefill-decode split | via llm-d | strong | via Dynamo | no | no |
| 4-bit AWQ / GPTQ | yes | yes | yes | signature | yes |
| FP8 / FP4 | yes | yes | yes | MXFP4 | FP8 |
| Structured output | xgrammar | xgrammar+cache | xgrammar | yes | outlines |
| Non-NVIDIA hardware | AMD, TPU, Intel | AMD | NVIDIA only | NVIDIA | AMD, Gaudi |
| License | Apache-2.0 | Apache-2.0 | Apache-2.0 | Apache-2.0 | Apache-2.0 |
| Status | active | active | active | active | maintenance |
Latest at time of writing: vLLM v0.26.0, SGLang v0.5.16 (Jul 24, 2026), TensorRT-LLM v1.2.0, LMDeploy v0.15.0 (Jul 31, 2026), TGI v3.3.7. Support cells are the current focus, not a claim of parity.
Which one for which job
- General production, ship fast, might change hardware. vLLM. Broadest coverage, simplest deploy, and the substrate the orchestration layers build on.
- Agents and RAG with heavy shared context, repeated schemas, new MoE models. SGLang. Measure your prefix overlap first; it pulls ahead as overlap climbs.
- One model, NVIDIA fleet, throughput is the whole game. TensorRT-LLM. Pay the compile, or start on its PyTorch backend and compile once the shape settles.
- Memory-constrained serving, 4-bit weights, InternLM or Qwen or DeepSeek. LMDeploy and TurboMind.
- Already on Hugging Face Inference Endpoints. TGI still works, but it is in maintenance mode. Plan the move to vLLM or SGLang for anything new.
Two things sit above all five and are worth a footnote. NVIDIA Dynamo and Red Hat's llm-d are not engines; they orchestrate a fleet of these engines across nodes with KV-aware routing and prefill-decode disaggregation. And for a single laptop rather than a datacenter, the pick is a different family, llama.cpp or Ollama, which this post leaves aside on purpose.
The honest part
Every number here is a snapshot. The V1 rewrite, the FlashAttention 4 kernels, the DeepSeek sparse-attention path, and the compiled-versus-eager default all landed inside the last eighteen months, and each reshuffled the ranking. Read a benchmark for its date and its versions, reproduce it on your own traffic, and remember that the biggest lever is often the one nobody benchmarks: how much of your prompt repeats. Pick the engine that reuses the most of your particular work, and the throughput number tends to follow.