Engine internals guide · August 7, 2026 · 20 min read

Inside the vLLM Engine

One GPU, hundreds of live conversations, and none of them waiting on each other. The engine that made this normal was not built around a faster matrix multiply. It was built around a memory system. vLLM sits at 88,000 GitHub stars, 5.4 million monthly installs, and a place in the PyTorch Foundation, and every one of those numbers traces back to a single 2023 observation about wasted KV cache. This guide reads the machine itself: the block pool, the scheduler with no phases, the loop split across two processes, the bets it places on tokens that do not exist yet, and the machinery that runs before you configure anything. Six figures you can step through, every mechanism verified against the V1 source and design docs as of v0.26.0.

A memory problem wearing a throughput costume

Start with what the cache actually is. A transformer generates one token at a time, and every new token attends over all previous ones. Recomputing the keys and values for the whole history on every step would make each new token cost more than the one before, so every serving system caches them: for each token, each layer, each attention head, a key vector and a value vector. That is the KV cache, and it grows with every token of every live request. For a 70B-class model a single long conversation can hold gigabytes of it. The model's weights are a fixed cost; the KV cache is the variable one, and it is the thing a serving engine actually manages.

Before vLLM, systems stored each request's cache in one contiguous slab, pre-allocated at the maximum possible length. A request that might reach 2,048 tokens reserved room for 2,048, then used 300. The reservation could not be shared, the slabs came in different sizes so the gaps between them were unusable, and both problems compounded as requests churned. The PagedAttention paper (Kwon, Li, and colleagues, SOSP 2023) measured the damage: in the systems of the day, only 20.4 to 38.2 percent of KV cache memory held actual token state. The rest was reservation and fragmentation. The GPU looked compute-bound. It was actually hoarding.

The fix came from operating systems, and it is the same fix your laptop applies to RAM: paging. Cut the cache into fixed blocks, 16 tokens each by default, place a request's blocks anywhere in GPU memory, and keep a per-request block table that maps logical position to physical block. Allocation becomes trivial: a request needing 56 tokens gets ceil(56/16) = 4 blocks, wherever they happen to be free. No pre-allocation, no contiguity requirement, near-zero waste. The paper reported 2 to 4 times the throughput of the state of the art at the same latency, and the June 2023 launch post mentioned, almost in passing, that the system had been serving LMSYS's Chatbot Arena since April on half the GPUs it previously needed.

What the current V1 engine builds on top of that idea is the part worth walking through slowly, because the block is no longer only a unit of allocation. It is a unit of identity. Each full block gets a hash of its contents chained with its parent's hash, and that turns the pool into a content-addressed store: any new request whose prompt starts with the same tokens can claim the same physical blocks. Step through what one small pool does under real traffic.

The pool, the queue, and the hash

Three details in that figure carry most of vLLM's cleverness, and each is worth naming precisely.

Freeing is not forgetting. When a request finishes, its blocks join the tail of the free queue but keep their hashes, so a later request with the same prefix can rescue them before they are recycled. The design doc calls this lazy invalidation: a cached block dies only at the moment its memory is actually handed to someone else. Until then the hash map still points at it, and a lookup can pull it straight back out of the free queue. The free queue itself is a doubly linked list of KVCacheBlock structs, each carrying a block_id, a block_hash, a reference count, and its two queue neighbors.

The queue is ordered for reuse. Freed blocks re-enter in reverse order, so the front of a prompt, the part most likely to be shared with a future request, survives longer than its tail. Eviction pops from the head, which makes the whole queue an LRU list with no extra bookkeeping and no separate eviction policy to tune.

The hash is more than the tokens. A block's hash chains three things: the parent block's hash, the tokens in the block, and a set of extra keys, the LoRA adapter ID if one is active, hashes of any multimodal inputs, and a per-request cache salt when isolation matters. Two requests running different LoRA adapters over identical text will never collide, and a tenant that must not share cache with another can salt itself into its own hash space. The default algorithm is SHA-256; a faster non-cryptographic xxhash variant is a flag away.

All of this ships enabled. The V1 team measured under 1 percent throughput cost even at a zero percent cache hit rate, so prefix caching is on by default; since agent and RAG traffic repeats system prompts and tool definitions constantly, real hit rates are anything but zero. And the same manager now handles models whose layers do not all cache alike: sliding-window layers, Mamba state-space layers, and full-attention layers get grouped by type, coordinated over one physical pool, so a Gemma-class or Jamba-class hybrid gets paged memory without special cases. I compared how vLLM's hash approach differs from SGLang's radix tree in the engine comparison essay; this guide stays inside one engine.

A scheduler with no phases

Classic serving engines treated a request as two-phase: a prefill phase that digests the prompt in one big parallel pass, then a decode phase that emits one token at a time. The phases have opposite personalities. Prefill is compute-bound, thousands of tokens multiplying through the weights at once; decode is memory-bandwidth-bound, the whole model streamed from VRAM to produce a single token. V0-era schedulers wrote special cases for both, and the special cases fought: a long prompt entering the batch could stall every running conversation while it prefilled.

The V1 rewrite, announced in January 2025, deleted the distinction. There is one number per request, how many of its tokens have been processed so far, and one decision per step: how many more tokens does each request get. The scheduler's entire output is a dictionary, {request_id: n_tokens}, filled under a fixed token budget, max_num_batched_tokens. A request whose processed count is still inside its prompt is prefilling; one past it is decoding; the scheduler does not care.

Everything that used to be a named feature falls out of that dictionary. Chunked prefill is no longer a mode; a 12,000-token prompt simply cannot exceed the per-step budget, so it gets sliced across steps automatically. Mixing decodes and prefills in one batch is no longer special; decodes are scheduled first, one token each, and whatever budget remains goes to prompt work, so short chats keep streaming while a heavy document grinds through beside them. Run the three steps below and read the dictionary each time.

Step 3 is the quiet punchline. Once every request is decoding, the batch uses five tokens of an 8,192-token budget, and the scheduler is not the constraint anymore. Decode throughput is bounded by how fast the GPU can stream weights and KV blocks from memory, which is a memory-bandwidth story this site has told before. The budget exists for the prefill spikes, and the dictionary makes the two workloads coexist without either starving the other.

Two more scheduler behaviors matter in production. Requests wait in a queue that is first-come-first-served by default, with a priority policy available. And when the block pool runs out entirely, V1 preempts the most recently scheduled requests, frees their blocks, and later recomputes them from scratch. Recompute-only preemption sounds wasteful, but the old alternative, swapping KV blocks out to CPU memory and back, cost more than redoing the arithmetic; V1 removed swap entirely, and the freed requests often win the recompute back through the prefix cache anyway.

Two processes, one heartbeat

The second thing V1 changed is where the loop runs. In V0, the Python that tokenized your request, the scheduler, and the CUDA calls all contended inside one process, and the GPU sat idle whenever Python was busy being Python. V1 splits the engine into processes with ZeroMQ sockets between them:

• The API server process owns everything request-shaped: HTTP handling, tokenization, multimodal preprocessing, detokenization, streaming. The class is AsyncLLM, and it delegates to an InputProcessor (raw request in, EngineCoreRequest out) and an OutputProcessor (token IDs in, text deltas out).
• The EngineCore process owns exactly two things, the scheduler and model execution, and runs them in run_busy_loop(): poll the input queue, schedule a step, execute it, hand the outputs to a dedicated output thread, repeat. Nothing else lives there.
• One worker process per GPU holds the model shards and runs the forward pass; a coordinator process appears once data parallelism enters. The architecture overview gives the process count as a formula: API servers, plus engine cores, plus GPU workers, plus one coordinator when replicated.

The point of the split is overlap. While the GPU executes step k, the API process is tokenizing the request that will join step k+1 and detokenizing the output of step k-1. CPU work still happens; it just stops happening between GPU steps. Toggle the two layouts.

The full path of one request

With the processes named, the whole journey of a curl to an OpenAI-compatible /v1/chat/completions endpoint fits in eight steps:

1. The API server process accepts the HTTP request (vllm/entrypoints/openai/api_server.py).
2. The InputProcessor tokenizes the prompt and loads any multimodal inputs, producing an EngineCoreRequest, off the event loop so a slow tokenization never blocks other requests.
3. AsyncLLM.add_request() ships the request over a ZMQ socket to the EngineCore process and returns a collector that the HTTP handler will stream from.
4. The busy loop admits it to the waiting queue; on some step soon after, the scheduler grants it tokens from the budget and the KV cache manager allocates blocks, skipping every block the prefix cache already holds.
5. The executor broadcasts the step to the GPU worker processes, whose model runner executes the compiled forward pass over one flattened batch containing every scheduled request's tokens.
6. Logits come back for each sequence; batch-level logits processors (temperature, penalties, grammar masks) run, then the sampler picks tokens. Sampling happens on the GPU, inside the EngineCore step, not in the API process.
7. The output thread pushes new token IDs back over ZMQ.
8. The OutputProcessor detokenizes incrementally and the handler streams SSE chunks to the client while the next GPU step is already running.

Lists like that stay abstract until you watch one. The figure below plays a single chat message against the engine events it triggers, both sides in lockstep. The left side is what the user sees. The right side is everything this guide has covered so far, in order, doing its job.

The loop in step 4 through 7 is the engine's heartbeat, and it is worth noticing what stays constant later: every scaling mechanism vLLM offers, more GPUs, more replicas, more nodes, keeps this exact loop intact and multiplies around it.

Betting on tokens that do not exist yet

Decode's one-token-per-step rhythm has an escape hatch. Each decode step reads the entire model from memory to produce a single token, but the same pass can verify several proposed tokens at once for nearly the same cost, because verification is one forward pass over all of them in parallel and the weights are being streamed anyway. So vLLM lets a cheap draft mechanism guess a few tokens ahead, then spends one pass of the real model checking the guesses. Every correct guess is a token you got without a full decode step. Every wrong guess costs nothing but the draft work: the verifier catches it, samples the correction from its own distribution, and rejection sampling makes the final output statistically identical to never having speculated at all.

The interesting part in 2026 is who writes the draft. A separate small model works but doubles your deployment surface, so vLLM's current roster leans on drafts that come from the target model itself or from the text: EAGLE reuses the target's own hidden states to propose ahead, MTP exploits models trained to predict several tokens natively (the docs recommend it when the target has native multi-token-prediction support), n-gram matching just looks for repeated spans in the context, startlingly effective when a model is quoting a document back at you, and suffix decoding extends that with dynamic speculation depth. A conventional draft model, a parallel-drafting variant called PARD, an MLP speculator, and a dynamic mode that adjusts speculation to fluctuating load round out the list. The rejection sampler that arbitrates all of them runs as a kernel inside the sampler, on the GPU, inside the same step loop from Figure 2.

Every token must pass the grammar

Structured output is the other place vLLM intervenes between logits and sampled token, and it earns a figure because the mechanism is prettier than its reputation. When you ask for JSON matching a schema, or a regex, or a choice from a list, vLLM compiles the constraint into an automaton (the xgrammar and guidance engines both do this; the default setting picks per request). The automaton tracks where the output stands inside the grammar, and each step it emits one bitmask over the entire vocabulary: a bit per token, allowed or forbidden. Forbidden tokens get their logits erased before sampling. The model cannot emit invalid output, not because anyone checks afterward, but because invalid tokens never had probability mass to begin with.

The cost model is what makes this production-grade: one bitmask per step, applied as plain arithmetic on the logits, no matter how complex the grammar is. The grammar compiles once (asynchronously, so a complex schema does not stall the batch while it builds), and agent frameworks that hammer the same tool-call schema get the compiled automaton from cache. If you are still passing the old guided_json family of parameters, they are deprecated aliases now; the current interface is a unified structured_outputs parameter covering choice, regex, JSON schema, EBNF grammar, and structural tags.

The compiler you did not invoke

Everything so far happens around the forward pass. The forward pass itself is also not what the model repository shipped. On startup, V1 runs the model through torch.compile by default, with one surgical exception: attention is wrapped as a custom op, torch.ops.vllm.unified_attention_with_output, so the compiler never traces inside it. The graph is then split at each attention call, and the pieces between attention are compiled by Inductor into fused kernels. Attention stays hand-written; everything else gets generated.

Those compiled pieces then become CUDA graphs, recordings of entire kernel sequences that replay with one launch instead of hundreds, erasing per-kernel launch overhead that decode steps otherwise pay every few milliseconds. The mode ladder runs from NONE through PIECEWISE and FULL to the default FULL_AND_PIECEWISE, which captures full graphs for the uniform batches decode produces and piecewise graphs for everything else. Each attention kernel declares how much CUDA-graph capture it tolerates, and the engine downgrades the mode automatically rather than crashing. Compilation artifacts cache on disk, and the design doc is explicit about the contract: all compilation finishes before the server takes its first request. Startup pays; serving does not.

Underneath the compiled graph sits a roster of attention kernels, selected per hardware generation: FlashAttention 2 through 4, FlashInfer in several variants, a Triton fallback, FlexAttention, plus a separate family for the latent-attention (MLA) models DeepSeek popularized. Since v0.26.0 the backend is chosen per KV-cache group, so one hybrid model can run different kernels for its full-attention and sliding-window layers. Quantization slots into the same machinery, FP8 and INT8 weight-activation schemes, 4-bit weight-only AWQ and GPTQ, FP8 KV cache, with the project steering newcomers toward the LLM Compressor toolchain. And hardware breadth is now a plugin story: NVIDIA, AMD ROCm, Intel, and CPU targets live in-tree, while TPU, Gaudi, and other accelerators attach as out-of-tree platform plugins.

Past one GPU

Scaling out never changes the loop; it multiplies it. Each layer of parallelism wraps the previous one behind the same interfaces:

• Tensor parallelism splits every weight matrix across GPUs in one node (--tensor-parallel-size); the workers execute the same step in lockstep and merge their partial results after every layer. Pipeline parallelism stacks layer ranges across nodes when one machine cannot hold the model. The executor broadcasts step inputs to workers over shared-memory message queues; the engine above it cannot tell one worker from eight.
• Data parallelism runs whole replicas of the engine, each with its own scheduler and KV cache, behind a coordinator process that tracks queue depth per replica and steers new requests toward the least loaded one. Three load-balancing layouts ship: one internal endpoint, external per-replica endpoints, or a hybrid. MoE models add a wrinkle, replicas must step in lockstep waves so their expert layers can exchange tokens, and expert parallelism shards the experts themselves across the fleet, with an expert-placement load balancer (EPLB) that migrates hot experts.
• Disaggregated prefill and decode, still marked experimental, runs prompt processing and token generation on separate machines entirely, because the two phases want different hardware economics. The bridge is a KV connector that streams cache blocks from the prefill fleet to the decode fleet: NIXL for fully async transfer, LMCache and Mooncake as cache layers, an offloading connector that spills to CPU memory, and a multi-connector that chains them. The v0.26.0 release notes describe the offloading side maturing into tiered storage, KV blocks spilling from GPU to CPU to object storage with metrics on every tier.

Notice what travels in that last bullet: the block from Figure 1. The 16-token page with a content hash is the unit of currency at every scale, allocated on one GPU, reference-counted across requests, and, at the far end, serialized over a network between clusters. One data structure, four orders of magnitude.

Reading the gauges

A guide to the engine should leave you able to read its instruments. Four numbers describe a serving deployment, and vLLM's own benchmark tooling (vllm bench latency, vllm bench throughput, vllm bench serve) reports them all: TTFT, time to first token, dominated by prefill and queueing; ITL, the gap between consecutive tokens, dominated by decode step time; TPOT, mean time per output token across the stream; and goodput, throughput that actually met your latency targets rather than throughput in the abstract. The tension between them is structural: batching more requests amortizes the weight streaming that dominates decode, so throughput climbs, but every added request makes each step slightly heavier, so ITL climbs with it. There is a knee where the GPU stops being memory-bound and starts being compute-bound; past it, throughput flattens and latency just gets worse. Finding that knee for your model, your GPU, and your service-level targets is what the benchmark harness is for, and the numbers only mean anything at your own prefix-hit rate and prompt-length mix. The bill those curves generate is its own essay.

Why this engine won

The adoption story reads like a flywheel with dates on it. February 2023, the repo is created at UC Berkeley's Sky Computing Lab. April 2023, it quietly starts serving Chatbot Arena. June 2023, the launch post; September, the SOSP paper. January 2025, the V1 rewrite lands as an opt-in alpha; by March it is the default, and by October V0 is deleted from the codebase entirely, a two-generation engine swap executed in nine months while serving production traffic. May 2025, the project joins the PyTorch Foundation at 46,500 stars; it has since nearly doubled that. January 2026, its core maintainers raise $150 million for Inferact, a company whose pitch is, in large part, keeping the open project healthy. Through all of it the release rhythm holds at roughly one minor version every two weeks, each carrying several hundred commits from contributor counts that read like conference attendance, 411 commits from 212 people in v0.26.0 alone.

The production names are load-bearing rather than decorative. LinkedIn runs more than fifty GenAI use cases on vLLM across thousands of hosts. Amazon's Rufus assistant served Prime Day on over 80,000 Inferentia and Trainium chips with vLLM in the serving path. a16z, announcing the Inferact round, put the fleet at 400,000 GPUs running vLLM concurrently, an investor's number worth the grain of salt that comes with it. Red Hat productized it, llm-d and NVIDIA Dynamo orchestrate fleets of it, Ray Serve and KServe embed it. And when a lab ships an open model, gpt-oss, Llama 4, DeepSeek, Kimi K3, the launch-day post that proves it runs is now routinely a vLLM post; day-zero support became the tax every model release pays.

None of that follows automatically from a good paper. It follows from the property the paper introduced and the V1 rewrite protected: the engine treats GPU memory as a first-class managed resource rather than a buffer, and every later feature, prefix reuse, phase-free scheduling, speculative verification, grammar masking, disaggregation, is a dividend on that one decision. The comparison essay asked which engine to pick; the honest summary of this one is that the block table is why the others get compared to vLLM.

Go deeper

This guide opens an engine-internals series; SGLang's radix-tree world and the TPU serving stack are natural next chapters. For vLLM itself, two sources beneath this piece deserve direct reading. The team's own design docs are unusually candid, down to flagging which documents are now historical. And Aleksa Gordić's Inside vLLM walks the same machine bottom-up from the source code with hand-worked examples; it is the best long read on the subject, covers the mid-2025 snapshot of V1, and this guide deliberately took the opposite route, memory first, so the two complement rather than repeat each other. To actually run models on this engine, the Qwen3 and Kimi K3 guides both drive vLLM end to end, and the hardware guide tells you what fits where.