Engine internals guide · August 9, 2026 · 30 min read

Inside the Transformer

Every chat reply you have ever received from a language model came out of one function: given text, score every possible next token. The function is a transformer, it has not fundamentally changed since 2017, and it is small enough to hold in your head. This guide builds it from zero the way you would in code, then applies nine years of architecture diffs to reach the models shipping in 2026, then runs it behind a chat window. Thirteen interactive figures, and every number in them comes from the source paper, the model's own config file, or arithmetic you can check.

Act I · Build it

One function, called in a loop

A language model does not write. It predicts. Give it any text and it returns a probability for every token in its vocabulary as the continuation. Generation is a loop around that single call: predict, sample one token from the distribution, append it, predict again. The model that answers you in a chat window is the same machine that would complete a grocery list; the chat part is a document it is completing, one token per call, and nothing more. Run the loop yourself.

Two things make this loop expensive to build and interesting to study. First, the function has to be good, and "good" turns out to require reading all of the context at once. Second, everything inside it is learned. The function is a stack of matrix multiplications with roughly 10 million to 2.8 trillion tunable numbers, none set by hand. The rest of Act I is what those matrices do; the numbers come from GPT-3, the last frontier model whose full dimensions were published, following 3Blue1Brown's bookkeeping: 175 billion parameters in just under 28,000 matrices.

Words become vectors

The function needs numbers, not words, so the text is first split into tokens: not quite words, not quite characters. The trade-off runs on one axis. A character vocabulary is tiny (Karpathy's Shakespeare model uses 65 symbols) but makes sequences long, and a character carries almost no meaning for the embedding to hold. A word vocabulary is semantically clean but explodes in size, treats "bear" and "bears" as strangers, and shatters on any word it has never seen. Subword tokenization sits between, and the algorithm most models use, byte-pair encoding, is greedy and almost embarrassingly simple: start from raw bytes, count every adjacent pair in the training corpus, merge the most frequent pair into a new token, repeat until the vocabulary hits a target size. Frequent words end up as single tokens, rare words split into familiar pieces, and nothing is ever out of vocabulary because the bytes are always there as a floor. Watch it work on your own text.

Vocabulary size is an economic dial, not a taste. Every token of vocabulary is a row of embedding parameters, but every merge shortens sequences, and sequence length is what attention's cost grows on. The dial has moved steadily up: GPT-2 and GPT-3 used 50,257, Llama 2 used 32,000, Llama 3 jumped to 128,256 by starting from OpenAI's tiktoken vocabulary and adding 28k more, and 2026 flagships sit between 128k and 262k, with Gemma 3 at the top at 262,208. Multilingual and code-heavy models push the dial hardest because their text fragments worst under a small English-trained vocabulary. Tokenizer quirks also leak into model behavior: digits are usually split so arithmetic stays learnable, and the tokenizer is why models miscount letters in a word; the model never sees letters, only token IDs.

Each token then picks its row from a learned embedding matrix: one vector per vocabulary entry, 12,288 dimensions in GPT-3, which makes the embedding table alone 50,257 × 12,288 ≈ 617 million parameters. The remarkable part is what training does to those vectors: directions in the space come to mean things. The classic demonstrations, made famous by word2vec and animated by 3Blue1Brown, still land: subtract the vector for "man" from "woman" and you get a direction that, added to "king", lands near "queen". Dot a candidate "plurality direction" (cats minus cat) against nouns and plurals reliably score higher. Meaning becomes geometry, and the dot product becomes the measuring tool the whole architecture is built on.

One more ingredient goes in before any computation: position. Attention, as you are about to build it, treats its input as a set, and sets have no order; "dog bites man" and "man bites dog" would be the same input. So every model injects position information into the stream. The original paper added sinusoidal waves to the embeddings, GPT-2 and GPT-3 learned one vector per position, and Act II replaces both with something better. For now: token vector plus position information, and the stack above sees both.

The trick that lets tokens talk

Here is the gap the transformer fills. Each token's vector knows its word and its position, and nothing else. Prediction needs context: the vector for "was" at the end of a mystery novel has to end up carrying the plot, because the next-token distribution is computed from the last vector alone. Tokens need a way to pass information to each other, and it has to respect time: a token may look backward, never forward, because the future is what the model is being trained to predict.

Karpathy's build of GPT introduces the mechanism through a trick worth walking slowly, because the whole of attention is hiding inside it. Start with the weakest possible communication: let each token's vector become the average of itself and everything before it. A loop over positions computes it, but so does one matrix multiply: build a lower-triangular matrix of ones, normalize each row to sum to 1, and multiply it against the stacked token vectors. Row four of that matrix reads "mix tokens one through four in equal parts". Then notice the same matrix can be produced a third way: start from all zeros, set the upper triangle to negative infinity, and take a softmax of each row. Zeros become equal weights; negative infinity becomes zero weight. Step through all three.

The punchline of the trick: that final matrix, zeros softmaxed under a triangular mask, is attention. The only thing wrong with it is that the weights are constant. Every token averages its past indiscriminately, a bag of words. What attention adds is exactly one upgrade: the zeros become data-dependent scores, so that each token can decide, per step and per context, whose information it wants. Everything else, the mask, the softmax, the weighted mixing by matrix multiply, you have already built.

Queries, keys, values

The scores come from a matching game. Every token publishes two small vectors, computed from its embedding by two learned matrices. The query (W_Q times the embedding) encodes what the token is looking for; Karpathy's paraphrase is "I'm a vowel at position eight, looking for consonants before me"; 3Blue1Brown's is a noun asking "are there adjectives in front of me?". The key (W_K) encodes what the token has to offer. The score between any two tokens is the dot product of one's query with the other's key, the same alignment-measuring tool from the embedding section. High dot product, high affinity. In GPT-3 these vectors are small, 128 dimensions against the 12,288 of the embedding, and each of W_Q and W_K is about 1.5 million parameters.

Scores in hand, the machinery you built in Fig. 4 takes over unchanged: mask the future with negative infinity, softmax each row into weights. What gets mixed is a third projection, the value (W_V): the token's actual payload, "what I will tell you if you find me interesting", in Karpathy's phrasing. Each token's output is the weighted sum of the values of everything it attends to, and that sum is added to its vector, nudging it. In the sentence below, the nudge is what moves "creature" from generic-creature toward fluffy-blue-creature. Click tokens and watch the arithmetic.

Three refinements complete the real head. First, the scale: raw dot products grow with dimension (a sum of 128 products has standard deviation about √128 if the entries are unit variance), and a softmax fed large inputs collapses toward a one-hot vector, so every token would hear from exactly one other. Dividing scores by √d_k keeps the distribution diffuse where learning needs it. That is the entire content of the famous formula: Attention(Q,K,V) = softmax(QKⁿ/√dₖ)V, and every symbol in it is now something you have built. Second, the mask earns a training justification beyond causality: one sequence of T tokens trains T predictions at once (what follows token one, what follows tokens one-two, and so on), which only works if position t cannot peek at t+1. Third, the value path is low-rank: instead of one 12,288 × 12,288 value matrix (150 million parameters for a single head), implementations factor it through the small 128-dimensional space, down then up, so all four matrices of a head match in size and the head totals about 6.3 million parameters. The up-projections of all heads get stapled into one output matrix; when a paper says "value matrix" it usually means only the down half. That bookkeeping trap comes straight from 3Blue1Brown, and it will bite anyone who reads the code after the papers.

It is worth pausing on how general the thing you just built is. Attention is a communication protocol over a directed graph: nodes hold vectors, each node aggregates a weighted sum from the nodes that point at it, and the weights are computed from the data. The causal mask is one choice of graph, a triangle where each token receives from its past. Delete the mask and every token talks to every token, which is what BERT-style encoders do for tasks that read text rather than continue it. Point the queries at one sequence and the keys and values at a different one and you have cross-attention, the bridge in translation systems and speech models like Whisper. The chat models this guide follows use exactly one variant, masked self-attention, applied everywhere; the others are the same five lines with a different mask.

Many heads, then a place to think

One head learns one kind of relationship. Real context is many relationships at once: adjectives modifying nouns, verbs binding subjects, a "wizard" three sentences back recoloring every "Harry" after it. So the block runs heads in parallel, 96 of them in GPT-3, each with its own Q, K, V matrices, each producing its own proposed nudge, and all 96 nudges sum into the token's vector. Nothing forces the heads to specialize differently; they start random and diverge because gradient descent finds it useful, and when researchers dissect trained models they find heads that track positions, heads that copy repeated patterns forward (the induction heads behind in-context learning), and many heads whose job resists any clean description. Per block that is roughly 600 million attention parameters; across GPT-3's 96 layers, about 58 billion, which is the origin of a fact most people find backwards: attention is only a third of the model.

The other two thirds sit in the layer nobody makes videos about. After attention lets tokens exchange information, a feed-forward network processes each token alone: project the 12,288-vector up to four times its width, apply a nonlinearity, project back down. Karpathy's framing is the cleanest: attention is communication, the MLP is computation; "the tokens looked at each other, but they have not had time to think about what they found". The interpretability literature adds that these wide layers are where much of a model's factual knowledge appears to live. Communication, then thinking, wrapped with two more pieces of wiring: a residual connection around each sublayer (the vector flows through untouched and the sublayer only adds to it, which keeps gradients flowing through 96 layers; Karpathy calls it the gradient superhighway) and a normalization before each sublayer to keep the numbers in a healthy range. That whole unit is a block. A transformer is that block, stacked.

Training: the ladder of loss

Nothing so far said how the matrices get their values. The answer is one number driven downward. For each position in each training sequence, the model produces a distribution over next tokens; the loss is cross-entropy, the negative log probability it gave to the token that actually came next: −ln p(truth). The formula has convenient teeth. A model guessing uniformly over Karpathy's 65-character vocabulary scores −ln(1/65) ≈ 4.17, so you can predict an untrained model's loss before running it; his network starts near 4.87 because random initialization is slightly worse than uniform, and the first hours of training are mostly the model discovering which characters exist at all. Loss is also exponent-honest: a drop from 4.17 to 1.48 means the model's average surprise per character fell from 65 effective choices to about e^1.48 ≈ 4.4.

The driving mechanism is backpropagation: run the forward pass, measure the loss, then push its gradient backward through every matrix, nudging each of the billions of parameters a little more toward the truth and a little away from everything else, then repeat, trillions of times. The residual connections from the block design are what make this survivable at depth; the gradient flows through the addition nodes unimpeded from the loss back to layer one. And one training trick from the attention section pays for itself here: because of the causal mask, one sequence of 2,048 tokens is 2,048 training examples computed in a single forward pass, one prediction per position. At GPT-3 scale the loop ran with a batch of 3.2 million tokens per step and a learning rate of 0.6×10⁻⁴, over roughly 300 billion tokens of text, a corpus a human reader would need more than 2,600 years to get through.

The reason to trust each component you just built is that each one measurably buys loss. Karpathy's lecture is an ablation run in disguise, and the numbers make the case better than any argument:

Scale is the last rung, and it has arithmetic. Training cost is roughly 6 FLOPs per parameter per token (Kaplan's C ≈ 6ND), so GPT-3's 175B parameters over 300B tokens cost about 3 × 10²³ operations; 3Blue1Brown stages it as a riddle, and the answer lands hard: at one billion operations per second you would need well over 100 million years. The Chinchilla result then set the exchange rate between the two factors: compute-optimal training wants tokens to scale with parameters, roughly 20 tokens per parameter (a derived ratio; the paper's own statement is that both should double together). Modern models deliberately overshoot it, 40 trillion tokens into Llama 4 Scout's 109B, because a smaller model trained longer is cheaper to serve, and serving is where the money goes, as the inference bill essay argues at length.

One more stage separates this machine from a chatbot. Pretraining produces a document completer: ask it a question and it may answer, continue your question, or start a news article, because all three continue documents. Turning it into an assistant is a pipeline of far smaller phases, and each one is worth naming because the industry now spends most of its innovation there. Supervised fine-tuning retrains on thousands, not trillions, of question-answer shaped documents; big pretrained models are startlingly sample-efficient, so this alone produces something assistant-shaped. Preference tuning then optimizes for what humans like rather than what documents contain: classically a learned reward model scored candidate answers and PPO-style reinforcement learning chased the score (the RLHF recipe behind the original ChatGPT); direct preference optimization later folded much of that machinery into a single loss on preference pairs. And since 2025 the frontier has been reinforcement learning on reasoning itself, GRPO-family methods that reward getting hard problems right and let the model learn to spend more tokens thinking before answering. The base machine is unchanged through all of it; what changes is which documents it believes it is completing.

Count the machine yourself

Everything in Act I is now countable, and counting is the fastest way to make an architecture concrete. A decoder-only transformer has exactly four kinds of parameter mass: the embedding table, each block's attention matrices, each block's MLP, and the norms (a rounding error). The unembedding usually costs nothing extra because it reuses the embedding table transposed, tied embeddings. Watch the ledger fill for the two models whose books are public.

The ledger generalizes into a mental shortcut worth keeping: per block, attention is 4d² (four d×d matrices, less under GQA) and the MLP is 8d² (up and down projections at 4× width; gated variants make it 12d² at 8/3× width), so a dense transformer is roughly 12d² parameters per layer plus one embedding table. Every scaling decision a lab makes is arithmetic on that expression.

Act II · Mutate it

Why 2017 happened

The transformer was not an aesthetic invention; it was an escape from two specific failures. Before it, language models were recurrent: read a token, update a hidden state, repeat. That design forgets, because a whole document has to squeeze through one fixed-size state, and its gradients decay or explode through the chain of updates (LSTMs softened this, never solved it). Worse for economics, it is sequential: state t needs state t−1, so a thousand-token document is a thousand dependent steps and the GPU's parallelism sits idle. Attention had appeared in 2014 as a patch, a direct connection from the prediction back to any input position; the 2017 paper's bet was to delete the recurrence and keep only the patch. Every token attends to every token in one parallel step, order comes from position information instead of processing order, and suddenly the architecture scales with exactly the hardware the industry was already building for graphics. The deep lesson of the decade, that scale itself buys qualitative capability, needed an architecture that could absorb scale. This one could.

Nine years of diffs

What you built in Act I is the 2017 transformer of "Attention Is All You Need" (Vaswani and seven co-authors), minus its encoder half; GPT dropped that half, kept the masked decoder, and the field followed, because next-token prediction turned out to be the simplest objective that scales, and one stack is simpler than two. Since then the block has been mutated component by component, each swap published, measured, and adopted for a stated reason. Flip them yourself.

Where the norms went

The 2017 block normalized after each residual add. Practice moved the norm inside the branch, before the sublayer, which keeps the residual path clean and the gradients well behaved from initialization (Xiong et al. 2020 supplied the analysis; GPT-2 had already made the move in 2019). The norm itself slimmed down: RMSNorm drops LayerNorm's mean subtraction and bias, keeping only the scale, after showing the re-centering was dispensable. Some current models (Gemma 3 among them) norm both before and after the sublayer, and most now also normalize queries and keys per head, QK-norm, to stop attention logits blowing up at scale; Gemma 3 adopted it in place of soft capping, Qwen3 in place of QKV biases.

What the MLP became

ReLU gave way to GELU, and GELU to gated variants: SwiGLU and friends compute two up-projections, pass one through the nonlinearity, and multiply them, letting the layer modulate its own signal. Shazeer's paper offered no theory, ran the ablations, won anyway, and closed with a line the field still quotes: the improvements are attributed "to divine benevolence". Llama-family models use SwiGLU; some use GeGLU; the structural change, gate times signal, is the same.

Where position went

Position encoding has its own nine-year story, and each step fixed the last one's stated flaw. Learned position vectors (GPT-2, GPT-3, BERT) work but hard-cap the context: position 2,049 has no trained vector, so the window cannot grow after training. The original paper's sinusoids fixed extrapolation with a lovely property: the dot product between two positions' encodings works out to a function of their distance alone, decaying as positions separate. But adding either kind to the embedding is indirect; the place position actually matters is the attention score, so later schemes moved it there. T5 learned a bias added directly inside the scores, bucketed by distance; ALiBi made the bias a fixed linear penalty, no learning at all. The winner, in nearly every 2026 model, keeps the sinusoids' frequency idea but applies it as geometry: rotary position embeddings. Pair up the dimensions of each query and key and rotate each pair by an angle proportional to the token's position, a different frequency per pair, fast frequencies for local order, slow ones for document-scale order. The payoff is a small theorem: a rotated query dotted with a rotated key depends only on the difference of their positions, so the score sees relative distance, exactly the thing language cares about. Turn the dials.

How attention got cheap enough to serve

Multi-head attention pays twice at inference: the KV cache stores every head's keys and values for every token. Multi-query attention (one shared K/V for all heads) cut the cache at some quality cost; grouped-query attention found the useful middle, a few K/V groups shared by many query heads, and became the 2024-26 default (Qwen3-32B: 64 query heads, 8 KV heads). DeepSeek's multi-head latent attention went further, storing one small latent per token that decompresses into all heads' K/V, cutting the cache by a reported 93 percent. Queries stay per-head throughout; diversity in what tokens ask is worth keeping, sharing what they answer is nearly free. The cost side has its own lineage: FlashAttention reorganized the exact computation to never materialize the T×T matrix, and models restructured the pattern itself, Mistral's sliding window, Gemma 3's five local layers per global one (local span 1,024), gpt-oss alternating banded and dense layers with learned attention sinks, a per-head bias in the softmax denominator that lets a head cleanly say "no token is worth attending to right now", the production descendant of the StreamingLLM observation.

The block that is not always there

The largest diff touches the MLP: replace the single feed-forward network with many (Switch Transformer proved the routing could be trivial; Mixtral made it open), route each token to a few of them, and suddenly total parameters and per-token compute are different numbers. The router is nothing exotic: one small linear layer scores every expert per token, a softmax picks the top-k, and the chosen experts' outputs are blended by the router's own weights. The engineering is in keeping the traffic balanced, because a router left alone collapses onto favorite experts while others starve; Switch added an auxiliary balancing loss, and DeepSeek-V3 showed the loss itself can go, replaced by a per-expert bias nudged whenever an expert runs hot or cold. Two design details recur: a shared expert that every token visits (common knowledge should not be replicated 256 times), and the honest finding that routed experts mostly do not map onto human topics; the router learns a partition that helps loss, not a taxonomy. DeepSeek-V3 runs 256 routed experts plus one shared, activating 8: 671B parameters, 37B per token. This is why "how big is the model" became two questions, and it connects backwards to Act I: the experts are the thinking layer, so mixture-of-experts is the statement that a model can know far more than it needs to think with on any one token.

What a frontier model actually looks like

Every row below is from the model's own config file or model card, fetched and verified this week. Read it as the state of the mutation, 2019 to 2026.

From each model's HuggingFace config.json or model card, checked August 2026. Kimi K3's 93 layers split 69 linear-attention (KDA) and 24 gated MLA, the furthest drift from the 2017 recipe in the table. Llama 4 routes to 1 expert plus a shared one; most MoE models route top-4 to top-16. Training tokens where disclosed: Llama 4 Scout ~40T, Qwen3 36T, DeepSeek-V3 14.8T, Gemma 3 27B 14T.

Two trends carry the table. Vocabularies grew five-fold (50k to 262k) because shorter sequences amortize everything else. And the gap between total and active parameters became the main scaling axis: Kimi K3 holds 27 times more knowledge than it applies to any single token. The 2017 paper's architecture is still legible in every row; not one of these models changed what a block is, only what each part is made of.

Act III · Run it

From chat to tokens

Nothing in the machine knows what a conversation is. A chat arrives at the model as one long document assembled by the harness: special tokens open and close each role, the system prompt and tool definitions come first, and every turn is appended to the same growing sequence. The model's entire chat ability is that fine-tuning taught it how documents shaped like this tend to continue: after the token that opens an assistant turn, helpful-assistant text is what comes next. This is also where serving economics start: the assembled prefix is identical from turn to turn, which is exactly the property vLLM's prefix cache monetizes, and why a harness that keeps its context stable is cheaper than one that rewrites it, the thesis of the harness comparison. Flip the view.

Sampling: the last fifty lines

The model ends every call with logits, one raw score per vocabulary entry, and softmax turns them into probabilities. The sampler is the only part of the whole system a user's settings touch. Temperature divides the logits before the softmax: below 1 it sharpens the distribution toward the favorite, at 0 it is argmax and the model becomes deterministic, above 1 it flattens toward uniform and prose drifts strange. Top-p then cuts the tail: keep the smallest set of tokens whose probabilities sum past the threshold, renormalize, sample. Two sliders, and they are the difference between a model that answers the same thing every time and one that surprises you.

The serving arithmetic

Running the loop in production adds one data structure and one asymmetry, and both were the subject of this guide's predecessor. The data structure is the KV cache: causality means a token's key and value never change once computed, so they are stored, and generation only computes the new token's query against old keys. The size is pure bookkeeping: 2 vectors × layers × KV heads × head dimension × 2 bytes, per token. GQA divides it by the group factor, MLA compresses it to a latent, sliding-window layers cap it, and the block-pool machinery of vLLM manages what remains. The asymmetry is prefill versus decode: digesting the prompt is thousands of tokens of parallel matrix work and saturates compute; generating is one token per pass and saturates memory bandwidth, which is why tokens per second is a bandwidth number and why serving engines schedule the two phases so differently. Drag the context and watch both costs move.

The honest limits belong in the same breath, and they deserve plain statement rather than fine print. The quadratic is real: double the context, quadruple the attention work; the figure above is why sub-quadratic architectures keep being proposed, and why production models mostly chose windowed and hybrid patterns over exotic approximations, paying compute for exactness. Long context dilutes: models demonstrably lose the middle of very long windows, and a million-token window is not a million tokens of equal recall; retrieval and attention compete as answers to the same question. Correlation is not deduction: next-token prediction learns how words follow words, and the model that completes your sentence flawlessly can fail a two-hop question about who saw whom. Chain-of-thought prompting was the first patch, borrowed reasoning through imitation; the 2025-26 answer trains reasoning directly with reinforcement learning, and it changed what these machines can do. But it is a training-stage change. The block you built in Act I is untouched underneath, which is the closing fact of this guide: three acts, one architecture, and every year's revolution so far has been a better way to feed it, shape it, or serve it.

Go deeper

This guide compresses a canon, and the canon is worth your hours. 3Blue1Brown's deep-learning series (the brief LLM explainer, then chapters 5 and 6) is the best visual intuition for embeddings and attention ever made, and its GPT-3 bookkeeping is the spine of Act I. Karpathy's build-GPT lecture is the build order this guide follows, live in code, with the loss ladder as receipts; StatQuest hand-computes a full translation example if you want every number on screen. Stanford's CME295 covers the same arc lecture by lecture through RLHF, reasoning, and agents, exam-grade. The papers cited inline are all first sources. And for what happens after the logits leave the model, the series continues in Inside the vLLM Engine.