Run open-weight models on your own hardware

Engine-by-engine guides for running local LLMs, deeper than the model cards. Each covers Ollama, llama.cpp, vLLM, LM Studio, and MLX on Apple Silicon, with real VRAM math, quant selection, tool calling, an OpenAI-compatible endpoint, and the failure modes nobody documents.

MarkdownAsk Claude, ChatGPT, or Perplexity

Engine internals

Evals and Benchmarks new Any model · any harness · 5 interactive figures Read a public score without being fooled, then build, grade, calibrate, and hillclimb your own eval, with every number measured on a local lab or cited. Inside the Transformer Build it · mutate it · run it · 13 interactive figures Attention built up from a masked average, nine years of architecture diffs to the 2026 block, then the chat, the sampler, and the serving math. Inside Miles new Miles main · RL on SGLang · 4 interactive figures One RL step through RadixArk's Miles: the refit session, four weight transports, async staleness, and the gap between sampler and trainer, with measured runs on a Mac. Inside SGLang new SGLang v0.5.22 · radix cache · HiCache · 8 interactive figures Why SGLang became popular, one prompt across CPU and GPU, real attention matrices, the radix tree, HiCache tiers, admission and retraction, and scale-out, with data captured on a Mac. Inside the vLLM Engine vLLM · V1 architecture · 6 interactive figures The paged block pool, the phase-free scheduler, the two-process loop, speculative decoding, and grammar masking, stepped through block by block.

Model guides

MiMo-V2.6 new Xiaomi · 9B / 309B A15B / 1.02T A42B · MIT Three sizes on one release day: which one your machine runs, the 128-token window that keeps the Flash cache at 22.5 KB per token, every quant with its measured perplexity cost, and the llama.cpp build that runs it. DeepSeek-V4.1-Flash new DeepSeek · 552B backbone, 16B active · MIT · 1M ctx The 510 GB checkpoint byte by byte: 289 GB of experts, a 203 GB Engram lookup table that can live in host RAM, 890 bytes of KV per token, and the vLLM, SGLang, and llama.cpp branch commands that run it. Qwen3.8-Flash-Next Alibaba · 125B A6B + 51B n-gram · 262K ctx The first open model whose biggest tensor is a lookup table. Where the 51B n-gram table goes, every GGUF size, the 12-of-48-layer KV ledger, and speed ceilings by machine. GLM-5.3 new z.ai · 744B MoE, 40B active · GLM-5.3 License GLM-5.3 at 744B with 40B active: why "Max" is a setting, the sparse-attention cache math, every checkpoint from 216 GB GGUF to 1.5 TB BF16, and a rig composer that says what runs. Hy4 Preview Tencent · 770B MoE, 49B active · Apache-2.0 Tencent's 770B A49B preview under Apache-2.0: what "preview" implies, MLA plus DSA cache arithmetic, every GGUF, MLX, NVFP4 and FP8 size, computed fit by machine, and llama.cpp, vLLM, and SGLang commands. GLM-5.3-Flash z.ai · 320B A18B · MIT · day-one Run z.ai's stealth-tested ox-alpha model locally: 320B total, 18B active, MIT, natively multimodal, with the hybrid attention KV math, every unsloth GGUF size, a fit planner for Macs and multi-GPU boxes, and the branch and container commands that run it before any tagged engine release does. Granite 4.2 IBM · 3B / 8B / 30B dense · Apache-2.0 IBM's three dense Apache-2.0 reasoning models. Every layer keeps a full cache, so at 128K the 8B's cache is four times its Q4 weights. Every file size, a size grid by machine and task, and the commands for Ollama, llama.cpp, vLLM, SGLang, and MLX. Muse Glimmer Meta · 30B dense agent · Apache 2.0 · day-one Meta's return to open weights: the 24GB envelope packer, every GGUF size, DFlash drafter math, ATEM template, and honest benchmarks. Qwen3.8-27B Qwen · 27B dense VLM · Apache-2.0 Run Qwen3.8-27B locally: why 48 DeltaNet layers keep a fixed 154 MB state while 16 attention layers grow 64 KB per token, every quant size, what fits on 24 to 48 GB, and why llama.cpp checkpoints the state. DeepSeek V4 Flash DeepSeek · 284B A13B · 1M ctx · lossless Q8 Interactive: pick a quant, see if it fits your RAM, every command rewrites itself. QAT means Q8 is bit-exact at 162 GB. LFM2.5 Liquid AI · 230M to 8B-A1B edge family · LFM Open License v1.0 Liquid AI's edge family on phones, laptop CPUs, and Macs: the license threshold read from the file, conv-plus-attention cache math, every GGUF/MLX size, and measured M1 Max numbers. MiniMax-M3 MiniMax · 428B MoE, ~23B active · MiniMax Community License MiniMax's 428B MoE whose attention reads 16 blocks of 128 tokens per layer: the MSA cache arithmetic, every quant size, where the 7,296 expert slots go on GPU, RAM and SSD, and commands for llama.cpp, vLLM, SGLang, KTransformers and mlx-vlm. Kimi K3 Moonshot · trillion-scale MoE The frontier open MoE. Dynamic quants, MoE-to-CPU offload, and multi-GPU serving for a model that does not fit one card. Qwen3 Alibaba · 4B to 480B, dense + MoE The everyman family. Runs on a 4090 at 32B, scales to 480B Coder. Thinking mode, tool calling, and the sweet-spot quants. gpt-ossOpenAI · 20B / 120B · Apache-2.0Native MXFP4 makes the 120B fit one GPU. Harmony format, reasoning_effort, and every engine. DeepSeek V3.2 / R1DeepSeek · 671B MoE · MITDistills on a consumer GPU, the full 671B on a big-RAM box with dynamic quants and expert offload. Gemma 4 Google · E4B / 12B / 26B-A4B MoE / 31B · Apache-2.0 Per-layer embeddings keep E4B's 2.8B table off the GPU, and a 5 to 1 local/global layout keeps the cache small. Every quant and what fits. Gemma 3Google · 1B to 27B dense, visionThe phone-to-workstation dense line. The 27B on a 4090, image input from 4B up. MistralMistral · Small / Large 3 · DevstralThe 24B Small on a mainstream GPU, Devstral for coding agents, Large for multi-GPU. Llama 4Meta · Scout / Maverick MoENative multimodal, huge context. Scout on a 64GB Mac or 24GB card with offload. GLM 4.6 / 5z.ai · MoE · MITStrong agentic coder. Air on a single card with offload, wired straight into a coding agent.

Read this first

Every model guide assumes the same four decisions, so learn them once. Which engine: Ollama for the fastest start, llama.cpp for control and huge-model offload, vLLM for multi-GPU throughput and an OpenAI-compatible server, LM Studio or MLX on a Mac. Which quant: Q4_K_M is the reliable 4-bit default, Q8_0 is near-lossless, and dynamic low-bit quants are what make trillion-parameter models fit at all.

How much memory: weight bytes are roughly params times bits-per-weight divided by eight, so a 32B model at 4-bit is about 18 to 20 GB, plus headroom and KV cache. A mixture-of-experts model needs memory for its total parameters but runs at the speed of its active parameters, which is why a 1T model with 32B active can run on a big-RAM box at usable speed. How you use it: every engine here can expose an OpenAI-compatible endpoint, so any agent or SDK drives a local model by pointing its base URL at your machine.

For the full breakdown of what fits a Mac Mini, MacBook Pro, Mac Studio, or a given GPU, and how much RAM each model needs, see the hardware and RAM guide.