Local model guide · August 28, 2026 · 17 min read

How to Run GLM-5.3-Flash Locally

For a week in August the most popular model on OpenRouter had no name. It was called ox-alpha, it ran on Chinese chips, and on August 26 z.ai revealed it as GLM-5.3-Flash and put the weights on Hugging Face under MIT: 320 billion parameters, 18 billion active, natively multimodal, with a hybrid linear-plus-sparse attention design that keeps the KV cache small at a million tokens of context. "Flash" describes the serving cost, not the download. The smallest usable quant is 93 GB, no inference engine ships it in a tagged release yet, and the fastest path today is a branch or a vendor container. This guide has the memory math with a KV ledger you can drive, every unsloth GGUF size, a fit planner for Macs and multi-GPU boxes, and the exact commands for each engine as of September 3, 2026.

What it is

GLM-5.3-Flash is z.ai's first natively multimodal GLM-5 model and, by its own description, the cheap one: the launch post says it "outperforms GLM-5.2 across benchmarks and real-world workloads at one-tenth the price, while approaching Claude Opus 4.8 on coding and agentic benchmarks." Unlike the GLM-4.5 Air and GLM-4.6 line, it starts from a newly trained base rather than a distillation of the big model, on what z.ai calls a 30T-token multimodal corpus, and it is the first GLM to mix linear and sparse attention in one stack.

The stealth test is the part people remember. z.ai served the model anonymously as ox-alpha on OpenCode and OpenRouter to collect feedback before the reveal, and says it "quickly became the most popular model of the week." The Hacker News thread on the Bloomberg story that connected the dots reached 435 points before the weights landed, and the launch thread itself hit 1,132 points and 580 comments (HN). The early reviews from the stealth week are worth holding onto: one HN commenter who ran it for two days on coding tasks put the quality "between Sonnet and Opus" but reported it degrading into a loop several times, "running the same bash command about a thousand times." That is a harness problem as much as a model problem, and a reason to run it inside a loop detector; more on that in the agent section.

What the release is not: small. The active parameter count is 18B, which sets the per-token compute and, with the right quant, the decode speed. The total is 320B, which sets what you must hold in memory. A model with 18B active parameters still has to page every expert it might route to, and the routed experts are where most of the 320B live. The Reddit reception line that stuck, "neither Flash nor Air," is about exactly that gap. Read the fit planner before you download 200 GB.

The architecture, briefly

Every row below feeds the memory math. The values are from the raw config.json (model type glm5_next), and if you have read the transformer internals guide the shape will be familiar, with two additions the GLM line did not have before.

z.ai's framing of the design is compute per token: compared with GLM-4.5's 355B, this model "nearly halves both the activated parameter count (18B vs. 32B) and the number of layers (45 vs. 92)." The attention split is the interesting half. Linear attention "captures local dependencies through state modeling," so 34 of the 45 layers carry a fixed-size recurrent state instead of a growing cache. The 11 sparse layers do the global retrieval through a lightweight indexer, and IndexPool "compresses four indexer key vectors into one through weighted pooling" so the indexer itself stays cheap at a million tokens. z.ai's own comparison: "3.0x less attention compute and a 4.4x smaller KV cache" than the full GLM-5.3, with the caveat in the same paragraph that the cache is "still slightly larger than Kimi-K3 and DeepSeek-V4-Flash."

Why the KV cache is small

Here is the arithmetic that makes long context cheap. A sparse MLA layer stores one latent vector per token: kv_lora_rank 512 values, and because the rope dimension is 0 there is no separate positional slice. In BF16 that is 512 × 2 = 1,024 bytes per token per layer, times 11 layers = 11,264 bytes per token. At 262,144 tokens the cache is 2.95 GB. At the full million it is 11.8 GB. The 34 linear layers add a constant: each holds a 64 × 128 × 128 state in BF16, about 2 MiB per layer, 71 MB for the stack, no matter how long the conversation runs. Compare that with the full GLM-5.3, where every one of 78 layers is MLA with a 64-wide rope slice on top of the 512 latent: 89,856 bytes per token, eight times more per token, and about 4.6x more per layer, which is close to z.ai's 4.4x once you count the linear state the way they presumably do. Drive it:

The practical reading: on this model the KV cache is a rounding error next to the weights until you are far past 100K tokens, so the context slider in the planner below barely moves the bar. That is the opposite of the dense 30B agent models where the cache decides whether you fit in 24 GB. Here the weights decide everything.

Pick a quant

z.ai ships the model as FP8 by default (about 306 GiB of safetensors per the vLLM recipe) with a BF16 repo beside it. For anything smaller you want unsloth/GLM-5.3-Flash-GGUF, the only broad quant ladder published so far. The sizes below are sums of the shard files from the Hugging Face tree API on September 3, 2026, in decimal gigabytes; the quality column is unsloth's own top-1 KL retention figure from their model page, where they list it, and the RAM guidance is theirs too.

Two things to notice. First, the 1-bit quant is 93 GB, which is bigger than the 4-bit build of most models people call "large" locally; there is no consumer-GPU story for this model without a lot of system RAM. Second, unsloth's own retention numbers say the 1-bit tier keeps about 71% of top-1 agreement with the full model and the 4-bit tier keeps about 92%. For an agentic coder that difference shows up as tool-call format errors and loops, so if you can hold Q4_K_XL, hold it. The image projector is a separate BF16 file of about 1.2 GB; the main GGUF is text-only without it.

What fits where

Four things share memory: the weights, the KV cache from the ledger above, the vision projector if you want images, and the engine's own buffers. On a GPU box the experts can live in system RAM with -ot offload while attention stays on the card, at single-digit tokens per second. On unified-memory Macs and Sparks the whole thing is one pool, and about three quarters of it is realistically yours after the OS. Pick your machine:

The reported numbers deserve their labels. A r/LocalLLaMA user posted a DGX Station GB300 run at about 206 tokens per second for a single user; another shared an RTX PRO 6000 setup driving a 3D tool through MCP, which collected around 600 votes; a four-Spark cluster recipe made the HN front page. None of these are my measurements and none are on tagged software, so treat them as existence proofs, not expectations.

Engines, day eight

This is the awkward part of a day-one guide written on day eight. As of September 3, no major inference engine ships GLM-5.3-Flash in a tagged release. Transformers has it on main since August 26. vLLM's support is an open pull request and the official recipe requires a version that does not exist yet. SGLang's model PR is open too, but z.ai and the SGLang team published a vendor container that runs it today. llama.cpp on master rejects the file with unknown model architecture: 'glm5next', and three competing pull requests are racing to fix that. mlx-vlm merged native support on release day. The board tracks it:

llama.cpp on a branch

Stock builds fail. Issue #27922 is the canonical "unknown model architecture: 'glm5next'" report, and the fix is in flight in PR #27754 by Daniel Han (opened August 26, still updating on September 1), with #27752 and #27773 as competing implementations and #27917 adding the MTP drafter on top. Unsloth's GGUFs were converted against their own fork branch, so the shortest path is to build that:

git clone https://github.com/unslothai/llama.cpp.git unsloth-llama.cpp
cd unsloth-llama.cpp
git fetch origin pull/61/head:glm5next
git switch glm5next

# NVIDIA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j
# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

./build/bin/llama-server \
  -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL \
  --jinja -ngl 99 -c 65536 \
  --temp 1.0 --top-p 0.95 \
  --host 127.0.0.1 --port 8080

Or fetch the upstream PR the same way from ggml-org/llama.cpp with pull/27754/head. Two CUDA caveats reported on the PR thread, both unconfirmed by me: set NVIDIA_TF32_OVERRIDE=0 and run with flash attention off (-fa off) if you see garbage or a crash; and bug reports against PR builds are still arriving, among them #28144 (a SOFT_MAX failure on Turing cards) and #28282 (a CUDA illegal memory access). If your card is older than Ampere, wait.

For the one-GPU-plus-RAM case, keep attention on the card and push the experts to system memory. This is the same pattern as the DeepSeek V4 Flash guide, and the regex is the one the GLM family has always used:

./build/bin/llama-server \
  -hf unsloth/GLM-5.3-Flash-GGUF:UD-Q2_K_XL \
  --jinja -ngl 99 \
  -ot ".ffn_.*_exps.=CPU" \
  --cache-type-k q8_0 -c 32768 \
  --host 127.0.0.1 --port 8080

Images need the projector. Download the mmproj file from the same repo and pass it with --mmproj; without it the model is text-only and silently ignores image parts. Multi-shard GGUFs load from the first shard; you do not need to merge them.

vLLM and SGLang

vLLM. The official recipe asks for vLLM 0.29.0 or newer, which has not been tagged, and FlashInfer 0.6.18 or newer; the model code is in PR #53906. The recipe's example is tensor parallel 4 on the FP8 weights (about 306 GiB), and it notes that Hopper GPUs need a BF16 KV cache. Until the release, install from the PR head the way the DFlash 2 drafters were installed last month:

pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/53906/head"

vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm45

The parser names above are the ones the SGLang cookbook uses for this model; the vLLM recipe carries the exact vLLM flags and is the file to copy from when it changes. Once paged KV and continuous batching matter to you, that is, when a team or an agent fleet shares the box, vLLM or SGLang is the right place, and the small per-token cache is what makes a high concurrency count fit.

SGLang. The model PR (#36507) is open, but the cookbook merged on August 26 (#36440) and it runs from a vendor image, with the built-in MTP head as the drafter, and DFlash drafter support merged a day later (#36708):

docker run --gpus all --ipc=host -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:glm-5.3-flash \
  python3 -m sglang.launch_server \
    --model-path zai-org/GLM-5.3-Flash \
    --tp 4 --ep 4 \
    --speculative-algorithm NEXTN \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --host 0.0.0.0 --port 30000

z.ai's own production serving is an SGLang derivative on Chinese chips, with the model's MTP head, W8A8 weights, and mixed INT8/FP8/BF16 cache quantization, per the launch post. That is the design target for this model: many concurrent sessions, each with a cheap cache.

On a Mac

Two paths, one of them not ready. The GGUF path is the fork-branch llama.cpp build above with Metal, and it works at any size the machine holds; count unified memory at about three quarters usable, so a 128 GB machine is a 96 GB machine, which means the 1-bit tier and a short context. The native path is mlx-vlm, which merged glm5_next support on August 26 (PR #2030). What it needs is a converted checkpoint, and as of this writing I could not find a published mlx-community conversion of Flash the way there is for the full GLM-5.3, so you convert it yourself:

pip install -U mlx-vlm
python -m mlx_vlm.convert \
  --hf-path zai-org/GLM-5.3-Flash-BF16 \
  --mlx-path ./glm-5.3-flash-4bit -q --q-bits 4

python -m mlx_vlm.generate \
  --model ./glm-5.3-flash-4bit \
  --max-tokens 4096 --temp 1.0 --top-p 0.95 \
  --prompt "Explain the bug in this screenshot" --image shot.png

A 4-bit MLX conversion of 320B parameters lands near 180 GB before the vision tower, so this is a 192 GB or 512 GB Mac Studio path. The conversion itself needs the BF16 repo (641 GB on disk) or the FP8 one with a dequant step, and hours. Check the mlx-community search before you start; someone will have uploaded it by the time you read this.

Effort, sampling, tools

Reasoning effort is a three-position switch, and off is not one of them. The model card says reasoning_effort "accepts three levels: low, high, and max. It defaults to max if not passed (or if set to any other value)." Max is what every benchmark number was produced with, and it is also why the launch-week complaint threads are about verbosity: at max effort the model thinks at length before a two-line answer. For an agent loop where most turns are tool dispatch, set low explicitly and reserve max for planning turns. The GLM-5.3 line dropped the ability to disable thinking entirely; the full model's notes say a request that sets thinking to disabled fails.

Sampling. z.ai's evaluations used temperature 1.0 and top-p 0.95 with a 163,840-token generation cap, and unsloth recommends the same pair. Those are also the defaults in the commands above.

Thinking in the transcript. The chat template's clear_thinking defaults to false, which keeps prior reasoning blocks in the conversation. z.ai says to pass clear_thinking=true for chat. For agents, leaving it false costs context but keeps decision consistency across turns, which is the same trade the Qwen3.8 family exposes as preserve_thinking.

Tool calling. The template emits GLM-style tool calls and both serving engines ship a parser for it: the SGLang cookbook pairs --tool-call-parser glm47 with --reasoning-parser glm45, and llama.cpp handles it through --jinja, which you should treat as mandatory rather than optional. Hand-assembled prompts will silently break the call format.

Wire it into an agent

The whole point of holding 200 GB of weights locally is an agentic coder that does not meter you. Serve an OpenAI-compatible endpoint with any of the commands above and point the agent at it, exactly as in the GLM guide's agent section:

# llama.cpp, vLLM, and SGLang all speak the same shape
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"glm-5.3-flash",
       "messages":[{"role":"user","content":"Find why the tests hang and fix it."}],
       "reasoning_effort":"low",
       "chat_template_kwargs":{"clear_thinking":true}}'

# then in the agent's config:
#   base_url: http://127.0.0.1:8080/v1
#   model:    glm-5.3-flash   (llama-server ignores the name)

Three practical notes from the stealth-week reports. Put a loop detector in the harness: the doom-loop reports on the ox-alpha endpoint were about repeated identical tool calls, and a harness that exits on a repeated command signature, the way several HN commenters described building, is cheap insurance; the essay on harness engineering covers why that belongs around the model rather than in it. Give it vision on purpose: z.ai's training loop for frontend work has the model render its own output and inspect the screenshot, so an agent that can pass a browser screenshot back gets more out of this model than a text-only loop. And watch the effort setting per turn; a max-effort tool dispatch is the verbosity everyone is complaining about.

Benchmarks, honestly

Every number here is z.ai's own from the launch post, produced under z.ai's harness choices, and the comparison columns are labelled the way z.ai printed them. Two independent references exist: Artificial Analysis scored the API at 57 on its Intelligence Index at $0.045 per task, and the Hugging Face card lists a community-run Terminal Bench 2.1 of 84.3 that matches the vendor row. Wins and losses, selected:

Read it plainly. Against its own predecessor the jump is large everywhere, and on the two agent rows z.ai leads with, Toolathlon and AutomationBench, it is ahead of the closed models in its own table. On terminal coding it sits a few points behind the frontier, on repository-scale generation it is well behind Opus 4.8, and on the video and perception rows the Gemini Flash it is priced against beats it clearly. The base-model table in the same post is the more honest signal for what you are buying: GLM-5.3-Flash-Base at 88.1 MMLU against GLM-5-Base's 88.3 with less than half the active parameters. That is the efficiency claim, and it is the one the architecture section explains.

The license split

Flash is MIT, the standard text, copyright Z.AI. The full GLM-5.3, whose weights followed on August 28, is not: it ships under a custom "GLM-5.3 License" that is permissive for almost everyone but requires any model-as-a-service provider with more than $10 billion of revenue in any twelve months to complete a z.ai security review before commercial use. z.ai's stated reason for the two-week delay on the big model was safety evaluation and hardening of its cyber capabilities, and the license clause reads as the commercial half of the same decision: the hyperscalers that would host it get a gate. For a local run of Flash none of this applies. Verify the license file of whichever checkpoint you actually download, because the two repos sit side by side and the difference is one clause.

Rough edges, day eight

• No tagged engine has it. llama.cpp master, vLLM 0.28, and SGLang 0.5.18 all predate the architecture. You are building a PR, installing a PR head, or running a vendor image. The board above lists which.
• Three llama.cpp implementations. #27754, #27752, and #27773 name the architecture the same way but are not guaranteed to load each other's converted files. Unsloth's GGUFs were built against the fork branch; use it, not a random PR.
• CUDA flags on the PR build. Reports on the PR thread say NVIDIA_TF32_OVERRIDE=0 and -fa off are needed for correct output on some cards; Turing cards hit a SOFT_MAX failure (#28144); an illegal memory access report (#28282) is open. Unverified here.
• "Flash" is 93 GB. There is no 24 GB story. The Reddit line "neither Flash nor Air" is the right expectation for anyone coming from GLM-4.5 Air.
• Max effort by default. Verbose by design unless you pass reasoning_effort explicitly, and thinking cannot be switched off.
• Hopper needs a BF16 KV cache in vLLM, per the recipe; the FP8 cache path is for Blackwell.
• Vision needs the projector. The main GGUF ignores images without --mmproj.
• Context claims. The config says 1,048,576. z.ai's own evaluations ran at up to 300K with a context management strategy. Plan around what your memory holds, which the planner tells you, not the config.

FAQ

Is this the same model that was ox-alpha?

Yes. z.ai says it tested GLM-5.3-Flash anonymously as ox-alpha on OpenCode and OpenRouter before the reveal, and that the stealth traffic ran on Chinese chips. The reviews from that week are reviews of this model at whatever settings z.ai chose for the endpoint.

Why is the KV cache small if the model is huge?

Only 11 of 45 layers keep a per-token cache, and each keeps one 512-value latent per token. The other 34 layers are linear attention with a fixed state. That is 11 KB per token against roughly 90 KB per token for the full GLM-5.3; the ledger figure computes it. The mechanics of latent attention and of linear attention are in the transformer guide.

What is the smallest machine that runs it?

A 128 GB unified-memory Mac or a single DGX Spark at UD-IQ1_S with a short context, or a 24 GB GPU with 256 GB of system RAM and expert offload. Both are slow and both pay the 1-bit quality tax. A 192 GB box at IQ3 or IQ4 is where it starts to feel like the API model.

Which quant?

UD-Q4_K_XL if it fits (about 200 GB). It is unsloth's default and their retention figure for it is 92%. Below 3-bit, expect more tool-call formatting errors in agent loops.

Should I wait?

If you want a stock build, yes, probably weeks: three llama.cpp PRs, one vLLM PR, and one SGLang PR need to land and be tagged. If you have the memory and are comfortable on a branch, it runs today, and the DGX Station and RTX PRO 6000 reports above say it runs well.