Which GLM model can I run on a consumer GPU?

GLM-4.5 Air is the compact mixture-of-experts variant, with a small active-parameter count, so a 4-bit quant runs on a 24GB card paired with system RAM using expert offload, or on a high-memory Mac. The full GLM-4.6 and the GLM-5 series need offload on a big-RAM box or a multi-GPU server.

Is GLM good for coding agents?

Yes, the GLM models are strong agentic coders and tool users. Serve one with an OpenAI-compatible endpoint and point a coding agent at it, and the model drives file and shell tools well. Enable tool calling in your engine so the tool format is correct.

What license is GLM under?

The GLM open-weight models are released under the MIT license, which is permissive for commercial use. Confirm the specific checkpoint's card, since terms can vary by release.

Local model guide · updated July 2026 · 10 min read

How to Run GLM Locally

GLM from z.ai has become a favorite for local agentic coding: MIT-licensed mixture-of-experts models that punch above their active size and drive tools well. The compact GLM-4.5 Air runs on a single high-end card with offload, the flagship GLM-4.6 wants a big-RAM box, and the GLM-5 series is server-scale. This guide sizes each and runs them across llama.cpp and vLLM, with a focus on wiring GLM into a coding agent.

Machine · GLM-4.5-Air on a 24 GB card with system RAM

Load the weights, prefill a prompt, then decode one token. Each of the 23 cells stands for 2 of the 46 full-attention layers, and every one of them adds to the KV pool. The expert drawers sit in system RAM, as the llama.cpp flag puts them, and the router reaches across to open 8 of the 128. The shared expert stays on the card, because the flag matches only the routed expert tensors.

Architecture map · GLM-4.5-Airconfig.json

The compact GLM. Each token runs 8 of 128 routed experts plus one shared expert per layer, about 12B parameters.

46 layers, first to last, one cell each

46 full attention layers1 with a dense FFN

128 routed experts per MoE layer, 8 picked per token

Lit squares are the 8 picks for one example token plus 1 shared expert that always runs.

hidden 4,096 · 96 query heads · 8 KV heads · head dim 128 · context 131,072

KV cache: 46 layers × 8 KV heads × 128 × 2 × 2 bytes = 188,416 bytes per token in BF16

Source: zai-org/GLM-4.5-Air config.json

Air compact MoE 4.6 flagship MoE 5.x frontier ~753B license MIT

Which GLM

Model	Class	Runs on
GLM-4.5 Air	compact MoE	24GB GPU + RAM offload, or 64GB Mac
GLM-4.6	flagship MoE	big-RAM box with offload, or multi-GPU
GLM-5 / 5.2	frontier MoE	multi-node or large server

Air is the one to start with. It keeps a small active-parameter count, so it runs fast, and a dynamic 4-bit quant fits a single high-end card once you offload the experts to RAM. Step up to 4.6 or the 5 series when you have the memory and want the top of the quality curve.

GLM-4.5 Air on a GPU

Build llama.cpp (steps here), then serve Air with expert offload:

./llama.cpp/llama-server \
  -hf unsloth/GLM-4.5-Air-GGUF:Q4_K_M \
  --jinja \
  -ngl 99 \
  -ot ".ffn_.*_exps.=CPU" \
  -c 32768 \
  --host 127.0.0.1 --port 8080

On a 64GB or larger Mac you can drop the offload flag and keep the whole model in unified memory.

GLM-4.6 and 5 with offload

The larger variants follow the big-model pattern: a dynamic low-bit quant plus -ot ".ffn_.*_exps.=CPU" to keep experts in system RAM, matched to your combined VRAM plus RAM. Merge any multi-part GGUF with llama-gguf-split --merge. Expect the same memory-bandwidth-bound throughput as other 600B-plus-class models on a single-GPU-plus-RAM box.

./llama.cpp/llama-server \
  -hf unsloth/GLM-4.6-GGUF:UD-Q2_K_XL \
  --jinja -ngl 99 -ot ".ffn_.*_exps.=CPU" \
  --cache-type-k q4_0 -c 16384 --host 127.0.0.1 --port 8080

Serve with vLLM

vllm serve zai-org/GLM-4.6 \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser glm

Use the GLM tool-call parser named on the model card for your checkpoint, and set --tensor-parallel-size to your GPU count.

Wire it into a coding agent

GLM's strength locally is agentic coding, so the payoff is connecting it to an agent. Start the server, then point your coding agent at the local endpoint:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Refactor this function for readability."}]}'

Any agent or framework that accepts an OpenAI base URL will then run against your local GLM: set the base URL to http://127.0.0.1:8080/v1 and any string as the key. Make sure tool calling is enabled so the agent's file and shell tools format correctly.

Security. Bind to 127.0.0.1. Expose on 0.0.0.0 only behind an authenticating proxy.

Tool calling

Enable tools with the GLM parser in vLLM and --enable-auto-tool-choice, or --jinja in llama.cpp for the model's template. Because the endpoint is OpenAI-compatible, the agent passes its tool schemas unchanged, which is what makes GLM a clean drop-in for a local coding loop.

Troubleshooting

Out of memory. Offload more experts with -ot, drop to a smaller dynamic quant, quantize the KV cache, and lower context.
Slow on the larger variants. Expected on single-GPU offload; a few tokens per second. Add GPUs or RAM for speed.
Tool calls not parsed. Use the GLM parser in vLLM, or --jinja in llama.cpp.
Multi-part GGUF will not load. Merge with llama-gguf-split --merge or point at the first shard.

FAQ

Which GLM runs on a consumer GPU? GLM-4.5 Air, at 4-bit with expert offload on a 24GB card plus RAM, or on a high-memory Mac. The full 4.6 and the 5 series need a big-RAM box or a server.

Good for coding agents? Yes, GLM is a strong agentic coder. Serve it with an OpenAI endpoint and point your agent at it.

License? The open-weight GLM models are MIT; confirm the specific checkpoint's card.

Keep reading

Run Qwen3 LocallyThe everyman family, 4B to 480B, engine by engine. What Fits on Your Machine55 models, quant and KV math, an interactive fit calculator.

AI Engineering from Scratch free course The curriculum behind these pages: every core AI algorithm built by hand, 503 lessons across 20 phases, from the math to the tokenizer to the agent loop. Open source, learn at your own pace. Start at aiengineeringfromscratch.com →

Rohit Ghumare

CNCF Ambassador and Google Developer Expert. I write guides for running open-weight models locally and about the engineering under the AI agent stack.

All local model guides · Qwen3 · X