# Local Model and Inference Guides by Rohit Ghumare

Canonical URL: https://rohitghumare.com/guides/

Last updated: 2026-08-22

Hands-on guides for running open-weight language models on local hardware and understanding the systems that serve them. Model guides cover engine choice, memory sizing, quantization, tool calling, OpenAI-compatible endpoints, and known failure modes.

## Model guides

- [Muse Glimmer](https://rohitghumare.com/guides/muse-glimmer/): the 30B agent model, 24 GB packing, GGUF sizes, DFlash draft model, prompt template, and measured results.
- [DeepSeek V4 Flash](https://rohitghumare.com/guides/deepseek-v4/): interactive quant selection, RAM fit, rewritten commands, and QAT-aware sizing.
- [Kimi K3](https://rohitghumare.com/guides/kimi-k3/): dynamic quants, mixture-of-experts CPU offload, and multi-GPU serving.
- [Qwen3](https://rohitghumare.com/guides/qwen3/): dense and mixture-of-experts variants, thinking mode, tool calling, and quant selection.
- [gpt-oss](https://rohitghumare.com/guides/gpt-oss/): 20B and 120B models, MXFP4, Harmony format, reasoning effort, and engine setup.
- [DeepSeek V3.2 and R1](https://rohitghumare.com/guides/deepseek/): consumer-GPU distills, full-model dynamic quants, and expert offload.
- [Gemma 3](https://rohitghumare.com/guides/gemma-3/): dense models from phone to workstation, 27B setup, and image input.
- [Mistral](https://rohitghumare.com/guides/mistral/): Mistral Small, Large 3, Devstral, consumer GPUs, coding agents, and multi-GPU use.
- [Llama 4](https://rohitghumare.com/guides/llama-4/): Scout and Maverick, multimodal input, large context, Mac memory, and GPU offload.
- [GLM 4.6 and 5](https://rohitghumare.com/guides/glm/): mixture-of-experts models, single-card offload, and coding-agent integration.

## Engine internals and hardware

- [Inside the Transformer](https://rohitghumare.com/guides/inside-the-transformer/): attention built from a masked average, architecture changes through 2026, chat, sampling, and serving math.
- [Inside the vLLM Engine](https://rohitghumare.com/guides/inside-vllm/): the paged block pool, scheduler, two-process loop, speculative decoding, and grammar masking.
- [What Actually Fits on Your Machine](https://rohitghumare.com/guides/hardware/): an interactive model-fit calculator for Macs, GPUs, RAM, quantization, KV cache, and offload.

## Selection notes

- Start with the hardware guide when the available RAM or VRAM is the main constraint.
- Choose a model guide when the model family is already known.
- Use Inside the Transformer for architecture foundations.
- Use Inside the vLLM Engine for production serving internals.
- Check each guide's cited model card and engine documentation before relying on a version-sensitive command.

Related resources:

- Essay index: https://rohitghumare.com/blog/index.md
- Home: https://rohitghumare.com/index.md
- Agent instructions: https://rohitghumare.com/agent-instructions.md
- Sitemap: https://rohitghumare.com/sitemap.xml