How to choose LLM models for local inference: a simple decision-making tool for humans.

Thanks to developments in open source LLM, it’s now possible to run models in almost any computer. I run Qwen 3.5 4b in my laptop with only 6Gb VRAM and the results are surprisingly good, especially if I enable web search. On my desktop rig, I have the luxury to have 128Gb of unified ram and can run larger models like Qwen 3.5 122b or GPT-OSS 120b.

The challenge many users face is choice overload. For a given model, there are dozens of ‘flavors’ you can download from Hugging Face. I hope my simple tutorial can help people make fast and wise choices.

Decision-making made simple #

1. Choose model maker and architecture:
   E.g. Qwen3.5, GPT-OSS, Gemma, etc.
   How to choose? start by deciding what you want to do (write, code, chat with a companion, etc). Then search for the best model. Communities like Reddit’s r/LocalLLaMa are great to understand what are the latest and highest performing.

2. Calculate how many B (billion parameters) can fit into your memory The parameter count represents the model’s capacity to understand and reason. The higher the number, the broader its knowledge base and the deeper its ability to handle complex tasks (as a general rule). For local inference, the working assumption is that large models require a GPU for fast interactions. Look at the typical file sizes and see if they match your available VRAM. Using the Qwen 3.5 family: a 4B model fits into ~6GB of memory (VRAM or System RAM), while a 122B model requires ~128GB. Don’t get discouraged if your video card has little VRAM. A small model with internet access can achieve great results. That is my experience with Qwen 3.5 4B on my laptop.
   Also important: the model architecture has substantial impact on quality. For example a smaller 27B dense model can outperform a 32B Mix of Experts even if the latter is larger on paper.

3. Choose your quantization
   TL;DR: download Q4_K_M models.
   Quantization is to LLM what jpg is to images. It compresses parts of the model weights aiming to keep quality as high as possible.

• Q6_K / Q8_0: “Gold Standard.” Quality losses are almost imperceptible for real world use cases. But if you have the luxury of a large memory you can choose these.
• Q4_K_M: this is the pragmatic and consistent choice. It performs well across models and configs. It considerably reduces model weight size with little noticeable impact to quality.
• IQ4: “The Smart 4-bit.” Uses an “importance Matrix” to protect critical weights. In theory it performs better than Q4_K_M by ensuring highly important weights are not compressed. I say in theory because my experiments bring mixed results and personally default to Q4_K_M.
• MXFP4: “The Speed King” for some rigs. Great for throughput, but as research shows, it “crushes” fine details (like subtle sarcasm or complex formatting).
• IQ3_M / REAP: The “Emergency” options. Go below Q4, or use REAP models in cases where you need a large model but can’t fit into your memory.

4. Use known uploaders:
   E.g. lmstudio-community, unsloth, bartowski, etc.
   Reliable uploaders ensure you won’t experience major quality issues like hallucinations, LLM stuck in thinking loops, outputting garbage, etc.
   Read the model and uploader notes to check the optimal model loading parameters: temp, repeat penalty, etc. In models like Qwen 3.5, it’s very important to set configs like K_top or repeat_presence.
   Use ‘abliterated’ or ‘uncensored’ only for specific use cases. ‘Official’ models tend to work better and present less problems.

5. Test and refine
   If speed suffers (ie below >15tk/s) - seek speed optimizations like a lower quantization, or choose a MoE model vs a dense one.
   If a model has a -Thinking or -Reasoning suffix, it will be much slower but significantly smarter. Don’t use these for basic chat; use them for “hard” problems only. Most models can switch between thinking and no_thinking mode with prompts like /no_thinking or configs in the template.

Appendix: First Principles for LLM models: Small vs. Large, MoE vs. Dense #

A quick summary of model architectures and what to expect from them:

Small Models: Pros: Fast, efficient, low VRAM requirements, ideal for simple tasks. Cons: Limited reasoning ability, struggles with complex problems, lacks nuance.

Large Models: Pros: Powerful, nuanced, excels at complex reasoning, broader knowledge base. Cons: Slow, resource-intensive, requires significant VRAM, prone to overfitting.

Dense Models: Pros: Cohesive knowledge representation, consistent responses, easier to fine-tune. Cons: Scaling is limited by parameter count, less efficient for specialized tasks.

MoE Models: Pros: Scalable, efficient, excels at specialized tasks, better at handling long context. Cons: Fragmented knowledge representation, can be less consistent, requires careful routing.