For AI tools, we need to increase focus on measuring user experience quality, and remove focus from synthetic benchmarks
Why benchmarks stopped predicting which model you’ll actually use. A Sunday afternoon with Opus 4.7 and Qwen 3.6 27B.
Synthetic benchmarks like SWE-bench, MMLU, GPQA, ARC, etc. get a lot of airtime in the community. Key players like Open AI or Anthropic publish their results around them. Enthusiasts add their own numbers from local inference experiments.
And it’s logical: humans are pattern-matching, competitive mammals that love having numbers to track performance and compare things to similar things.
But synthetic benchmarks miss to a large degree what really makes AI products good or bad: user experience quality. Often perceived as ‘I’m getting the answers I need, most of the time’.
Flagship AI tools seem to keep the user experience quality as secret sauce. Powered by impressive budgets in research and backed by robust Reinforcement Learning from Human Loop methodologies, SOTA tools like ChatGPT or Claude get most answers right, most of the time. Experience quality one of the key moats by which they keep on being leaders quarter by quarter.
That evidences a gap in everybody else but flagship AI providers. How do we know if the experience quality is good, and what can we do to start measuring it?
Some of the questions that illustrate experience quality are:
- Did the LLM response match what the user actually needed in that moment, or was the answer so off that it sent the user off-track, creating unecessary ’extra homework'?
- Did the LLM break trust in such a way that the user is gone forever to never come back to the provider?
- Is the LLM wasting tokens and giving poor quality answers in ways that annoy users at a low level, and that annoyance becomes big enough for the user to switch providers at some point in time?
- Did the model read the room or was it completely tone-deaf?
These are questions that are hard to answer because they rely on something as complex and subjective as human perception. That’s precisely why we should start tackling them and relying less on synthetic benchmarking.
What synthetic benchmarks won’t capture: a sunday story about SOTA vs small models. #
I’ve been stress-testing Claude Opus 4.6 over 1.5 months for every use case in the book. That allows me to compare response quality vs my own models (Qwen 3.5/3.6 family, Gemma 4, etc.) and catch where SOTA gets it right, where the self-hosted model shows cracks, etc.
This Sunday I was stress testing the ‘personal companion’ side of things. I shared a small Sunday joy, an aromatic herbal infusion created without recipe. Just to see what LLMs would do to uplift that joy. To my surprise, Claude Opus 4.7 responded with a health warning about pomelo and CYP3A4 enzyme inhibition.When I tried to hint it towards another kind of answer, it didn’t read the room. Persisted about pomelo warnings like if I was home cooking uranium.
To make things worse, Thinking budget was the highest. My memory was full of data and Opus had access to all my chats where the intention of home cooking was clear, and the kind of answers are expected were clear.
Out of curiosity, I sent the same message to Qwen 3.6 27B running locally on my desktop. No thinking budget. No memory and no context. No project files. 200 tokens of “I made a thing in my kitchen lol.”
Qwen gave the answer I was looking for. Just celebrating the small ritual and giving insights about how the infusion is refreshing for different senses.
I tested Gemma 4 31B too, and Gemma also nailed it.
Two small open-weight models, one of them at less than 1/30th the parameter count of the flagship, read the room better than the SOTA model with full context and thinking enabled.
Small insignificant test, I know. I’m not pretending this is some sort of breakthrough moment. But it’s an example that illustrates some interesting patterns I’ve been observing over the past months. Namely:
- Context size and memory can actually hurt the quality of results. More is not always better, as counter-intuitive as this sounds.
- Larger models don’t always provide better answers than smaller models. The size moat is only effective for a few nuanced use cases.
- Past a certain level of capability, small models can meet SOTA quality for a range of use cases. Small models with the right context and set up can surpass generic SOTA large models with large computation budgets.
- Pressure on SOTA providers to mitigate legal risk is leading to poorer quality experiences, with refusals and ’legal warnings’ gettin into the way of what users try to accomplish.
The above is consistent with more serious use cases like coding, admin support, strategic thinking pattern, etc.
Past a certain level of competence, there’s disminishing returns for parameter size, context size, and other metrics that traditionally were the focus. #
The synthetic benchmark obsession made sense in 2023 and 2024. Capability was genuinely the bottleneck. Models couldn’t reason, couldn’t follow instructions, hallucinated constantly. More parameters meant fewer of these failures and the numbers tracked something real.
In 2026 the bottleneck has shifted. Frontier models reason well, hold a million tokens of context, use tools, run agentic workflows. The remaining bottleneck is calibration: weighting the current input correctly given everything else in scope. Calibration doesn’t scale linearly with parameter count or context size. Sometimes it inversely correlates.
The practical takeaway is uncomfortable for the lab-vs-lab benchmark wars: two models with very different parameter counts can deliver indistinguishable user experience for most tasks. A 27B running locally with no monthly fee, no rate limits, no privacy concerns, no internet dependency, is the better tool for most of what you’ll do in a given week. The flagship still wins on the hard 20%. Most tasks aren’t in the hard 20%.
If you already run local models, you’re probably underweighting what you have. Most of what people attribute to “small model limitations” is configuration, prompts, or expectations. A well-configured Qwen 3.6 27B isn’t a downgrade from frontier models for most real work. It’s a different tool with different trade-offs, and for a wide range of practical use the trade-offs favour the local model.
The benchmark that matters here is the felt sense of “this answer matched what I needed.” That benchmark belongs to the user, not to the lab. It won’t appear on a leaderboard. The job is to develop your own taste for it.