Building an eval framework after my benchmark flipped

There are two mainstream ways to run a mid-size LLM locally on a Mac: llama.cpp (C++, broadly compatible, works everywhere) or Apple’s newer oMLX (optimized for Apple Silicon). Same model weights, different runtimes. Which one is faster depends on the model, and hobbyists benchmark this constantly.

In early June, I ran a benchmark that said llama.cpp was faster for certain models and I published the writeup. Three weeks later, I ran another benchmark that said oMLX was 46 percent faster with Qwen3-14B and 14 to 21 percent faster with the other mid-size models. So this caused me some concern. I don’t expect the benchmarks that I perform on a single M1 macbook to be scientific or anything, but they should be actionable and reliable to some degree.

Digging into the reversal, I found three separate methodology issues had pushed the June run in one direction: the workload was prefill-heavy with tiny completions which starved oMLX’s decode advantage, the models weren’t warmed up consistently, and the qwen3-14b judge stayed resident in unified memory while other models were being measured.

The reversal was caught by luck: I happened to run a new bench, spotted a discrepancy against the old one, and started looking for an explanation. From investigating this further, I realized it was going to be too time consuming to find the root of the issue. There were too many variables (checking if warmups were correctly ran, prefill-heavy corpus, etc..) and not enough ways to track down each. I needed to stop wasting time with my benchmark scripts, and graduate to something more rigorous.

So I self-hosted LangFuse (an open-source LLM observability tool) on my home Kubernetes cluster, wired the local API gateway to log every request through it, and wrote a small Python harness that turns a benchmark into a versioned experiment. Dataset with a SHA, git-clean check, mandatory warmup pass, structured CSV, and a LangFuse trace for every completion. The point of my earlier benchmarking scripts was mostly educational. To get some insight into how I could confidently measure and choose a model to run on my local hardware, and I achieved that.

For the first real use of the new framework, I just re-ran the last one. Same 30 prompts, same eight arms (four models times two runtimes), the machine was idle both times (everything closed except Ghostty + Herdr). If the numbers match my latest benchmark then I have some confidence that this is working.

Three of four results reproduced within (my) tolerance (±5%). Qwen3-14B did not: the new framework measured a 30% oMLX advantage, not the 46% my last benchmark had shown. But still a decisive increase that backed up the claim that my earlier benchmarks were flawed.

Either way, the conclusion here is that I’m moving in the right direction in terms of rigorous testing and ability to diagnose more effectively.