MLX vs llama.cpp on M1 Pro (bench v1)

TL;DR

• What: Side-by-side eval of llama.cpp and MLX (via oMLX) as the on-prem inference backend for my homelab: coding daily-driver, Obsidian vault chat, long-context planner, multimodal experiments. Four-metric weighted scoring (quality 0.40, decode tok/s 0.25, TTFT 0.20, peak RSS 0.15) with blind quality grading.
• Outcome: Zero models met the migration threshold. All six evaluable classes plus DeepSeek-R1-32B verdict to stay on llama.cpp. The most extreme gap was Mistral-Nemo TTFT at 89 ms vs 1,687 ms (~19×).
• Why interesting: The eval started as a tok/s benchmark and ended by surfacing a silent audio bug in oMLX: every audio clip returns the same hallucinated English sentence with HTTP 200 and no diagnostic. A migration based on throughput posts alone would have silently lost capability.
• Lesson: Weight quality higher than throughput when the decision is “switch backends,” and blind-score every quality judgment. Without the 40% quality weight, two models would have flagged “migrate” on speed and lost real capability silently.

What I built

The question was simple: should I move my self-hosted LLM stack (coding daily-driver, Obsidian vault chat, long-context planner, and Gemma 4 multimodal experiments) from llama.cpp to Apple’s MLX framework? MLX benchmark posts had been claiming significant wins on Apple Silicon, and the current llama.cpp setup behind a LiteLLM proxy works fine but isn’t the point of comparison those benchmarks use. I wanted to know whether the gains were real for the workloads I actually run.

The eval rig ran side-by-side with the production stack, never interrupting cluster traffic. Both backends sat behind the same LiteLLM proxy, so the harness measured the end-to-end path the cluster uses, not raw server throughput. Per-model llama-server instances were started and stopped on demand by a small shell script because 32 GB of RAM wouldn’t hold them all resident; oMLX served all MLX models from a single process with LRU eviction. A small Python harness wrote one CSV row per streamed request, capturing TTFT, decode tok/s, peak RSS, wall time, and both backend versions.

Three methodology calls mattered. First, four weighted metrics with quality at 0.40: a faster wrong answer is worse than a slower right one, and the weight explicitly prevents “MLX wins on tok/s” from auto-meaning “migrate.” Second, blind quality scoring: the scoring interface hides which backend produced each completion until after the rubric score is submitted, preventing me from unconsciously favoring whichever backend I wanted to win. Third, matched-pair sweeps with per-cohort orchestration so the RAM ceiling never got violated; one non-production server resident at a time. Multimodal evaluation used OpenAI-format input_audio and image_url content types, scored with jiwer-WER against ground-truth transcripts for audio and the same blind rubric for text and image.

What worked, what didn’t

• Worked: Six matched model pairs, 780 measurement rows, production stack untouched throughout. All six classes plus DeepSeek-R1-32B verdict to stay on llama.cpp. Qwen3-14B (coding) warm TTFT 318 ms vs 3,180 ms (10×). Qwen3-8B tied at gap 0.000. Mistral-Nemo TTFT 89 ms vs 1,687 ms (19×, the most extreme in the eval).
• The audio finding: oMLX 0.3.12, despite bundling mlx-audio, returns the same canned hallucination for every audio clip: a fixed English sentence regardless of input. Diagnosis is a mel-filter configuration mismatch in oMLX’s preprocessing producing degenerate features, so the model falls back to a generic completion. Failure is silent: HTTP 200, plausible text, no diagnostic to the client. llama.cpp transcribes the same clips at WER ~0.15 (~85% accuracy). A reproduction + diagnostic was drafted as a GitHub issue against the oMLX repo.
• Honest concessions: RSS systematically under-reports MLX because Apple’s unified-memory model parks Metal buffers under wired_count, not the process RSS field psutil samples, so the 0.15 RSS weight effectively favored MLX by an unknown margin. A proper workaround would sample vm_stat wired-page deltas; logged as follow-up. Sample size is also small (1 to 7 quality scores per text combo, 15 reps per audio combo), directional rather than statistically significant.
• Excluded by design: Single-stream measurement only; oMLX claims good multi-request behavior, but verifying that is a separate eval. Quant parity is also a polite fiction: Q4_K_M (llama.cpp) and MLX -q4 (gs=64) have different bit-budgets and error distributions. I judged each backend at its “recommended 4-bit” rather than normalizing the difference away.

What I learned

The framework that made the eval credible was the 40% quality weight plus blind scoring. Without either piece, the verdicts would have drifted. With both, the speed gaps that looked decisive on paper (10×, 19× TTFT) still didn’t justify a migration because the quality side didn’t separate the backends enough to matter. I predicted “partial migration, multimodal only” before any data came in; the data inverted that. I’d reach for the same framework for the next backend decision and trust the data over the prior.

The audio bug is the meta-lesson about benchmark-only evaluation. Throughput posts wouldn’t have caught it. The MLX path returns plausible text fast, and only running real audio through the production code path with WER scoring against ground truth made the failure visible. The scope of the bench mattered more than the metrics in the bench.

One scope caveat worth being explicit about: this eval was on an M1 Pro with 32 GB. MLX’s gains scale with memory bandwidth, and a higher-bandwidth Apple Silicon part (M3 Max, M4 Max, M-Ultra) could plausibly flip multiple verdicts. The conclusion is “stay on llama.cpp for this hardware,” not “MLX is worse.”