MLX vs llama.cpp on K8s triage (bench v2)

TL;DR

• What: Round 2 of the local LLM backend eval. 13 model and backend combinations run against 20 hand-written Kubernetes triage scenarios (260 rows total), with qwen3-14b-tuned as the LLM-as-judge. Composite quality is format-gated and built from deterministic verifier passes, action safety, and judge rubric scores.
• Outcome: Stay on llama.cpp confirmed. llama matches or beats MLX on quality in 5 of 6 fairly-comparable model families and is 10 to 64 percent faster on throughput in the same 5. Default cluster model: granite-4.1-8b-tuned at 0.79 composite quality with a 12 second median wall time.
• Why interesting: The first eval flagged the headline (no migration); this one stress-tested it on a workload that mirrors how the cluster actually uses these models. It also surfaced operational fragility in oMLX, a thin MLX serving wrapper I maintain (two Metal-OOM crashes from mid-cohort swap-evict races) that doesn’t appear in throughput-only benchmarks.
• Lesson: Production-shaped scenarios surface failure modes that synthetic benchmarks miss. The two crashes, three measurement bugs, and the granite-8b discovery all came from running the eval through the real cluster path, not from running models in isolation.

What I built

The first eval established that MLX did not justify migration on general workloads. The open question was whether the verdict held on the specific shape of work the cluster runs most often: short structured Kubernetes triage prompts (0.5 to 8 KB snapshot dumps, structured answers, deterministic verification possible). Bench-v2 was built to answer that.

The setup ran 13 model and backend combinations across 20 hand-written K8s triage scenarios (single rep per cell, 260 rows). The scenarios target a known fault in a vcluster sandbox running K8s 1.35.0, and each model is asked to identify the fault and propose a safe action. Scoring is format-gated: format compliance must be strict before the composite is computed; the composite then averages deterministic pass, action safety, and three judge-derived scores. The judge model is qwen3-14b-tuned running on llama-server.

Three sweeps ran in sequence. A 13-cohort all-in-one crashed silently because oMLX died before launch, which surfaced the need for an oMLX health precheck and PYTHONUNBUFFERED=1. A llama-only resume ran clean. An MLX-only resume used a fresh oMLX process per cohort to dodge the swap-evict OOM that hit on May 31 and again on June 3 during mid-cohort model evictions.

What worked, what didn’t

• Worked: Verdict held across the production-shaped workload. In 5 of 6 fairly-comparable families, llama matched or beat MLX on quality and was 10 to 64 percent faster on throughput. Per family: qwen3-14b tied at 0.40 vs 0.39 (llama 10% faster), qwen3-8b won on quality 0.36 vs 0.26 (MLX 11% faster, did not compensate), mistral-nemo-12b tied at 0.60 vs 0.59 (llama 20% faster), gemma3-12b tied at 0.81 vs 0.79 (llama 51% faster). No configuration where MLX is the right call on this workload.
• The granite-8b discovery: granite-4.1-8b-tuned landed at 0.79 composite quality (tied for second overall) with a 12 second median wall time, the best speed-at-quality tradeoff in the lineup. It was added late as a v2 addition; in hindsight it should have been a Phase 0 baseline.
• Operational fragility in oMLX: Two Metal-OOM crashes during the eval, both caused by swap-evict races during mid-cohort model swaps. The fresh-oMLX-per-cohort discipline works but is overhead that llama-server doesn’t impose. Production use would need either single-model invocations or the per-cohort restart discipline as standing practice.
• Three measurement bugs that bound the conclusions but don’t change the headline:
  • Granite-4.1-30b: judge integration fails 40 of 40 calls with judge-unavailable errors for both backends, suppressing composite quality by about 0.4 points purely from missing judge data. Deterministic pass is 90 percent on both, the best of any model. Granite-30b may actually be the highest-quality model in the lineup.
  • Gemma4-e4b on llama-server: 14 of 20 scenarios return unparseable format compliance, vs 95 percent strict under MLX with the same weights. Almost certainly a sampling, chat-template, or EOS-token mismatch in the llama-server invocation. The gemma4-e4b row in the per-family verdict is not a backend comparison until that’s fixed.
  • Three scenarios fail at 0 to 23 percent deterministic pass across all 13 models while remaining 100 percent action-safe: the missing-ConfigMap scenario (0%), the missing-image-pull-secret scenario (0%), and the wrong-verb RBAC scenario (23%). When every model produces well-formed answers that the verifier rejects, the truth definition is more likely wrong than the models. Likely bugs in the deterministic verifier.
• Excluded by design: Single rep per cell, deterministic and quality bucket only (no throughput characterization). The plan called for 5 reps in the throughput bucket; that’s separate work. The judge was not held out (qwen3-14b-tuned’s own scores include judge calls on its own outputs), acceptable for a relative comparison.

What I learned

The mental model shift from this round was operational, not technical. MLX’s theoretical prefill and TTFT edge did not show up as throughput wins on this prompt shape (short structured prompts, short structured answers). At matched 4-bit quant, quality is genuinely indistinguishable across 5 of 6 families within ±0.10 composite. The “Q4_K_M vs MLX -q4(gs=64) is a polite fiction” caveat from the plan held: the differences are real but small.

The operational fragility finding is the one I would not have predicted. Both crashes came from swap-evict races during model rotations, which is the path a real serving setup exercises but a throughput-only post never does. The granite-8b find was a happy accident: added late as a v2 baseline, it won the speed-at-quality leaderboard. The lesson worth carrying is to include smaller, less-hyped models as baselines; they sometimes lap the field.