llama.cpp vs oMLX on M1

Jul 2026

TL;DR

Bench-v6-dspark: 5-arm speculative-decoding comparison on Qwen3-8B-4bit. 600 runs, 4.3 hours. Plain decode beat every drafter arm.
Bench-v6.1: 2-arm plain-decode comparison (llama.cpp vs oMLX) across four daily-driver models. 968 runs, 4.8 hours. oMLX faster on three, wash on one.
Interpretation: on M1 at 4-bit, decode is closer to compute-bound than memory-bandwidth-bound, so batched-verify isn’t the free lunch the spec-decode literature assumes. Plain oMLX is the faster backend on this hardware and this workload for the Qwen and Gemma daily drivers.

Setup

Hardware: 32 GB M1 (see the portfolio hardware note).
Corpus: 30 hand-picked prompts frozen at corpus/prompts.jsonl. 10 in the style of HumanEval, 10 in the style of GSM8K, 10 open-ended chat.
Each backend on its own port (8091-8094) so the production LiteLLM stack stayed live during the sweep.
Harness hits each server directly with httpx to keep the proxy hop out of TTFT.
Warmup pass per server before the timed loop.
CSV flushed per generation.
No LLM-judge process running during the sweep; a resident judge would compete with the model under test for unified memory.

Bench-v6-dspark: speculative decoding on Qwen3-8B-4bit

Five arms, all targeting Qwen3-8B-4bit. Median throughput:

arm	median tokens/s
oMLX (plain)	29.34
llama.cpp (plain)	23.29
llama.cpp + Qwen3-0.6B GGUF drafter	18.29
mlx-dspark (DSpark drafter)	16.21
mlx-dflash (DFlash drafter)	7.62

Drafter acceptance for the three spec-decode arms:

DSpark: 2.29 tokens accepted per drafter invocation.
DFlash: 2.50 tokens accepted per drafter invocation.
llama.cpp drafter: 0.58 median acceptance rate (from its own server logs, n=122).

What this means. The drafters aren’t broken. They accept at rates comparable to what the upstream projects report. Wall-clock throughput drops anyway because on M1 at 4-bit the target model’s decode isn’t memory-bandwidth bound the way it is on a datacenter GPU. Apple’s matrix kernel is tuned for 4-bit, so verifying 5 tokens costs roughly 5x the compute of verifying 1. The batched-verify step isn’t nearly free, the drafter’s cost has to fit under a tighter margin, and it doesn’t. Plain decode wins every matchup.

Bench-v6.1: llama.cpp vs oMLX across four daily drivers

Two arms per model (llama.cpp plain vs oMLX plain), same corpus and warmup discipline. Median throughput:

model	llama.cpp tokens/s	oMLX tokens/s	delta
Qwen3-14B	11.46	16.76	+46%
Gemma4-e4b	31.75	38.41	+21%
Gemma3-12b	15.06	17.23	+14%
Mistral-Nemo-12b	23.84	23.34	-2%

Qwen3-8B isn’t in v6.1, but its plain-arm medians are in v6-dspark: llama.cpp 23.29, oMLX 29.34, delta +26%.

What this means. On this hardware and this corpus, oMLX decodes faster than llama.cpp on both Qwen models and both Gemma models. Mistral-Nemo is a wash. The gap widens on larger models (Qwen3-14B +46%) and narrows on smaller ones (Gemma3-12b +14%). If your daily driver is Qwen or Gemma, oMLX is the faster backend on this workload. If it’s Mistral-Nemo, either is fine.

What I learned

Speculative decoding is pitched as a strict win in the literature, and the acceptance numbers here confirm the drafters are doing their job, but the wall-time result runs the other direction. The hardware assumption underneath the technique (target-model decode is memory-bandwidth bound) doesn’t hold at 4-bit on Apple Silicon. A technique that wins on an H100 can lose on an M1 for reasons that never make it into the abstract.
A backend ranking is a workload ranking. Prompt length, completion length, and prompt mix drive the result more than the backend name. This corpus reflects what I route through LiteLLM day-to-day; a different mix would rank differently.