ASR engines on Apple Silicon

TL;DR

• What: A 4-engine reproducible ASR benchmark on Apple Silicon (whisperx 3.8.6, parakeet-mlx 0.5.2, mlx-whisper 0.4.3, whisper.cpp 1.8.6) at two model sizes each, scored on Word Error Rate and Real-Time Factor against 16 LibriSpeech test-clean clips.
• Outcome: parakeet-mlx tdt-1.1b is the practical winner at 0.55% mean WER and RTF 0.055 (about 18x real-time). It ties whisperx large-v3 on accuracy (0.48%) at roughly 20x the throughput.
• Why interesting: mlx-whisper large-v3 disqualified itself on a single normal 12-second clip. The decoder looped “after the arrival of the mystics in the mystics turnouts” six times, drove a 164% WER on that one clip, and pulled the cohort mean from about 1.4% to 12.6%.
• Lesson: Same Whisper weights, different runtime, very different reliability. whisper.cpp large-v3 hit 0.87% mean WER on the same content. The runtime’s anti-hallucination heuristics matter as much as the weights they wrap.

What I built

A reproducible benchmark comparing four ASR engines at two model sizes each (8 cohorts total) against 16 LibriSpeech test-clean utterances, about 3 minutes of audio bucketed into short (3 to 7 seconds), medium (7 to 15 seconds), and long (15 to 35 seconds) durations. The goal was to settle whether to keep whisperx as my default transcription engine across the projects that need one, or switch.

The architecture made engine swaps cheap:

• Per-engine virtual environment. Each engine ran inside its own isolated Python virtual environment because their dependency trees are incompatible: torch CPU, MLX, and system Python cannot coexist in one environment.
• Standard transcribe contract. Every driver implemented the same simple contract: given an audio file and a model size, return the transcript text, per-word timing, SRT output, wall time, model load time, and engine version.
• Shell dispatcher. A thin shell dispatcher picked the right virtual environment and invoked the matching driver, which kept the runner Python-version-independent.
• Runner + scorer. The runner looped over every engine, clip, and repetition, wrote each transcript to disk, scored it with a Levenshtein-based WER on normalized tokens, computed RTF from wall time over audio duration, and appended one row per run to a results CSV. 128 rows after deduplication covered all 8 cohorts across the 16 clips.

The corpus came from openslr/librispeech_asr via HuggingFace datasets, bucketed by duration and resampled to 16 kHz mono.

What worked, what didn’t

Final ranking, sorted by WER mean:

What worked. parakeet-mlx is the clear story. The 1.1b TDT model is essentially tied with whisperx large-v3 on accuracy but runs roughly 20x faster because the MLX runtime targets the M-series GPU directly. The 0.6b variant is even faster (about 30x real-time) and still under 1.1% WER, which is good enough for bulk transcription where light post-processing can catch the remaining errors.

What didn’t work. mlx-whisper large-v3 produced a textbook Whisper hallucination on one clip. The audio said “The governor, on his part, becoming doubtful of the legality of employing Missouri militia to enforce Kansas laws, was also eager to secure the help of federal troops.” The model transcribed the first sentence correctly, then looped into “after the arrival of the mystics in the mystics turnouts after the arrival of the mystics in the mystics turnouts” for another 40 words. Single clip, 164% WER, but it pulled the cohort mean from about 1.4% to 12.6%.

The root cause is the well-known Whisper repetition-loop failure mode. faster-whisper (via whisperx) applies more aggressive anti-hallucination heuristics (logprob thresholds, no-speech detection, condition-on-previous-text=False); mlx-whisper’s defaults do not. The fix is parameter tuning or post-processing to detect n-gram repetition and re-decode, but that is work I would rather not do when parakeet sidesteps the issue entirely.

What I learned

MLX-native models change the speed regime on M-series. parakeet at 30x real-time is not a marginal improvement over whisperx, it is a different kind of tool. Real-time captioning, batch transcription of large archives, and on-device pipelines all become tractable rather than aspirational.

Same weights, different runtime, different reliability is the meta-lesson. mlx-whisper and whisper.cpp both run OpenAI Whisper weights. whisper.cpp large-v3 hit 0.87% mean WER; mlx-whisper large-v3 hit 12.59% on identical content. The runtime’s decoding heuristics matter as much as the weights, and benchmark posts that omit the runtime story bury this.

Half the work in any WER benchmark is normalization. The first run had whisper.cpp at 25% WER because one CLI flag forced BPE sub-tokenization, splitting words like “Concord” into fragments. Removing that flag and fixing the scorer’s hyphen handling dropped whisper.cpp from broken to the third-best non-MLX option. A small rescore helper was worth writing on day one: whenever normalization changed, it re-scored every saved transcript and updated the results CSV in seconds instead of re-running a 2-hour benchmark.

Self-contained dispatch makes engine swaps cheap. The thin dispatcher boundary meant adding a fifth engine later, for example a new Voxtral release, would only require a new virtual environment, a new driver, and one new case in the dispatcher.

Decisions made

• Replace whisperx with parakeet-mlx tdt-1.1b as the default transcription engine in automated pipelines and future projects that need transcription.
• Keep whisperx as the word-timestamp gold standard for cases where forced alignment quality matters more than throughput (for example, very tight caption sync).
• Do not use mlx-whisper large-v3 without parameter tuning. The hallucination risk on normal content is real.
• Use parakeet-mlx tdt-0.6b for bulk archive transcription, where roughly 30x real-time turns “transcribe everything in the archive” from a multi-day job into a tractable evening.