back to AI portfolio

Zero-shot voice cloning on Apple Silicon

Jun 2026

TL;DR

  • Replaced a planned Piper fine-tune loop with zero-shot F5-TTS-MLX for the faceless video pipeline’s narration stage. No training dataset required.
  • A single 12-second reference clip + exact transcript clones my voice well enough for narration work. Total prep time: ~5 minutes (one take + a one-line ffmpeg loudness-normalize + writing down what I said).
  • Default F5 params (euler/32) sound robotic. --steps 64 --method midpoint is roughly a 5x perceived improvement. --method rk4 is audibly indistinguishable from midpoint at ~2x the wall-clock, a trap if you go looking for more quality.
  • Wall-clock cost is ~19x real-time on the M2 Pro. A 5-minute narration takes ~95 minutes to generate. That sets the per-video latency floor for the entire pipeline and shaped the background worker’s concurrency design.

What I built

The voice stage of a faceless video pipeline I’m building, a multi-stage system that takes raw recordings (screen captures, hands-on builds, slide-driven explainers) dropped into Google Drive and produces publish-ready video distributed across YouTube, TikTok, Instagram Reels, and LinkedIn. The voice stage takes a generated narration script + a reference clip of my voice, and produces a WAV in my voice ready to be muxed into the rendered video.

The original plan was Piper: record 50 to 100 sentences as a training dataset, fine-tune a custom .onnx voice on a GPU, then run cheap inference forever. Piper produces excellent voices and runs at hundreds-of-X-real-time on a Raspberry Pi, but the training step is real work and locks the voice in.

I swapped to F5-TTS-MLX, a flow-matching transformer model running on Apple Silicon via the MLX framework. It’s zero-shot: hand it a clean reference WAV + the exact transcript of that reference + the new text to synthesize, and it generates the new text in the voice of the reference. No training step at all.

What worked, what didn’t

What worked

  • A 12.24s dedicated reference recording. I tried slicing the reference from a longer 2:09 archive recording first; quality was lower than re-recording a dedicated calm-technical take. The reference content matters because F5 inherits cadence and energy, not just timbre. The dedicated take described a Ruby agent project I’m working on, narration-shaped subject matter.
  • Loudness-normalizing the reference from -34.7 LUFS to -18.9 LUFS with ffmpeg -af loudnorm=I=-18:TP=-2:LRA=11. F5 inherits loudness directly from the reference; without normalization the cloned narration would have been whisper-quiet at normal playback levels.
  • --steps 64 --method midpoint on the ODE solver. Default euler solver with 32 steps sounded clearly robotic. Switching to a 2nd-order solver and doubling the steps was the production sweet spot, meaningfully more natural for ~4x the wall-clock.
  • Documenting the recipe the moment it landed. Future automation steps inherit the recipe plus the three gotchas below without re-discovering them.

What didn’t

  • The default duration predictor. Without an explicit --duration argument, F5 emits ~0.08s of silence. The duration predictor model is supposed to estimate this from the text, but it failed cold. Rule baked into the pipeline runbook: always pass --duration explicitly.
  • My first guess at what --duration meant. For a 6-second target sentence I passed --duration 6 and got 0.04s of silence. Turns out --duration is total audio length including the reference, so for a 12.24s reference + 6s target you pass 18.24, not 6. The duration argument must be the total output length, so I now add the reference clip length to the estimated speech length of the target script.
  • --method rk4. Tried the 4th-order solver expecting another quality bump beyond midpoint. Audibly indistinguishable, ~2x the wall-clock. Diminishing returns hit hard at this point on the solver-order axis.

What I learned

  • Zero-shot voice cloning has crossed the threshold where it’s a viable production replacement for fine-tuned voices for narration work on Apple Silicon. The cost is wall-clock at inference time, not training time upfront. For a one-person pipeline that wants to iterate on the voice (“re-record after 6 months”), the trade is unambiguously good.
  • Reference quality > reference length. A 12s clean take with matching emotional energy beat a longer, less-focused recording. F5’s failure modes (whisper-quiet, robotic, drifting) all map back to reference quality.
  • Document the param-space gotchas at the moment of discovery. I now have three rules baked into the pipeline runbook that future me won’t re-discover. The cost of writing them up at discovery time was ~5 minutes; the cost of re-discovering them later would have been an hour minimum.
  • Wall-clock budget is a real architectural constraint. ~19x real-time means voice generation is the single biggest wall-clock item per video. That’s the reason the background worker serializes voice generation, alignment, and final rendering on a single GPU lane rather than trying to parallelize them.

Related