back to AI portfolio

Where a video pipeline spends its time

Jun 2026

TL;DR

  • Built a 6-step Windmill benchmark flow that mirrors the production draft_from_drive flow minus the Drive fetch, deployed it to the cluster, and swept 16 clips x 3 reps = 48 runs end-to-end on Apple Silicon.
  • Median wall 10.90 s on 10.95 s of audio, RTF 1.00. 48/48 successful. Where bench-v4-asr measured parakeet-mlx tdt-1.1b standalone at RTF 0.055, the production flow runs at RTF 1.0, an ~18x tax for going from “engine call” to “production pipeline.”
  • Three ffmpeg-bound steps each eat 25 to 28% of total wall: transcribe (Parakeet-MLX, 28.5%), assemble_draft (h264_videotoolbox, 25.3%), burn_captions (libass, 25.3%). Normalize and auto-editor split the remaining ~18%.
  • Windmill itself is not the bottleneck. SSH transport + paramiko session setup + cluster scheduling totals 3.1% of median wall. The cost is the ffmpeg work the production pipeline does around the ASR call, not the orchestrator.

What I built

A reproducible benchmark that times the production video-creation-pipeline flow end-to-end. Successor to the earlier ASR engine bench: that one settled which ASR engine wins in isolation (parakeet-mlx tdt-1.1b), this one measures the production architecture that wraps it.

The video-creation-pipeline has been re-architected: HyperFrames-as-orchestrator is gone, replaced by a Windmill flow that runs in-cluster on Talos and drives the M1 over SSH via a Tailscale-routed Service. The production flow has six steps: fetch the source from Drive, normalize the clip, transcribe with Parakeet-MLX, run auto-editor silence trimming, assemble the draft with h264_videotoolbox, and burn ASS captions.

The bench at a glance:

  • Corpus: reused the 16 LibriSpeech test-clean WAVs (3 to 35 s) from the earlier ASR engine bench, wrapped in minimal MP4s (1920x1080 still frame + AAC audio, 30 fps CFR, explicit duration bound for less than 30 ms drift). No human labor; existing corpus plus one ffmpeg recipe.
  • Flow variant: copy of the production Drive-to-draft flow with the Drive-fetch step removed and the schema input changed from a Drive file ID to a path on the M1 worker. Steps 2 through 6 byte-identical to production. Deployed via a scoped wmill sync push so the benchmark add could not mutate production flows.
  • Trigger + collect: a shell loop runs the clips and repetitions serially because the M1 worker is single-worker, invokes each run through the Windmill CLI, and captures the top-level run UUID. A collector then queries the Windmill job API for each run to extract per-step durations and writes one CSV row per run with columns for the six steps, total duration, SSH overhead, and source audio duration.
  • Reports: an analysis script computes per-step median, p95, min, max, and share-of-total, flags outliers (there were none; every run stayed within 2x of the median), and writes a dark-themed HTML status report mirroring the previous benchmark’s template. A fixes report sits alongside.

What worked, what didn’t

What worked. The whole bench shipped clean (corpus, flow deploy, trigger, collect, report) in one session. The 48-run sweep finished in ~15 min unattended. Zero failures. The per-step timing extraction from Windmill’s job API was tighter than I expected: every step’s start and completion timestamps are captured to millisecond precision, and the gap between sum-of-steps and top-level flow duration cleanly surfaces the SSH/scheduling overhead.

The headline finding was the surprise. The earlier ASR bench measured parakeet-mlx at RTF 0.055 (18x real-time). This run measures the same engine inside the production flow at RTF 1.0. Naïvely, “going to production cost us 18x” sounds like an orchestration problem, but the per-step breakdown shows the orchestrator itself is 3% of wall. The other 97% is the work the production pipeline does around the ASR call: CFR-normalize the video, run auto-editor for silence trim, assemble a draft via h264_videotoolbox, burn ASS captions via libass. Each ffmpeg-bound step costs 1 to 3 seconds of fixed startup on short clips, and there are five of them.

What didn’t. Four build issues, all minor, none blocking:

  1. wmill CLI package naming: the npm package is windmill-cli, not wmill. Took two minutes and a npm search to figure out. Installed via bun install -g windmill-cli.
  2. AAC frame rounding in MP4 wraps: the first pass of the corpus builder used -shortest and produced MP4s up to 0.82 s longer than the source WAVs because AAC encodes in 1024-sample frames and the last frame extended past the audio. I fixed it by probing the WAV duration first and passing an explicit -t bound equal to that duration. Drift dropped to under 30 ms.
  3. Unscoped wmill sync push would have mutated production: the dry-run reported two changes: the new benchmark flow, which was expected, and an unexpected modification to the existing production flow that would have stripped a remote-only permission flag. Pushing unscoped would have silently modified production. Fixed by scoping the push to the benchmark flow only. wmill’s own tool prompt explicitly warns AI agents not to push without confirmation; this is exactly the situation that warning exists for.
  4. wmill --json mixes ANSI color codes with JSON output: the collector crashed on JSONDecodeError: Extra data because my naïve parser searched for the first open bracket to locate the JSON start, but it matched an ANSI color code in the workspace banner instead of the actual JSON array. Fixed by stripping ANSI codes line-by-line and returning the first line that starts with { or [.

What I learned

  • The 18x tax from engine-only to production isn’t about Windmill. It’s about the production pipeline’s pre/post-processing being five ffmpeg-bound steps that each pay a ~1 s startup cost. SSH overhead is 0.34 s median (3%). If I wanted to drop the wall time, the lever is not “remove Windmill”; it’s “fewer ffmpeg invocations” or “long-lived M1 worker that holds Parakeet in memory.”
  • RTF around 1.0 on short clips is the floor, not the steady state. Each Windmill step spawns a fresh Python process on the M1 via SSH, so parakeet-mlx and auto-editor both pay their cold-load cost on every run. Longer clips would amortize this; at audio durations of 5+ minutes the floor becomes negligible. The benchmark deliberately exposed the floor by using a corpus of 3 to 35 s clips.
  • Deploy-diff audit is non-optional. Issue #3 would have silently changed a field on a production flow if I’d run wmill sync push without scoping it. The lesson: scope every sync push, even for greenfield adds; never trust the global diff to “only do what I want.” wmill’s tool prompt encodes this exact rule.
  • --json flags are not enough to make CLI output parseable. Both issues #2 (ffmpeg frame rounding) and #4 (wmill ANSI in JSON) are the same shape: a tool optimized for human readers leaks human-formatting artifacts into machine-consumed output. Scripted pipelines need explicit bounds (-t WAV_DUR) or filters (strip ANSI, find JSON sentinel) on top.
  • Reusing the earlier corpus was the right call. I considered hand-transcribing real video-pipeline inbox clips for this run, but the benchmark question changed from “what’s the WER” (the ASR bench) to “where does wall time go” (this run). Wall time doesn’t care about transcript accuracy, and reusing the existing 16 clips saved 2+ hours of human labor. The wrap recipe (still frame + WAV) made them flow-compatible without changing the engine.

Related