Frontier vs local LLMs on structured HTML

Jun 2026

I run a local-inference stack on a 32GB M-series MacBook Pro. The whole point of building it was to figure out which jobs I can keep on the laptop and which ones I have to send out to a frontier API. So I built a benchmark.

The headline finding: kimi-k2.5 scored a perfect 22 out of 22 on strict grading, with a median wall-clock time of 36.0 seconds, beating every other model on both axes. The best local model, gemma3-12b-tuned, scored 14 out of 22. That ties the bottom of the working frontier tier, models like mimo-v2-pro and mimo-v2.5-pro. Every other functional frontier model beat the best local.

That is not the headline I expected going in. I expected the local-vs-frontier gap to be smaller, and I expected reasoning models (deepseek-v4-pro, the slower variants) to dominate the structured-generation task. Neither happened. The non-reasoning frontier models with sharp instruction-following ran the table.

This post walks through how the benchmark works, what the numbers say, and the one cross-class observation that surprised me most: the practical gap between local and frontier on this task is not about raw accuracy, it is about consistency in the tail. The accuracy gap is real but smaller than common framing suggests. The reliability gap is wider than I expected.

If you run local inference and you want a triage rule for what to keep local versus what to send to a frontier endpoint, the last section gives you one.

Why HyperFrames HTML for the task

Most local-vs-frontier comparisons run on broad reasoning benchmarks. Those are useful for the labs and not very useful for me. The jobs I actually want LLMs to do are structured generation jobs: emit a tool call, emit a config file, emit code that compiles, emit a structured document that a downstream tool can ingest. Reasoning benchmarks do not tell me how a model behaves on those workloads.

So I picked a task that more closely resembles production LLM use. HyperFrames is an HTML-based video composition framework. Each composition is a self-contained HTML file with a specific structure: scene definitions, timing metadata, asset references, and a strict lint contract. To pass the benchmark, a model has to read a structured prompt, generate a complete HyperFrames composition, and produce output that lints clean and renders to video.

Three properties make this task useful as a benchmark:

Deterministic grading. Every output runs through hyperframes lint and hyperframes render. Either it parses and renders or it does not. There is no LLM judge in the grading path, no rubric ambiguity. The verifier returns a hard pass or fail per prompt.
Diverse failure modes. Outputs do not fail in just one way. They can be unparseable, parse but fail lint, lint but fail render, or render but drift from prompt intent. The grading buckets capture this: strict (renders cleanly and content matches), partial (renders but content drifts), unparseable (does not get through lint/render at all).
Realistic complexity gradient. I authored 22 prompts split 6 easy / 12 medium / 4 hard. Easy prompts are single-scene compositions. Medium prompts are multi-scene with timing dependencies. Hard prompts force the model to reason about ordering, asset reuse, or non-obvious structural choices.

The benchmark has three pieces: a single-script runner, a deterministic verifier that grades each output via hyperframes lint and hyperframes render, and a shell-based cohort orchestrator that handles the per-model setup. The whole sweep reruns with one command. The architectural pattern (one gateway, three grade buckets, cohort orchestration, timeout-recycle) is the part worth copying if you build something similar.

How the benchmark actually runs

The pipeline:

prompt ──> LiteLLM gateway @ localhost:4000 ──> model (local or frontier)
            │
            ▼
       response text
            │
            ▼
   materialize as HyperFrames project
            │
            ├──> hyperframes lint   ──┐
            ├──> hyperframes render ──┼──> verifier.py ──> strict | partial | unparseable
            └──> verifier.py        ──┘

The runner is a single Python script that walks every prompt, hits the gateway, materializes the response into a project directory, invokes the linter and renderer, and writes one CSV row per call for results and a separate cost row per call. Each model runs the same 22 prompts in the same order.

Three design choices matter.

One gateway, every model. Local llama-server processes, MLX models, and frontier APIs all sit behind a single LiteLLM proxy at localhost:4000. The runner does not know or care whether a request will be served by a 12B model on the laptop’s GPU or a frontier endpoint sitting behind an OAuth wall. This is a real ergonomic win. It also gives you consistent telemetry: cost, latency, retry, circuit-breaker behavior all flow through the same surface.

Three grade buckets, not two. Most benchmarks report a single number per model. I report three. The split matters because different failure modes carry different signal. A model that produces unparseable output is failing to follow structure. A model that produces partial output understands structure but misreads intent. Both are “wrong” but they tell you different things about where to spend your tuning effort.

Cohort orchestration handles the messy resource constraints. The cohort orchestrator knows that the laptop can only hold one large llama-server resident at a time, so it stops qwen3-14b before starting granite-30b and restores it after. It handles the MLX stop/start handshake the same way. None of this is interesting research, but it is the kind of thing that distinguishes a benchmark you can rerun reliably from one you ran once and got lucky.

The most interesting infrastructure detail is the timeout-recycle path. The per-call timeout defaults to 600 seconds. After a TimeoutError, a restart hook recycles the upstream model server because LiteLLM does not cancel in-flight llama-server slots when the runner gives up.

That detail is worth a paragraph of its own. When a client cancels a slow streaming request, you might expect the gateway to propagate the cancellation upstream. LiteLLM does not. The upstream llama-server slot keeps trying to generate, occupying the GPU. Your next prompt is now competing with a zombie slot, and throughput on a 32GB MBP collapses to something like 4 tokens per second. I learned this the slow way the first time I ran the full local sweep. The fix is the restart hook above: when the runner gives up, recycle the upstream service before continuing. Once that hook is in place, the local cohort runs at expected throughput end-to-end.

There is a harder version of this problem on the frontier side: HTTP 408 cluster failures, where the provider gateway times out before the runner does. No client-side fix recovers those. I marked them as known unrecoverable and moved on.

Frontier results: who actually won

Tier	Model	Strict / 22	Wall median (s)	Notes
top	kimi-k2.5	22	36.0	perfect AND fastest
top	deepseek-v4-flash	22	96.8	perfect
top	deepseek-v4-pro	21	217.7	reasoning model, slower
top	kimi-k2.6	21	38.7	near-perfect, fast
top	hy3-preview	20	42.5
mid	glm-5	18	86.9	post-rerun at 1200s timeout
mid	qwen3.7-plus	18	113.5
mid	glm-5.1	17	125.0	post-rerun at 1200s timeout
mid	qwen3.5-plus	17	99.4
mid	mimo-v2.5	17	45.0	post-rerun
mid	minimax-m2.5	16	48.3
mid	minimax-m2.7	16	48.4
mid	mimo-v2-omni	16	41.0	post-rerun
mid	mimo-v2-pro	14	77.0	rerun unlucky; HTTP 408 dominant
mid	mimo-v2.5-pro	14	102.0
bottom	qwen3.6-plus	11	340.7	11 unparseable (provider timeout)
bottom	minimax-m3	9	379.8	12 unparseable, all HTTP 408
unavail	qwen3.7-max	0	0.2	provider rejected route, see section 7

Three things to call out.

kimi-k2.5 is the sweet spot. Perfect score, fastest median wall time. Nothing else on the frontier list dominated on both axes. If I had to ship one model behind a production endpoint for this kind of structured generation, that is the one I would ship.

Reasoning models did not dominate. deepseek-v4-pro is a reasoning model. It scored 21 in 217.7 seconds. deepseek-v4-flash, the non-reasoning sibling, scored 22 in 96.8 seconds. Same family, the reasoning variant was both slower and slightly less accurate on this task. On structured generation, more thinking did not help. That is a pattern worth watching across other benchmarks.

Bottom-tier failures are infrastructure, not capability. qwen3.6-plus and minimax-m3 both produced double-digit unparseable counts. Reading the logs, those are not bad outputs. They are HTTP 408 gateway timeouts where the upstream provider failed to return a response inside the runner’s wait window. You cannot benchmark a model that the gateway will not let through. You are always benchmarking the delivery surface, not just the model.

Local results: the laptop side of the gap

Model	Strict / 22	Wall median (s)
gemma3-12b-tuned	14	58.8
qwen3-14b-tuned	13	342.1
granite-4.1-30b-tuned	12	119.4
gemma4-e4b-mlx	11	39.2
granite-4.1-8b-tuned	11	37.8

gemma3-12b is the local winner. 14 strict out of 22 at a 58.8 second median. On a 32GB MBP, that is a credible production-grade model for HyperFrames composition tasks.

Speed versus accuracy on local is brutal. qwen3-14b is the second-best on accuracy (13) but takes 342.1 seconds median. That is roughly six times slower than gemma3-12b for one fewer strict score. The lesson: do not chase the largest local model on the accuracy chart without checking the wall-time column.

The ~30b tier did not pay for itself. granite-4.1-30b-tuned scored 12 strict in 119.4 seconds, beaten by gemma3-12b on both axes. Memory headroom on a 32GB MBP is precious. Spending it on a 30b model that underperforms a 12b model is a bad trade.

The smaller end (gemma4-e4b-mlx, granite-4.1-8b at 11 strict) is valuable as “always on” models you can keep resident without GPU pressure, but they leave a non-trivial gap versus gemma3-12b.

The cross-class insight: where the gap actually is

Here is the comparison that surprised me most. Best local scored 14 out of 22 strict. The bottom of the functional frontier tier (mimo-v2-pro, mimo-v2.5-pro) also scored 14. The gap most people imagine between local and frontier is that local does hobby work and frontier does real work. The numbers do not fully support that framing. Best local genuinely matches the bottom of the working frontier.

The accuracy gap is real but smaller than the common narrative suggests.

The gap that is large is the reliability gap. kimi-k2.5: 22 of 22 at 36.0 seconds. deepseek-v4-flash: 22 of 22 at 96.8 seconds. Both perfect. The frontier top tier did not just score higher in expectation; it scored higher with much tighter variance. The hard prompts did not trip them. The medium prompts did not surprise them.

Local models did not have that property. gemma3-12b at 14 strict means 8 prompts went wrong. Some were partial, some unparseable. The model is not catastrophically failing; it is failing in the tail. And the tail is where production LLM use actually hurts. You do not lose customers because the model handled the easy prompt well. You lose them because it occasionally generated unparseable nonsense on a harder one and you had to recover.

The real frontier advantage on this task is not “smarter.” It is “more consistent in the tail.”

That suggests a triage rule for local-inference users:

Routine structured generation: local is fine. The 14 out of 22 from gemma3-12b will cover easy and medium prompts well. If your application is forgiving (drafts, prototypes, internal tools where a human reviews output), this is your default.
Reliability-critical structured generation: still goes to frontier. If the application fails badly on the unparseable path (auto-applied changes, downstream pipelines, customer-facing output), buy the consistency. kimi-k2.5 at 22 out of 22 is what that money is buying.

The interesting design question, once you frame it that way, is not “local versus frontier.” It is “which prompts route to which surface.” If you can classify prompts as easy/medium/hard before issuing them, you can keep easy and medium ones local and send only hard ones to frontier. That is a routing problem, not a model problem.

Surprises and footnotes

A few results did not fit the headline, and they are worth separating from the main story because they are mostly about infrastructure, not model capability.

qwen3.7-max scored 0 out of 22. The model did not fail the task. The OpenCode Go OAI-compat endpoint rejected the route with "Model qwen3.7-max is not supported for format oa-compat". LiteLLM converted that to HTTP 401 and circuit-broke the model. Wall median was 0.2 seconds because every call failed immediately. The lesson is simple: when a benchmark shows 0/n, suspect infrastructure before you assume the model is useless. A model you cannot route to is not a model you have benchmarked.

The HTTP 408 cluster on minimax-m3, mimo-v2-pro, and mimo-v2.5-pro was also provider-side. minimax-m3 returned 12 unparseable results, all HTTP 408. mimo-v2-pro and mimo-v2.5-pro showed the same pattern at lower severity. These are upstream gateway timeouts: the provider’s load balancer or model server gave up before the runner’s 600-second timeout fired. Bumping BENCH_LITELLM_TIMEOUT_S would not help because the failure happens upstream. The only recovery is rerun or route around the provider. I reran where I could, but some runs stayed unlucky.

Cost across the entire frontier sweep was reported as $0. The calls ran through an OpenCode Go subscription, not per-call billing. That does not make frontier inference free in general, but it does change the framing of this particular comparison. The cost axis here is latency and reliability, not line-item dollars per thousand calls.

These footnotes matter because they keep the benchmark honest. A clean table hides dirty wires. The wires are part of the result.

What’s next

This benchmark is one snapshot of one task class. The obvious next question is whether the gap looks the same on other workloads.

bench-v4-asr is already running. It compares parakeet-mlx tdt-1.1b against whisper-large-v3 on local speech-to-text. The early leader is parakeet-mlx at 0.55% WER with an RTF of 0.055. That is a case where a small local model is competitive on both accuracy and speed, which is the opposite pattern from what we saw here.

The nvitro Gemma 4 journal is the concrete application side of the same question. nvitro is a structured-generation workload that emits video composition metadata, and Gemma 4 (specifically the 4b MLX variant from the local table) is the “good enough” local candidate I want to test in production. If it handles the routine cases and only escalates the hard ones, the routing rule from section 6 pays off.

The bigger question is agentic and tool-using work. This benchmark is one-shot structured generation: one prompt, one response, one verification. Real systems rarely look like that. They look like loops: plan, call a tool, observe, call another tool, emit a structured artifact. The bench-v5 hook is to measure whether the local-vs-frontier gap changes when the task is not one shot but a multi-step trace where compounding errors matter. My guess is the reliability gap gets larger, not smaller, once errors compound. That is a guess until the numbers say otherwise.

Methodology appendix

The key components are a per-prompt loop that calls the gateway and records telemetry, a shell orchestrator that handles model-specific stop-start handshakes, and a deterministic grader.

The grader extracts an index.html from the model response, materializes it into a scaffolded HyperFrames project, runs hyperframes lint and hyperframes render, and checks the rendered duration and required slots against the prompt spec. The three buckets are strict for a clean render with matching content, partial for a rendered output that drifts from intent, and unparseable for anything that fails lint or render.

Every model saw the same 22 prompts in the same order. Tiers were 6 easy, 12 medium, 4 hard. Median wall-clock times are computed across successful calls only; unparseable and timeout rows are excluded from the latency column but counted in the strict score.

As noted in section 7, the frontier sweep was routed through an OpenCode Go subscription, so the reported cost was $0. That is not a claim that frontier APIs are free; it is a note that the economic axis in this run was subscription latency and reliability, not per-token spend.