I build a local-first meeting transcription app for macOS. Everything runs on-device: audio capture through Core Audio, transcription through whisper.cpp with Metal acceleration, no cloud round-trips. The pitch is privacy, but privacy only sells if the local experience doesn’t feel like a downgrade. Live transcription has to appear while you’re still in the meeting, and a 3 GB model download on first launch is a hard sell.
So before tuning anything, I benchmarked it. The defaults were wrong for my product on every axis I measured. This post is the writeup: what I measured, what surprised me, and what shipped. If you’re putting Whisper in a real-time product, the numbers should transfer.
The setup
All experiments ran on a MacBook Pro (Apple Silicon), whisper.cpp v1.8.4, 12 threads. The model under test is large-v3-turbo, because that’s the quality bar I want to ship, not the model that makes benchmarks look good.
The test corpus: 100 real voice recordings, 0.5s to 125s long, a mix of English and Portuguese. Real dictation, real background noise, real silence, because Whisper’s failure modes (hallucinating “Thank you.” into silence, mangling proper nouns) don’t show up on clean sample audio.
Four questions:
- Is CoreML (Neural Engine) worth it over the default Metal backend?
- What does q5_0 quantization cost in quality, and buy in speed and size?
- Does VAD (voice activity detection) actually help, or just skip audio?
- How small can streaming chunks get before quality falls apart?
CoreML: not worth it
whisper.cpp can offload the encoder to the Neural Engine via CoreML instead of running everything on the GPU through Metal. Sounds like free performance. It isn’t.
On the 11s JFK sample:
- CoreML, first run: 31.2s (compiling and caching the model for the Neural Engine)
- CoreML, warm run: 1.57s
- Metal (ggml), warm run: 1.58s
A 0.01s difference on warm runs, paid for with a 31-second cold start the first time a user transcribes anything. That’s a support ticket generator, not an optimization. I kept the conversion steps documented in case a future whisper.cpp release changes the math, and shipped Metal.
Lesson: “uses the Neural Engine” is a spec-sheet feature, not a user-facing one. Measure the warm path.
Quantization: q5_0 is free money
Quantizing large-v3-turbo from f16 to q5_0 is one command and gets you:
| Metric | f16 | q5_0 |
|---|---|---|
| Model size | 3086 MB | 547 MB |
| Total time, 100 files | 232.6s | 196.6s |
That’s a 6x smaller download and ~15% faster inference. The catch is supposed to be quality, so I diffed all 100 transcripts. The differences were trivial: capitalization, punctuation, “webview” vs “web view”, “dragable” vs “draggable”. Meaning was preserved in every file.
For this model at this quantization level, there is no quality argument for shipping f16. A 547 MB download instead of 3 GB changes the onboarding funnel; 15% faster changes battery and latency. Take the free money.
VAD: the biggest win, and not for the reason you’d guess
I tested Silero VAD v6.2.1 in front of the model. Speed results across the 100 files:
| Config | Total time | vs f16 baseline |
|---|---|---|
| f16 | 232.6s | baseline |
| f16 + VAD | 194.2s | -16.5% |
| q5_0 | 196.6s | -15.5% |
| q5_0 + VAD | 167.8s | -27.9% |
Stacked with quantization, VAD gets total transcription time down almost 28%. But speed is the secondary benefit.
The primary one: VAD kills hallucinations on silence. Without it, Whisper confidently transcribes silent or noisy clips as “Thank you.”, ”*Dramatic music*”, ”*thud*”. In a meeting app, those show up as ghost lines in the transcript and users assume the whole thing is broken. With VAD, those clips correctly come back empty.
It’s not a pure win. 73 of 100 files had non-trivial text differences with VAD enabled, and while most were improvements (better sentence structure on long recordings, “iTerm” recognized instead of “item”, “Lua” instead of “Luan”), there were real regressions: one case where “should use it” became “should not use it”, one short clip that came back as Korean, and minor word swaps. Inverted meaning is the scariest failure mode in transcription, and it happened once in 100 files.
Net verdict: VAD helps far more than it hurts, and the silence-hallucination fix alone justifies it. But if you’re building on Whisper, know that VAD changes the output, it doesn’t just gate it.
Chunk size: there’s a floor, and it’s ~1.6s
Real-time display means chunking the audio stream and transcribing chunks as they complete. I tested fixed-size chunks from 1s to 10s (q5_0, 12 threads) against full-file transcription as ground truth.
First surprise: per-chunk latency is ~1.6s regardless of chunk size. Model warm-up dominates, so a 1s chunk and a 10s chunk both cost ~1.6s to process. That single number defines the whole design space:
| Chunk size | Latency | Real-time viable? |
|---|---|---|
| 1s | ~1.6s | No, processing is slower than the audio |
| 2s | ~1.6s | Still slower than real-time |
| 3s | ~1.6s | Barely (1.6s to process 3s) |
| 5s | ~1.6s | Comfortable |
| 10s | ~1.6s | Safe margin, best quality |
Second surprise: quality collapses below 5s. Whisper needs context. At 1s chunks the output is pure garbage, hallucinations and random languages. At 2s it’s still noisy. 3s has missed words and wrong proper nouns. At 5s, quality is good with minor boundary errors. At 10s it’s near full-file quality.
From a 34s recording, the same passage at different chunk sizes:
FULL: Sometimes cloud sessions use hs timer do after hs reload but i believe
that on the main cloud code in the user folder we have an explicit
instruction to use hs reload hs - reload. Double check that.
3s: Sometimes cloud sessions use H.S. Timers. Do laughter. Hey, Jack. yes
reload But I believe that on the main platform. Hardcoring the user folder
1s: - Sometimes. Sometimes cold sessions. Ew. is H.S. Timer. - It's really fun.
The laughter...“Do laughter. Hey, Jack.” is not what anyone said in a meeting.
The floor also sets your honest latency budget: live text will always lag by chunk_duration + 1.6s. With 5s chunks that’s ~6.6s behind live speech; with 10s chunks, ~11.6s. There’s no clever engineering around it with this model, only the tradeoff of freshness vs quality.
Overlap: tested, rejected
The obvious fix for boundary errors is overlapping chunks: prepend a few seconds of the previous chunk so Whisper has context across the seam (what whisper-stream’s --keep does). I tested step=5s with 1-3s overlap, step=10s with 2-5s, step=3s with 1-2s.
Result: overlap doesn’t improve quality. It duplicates text at boundaries, which then needs deduplication logic to clean up, and the extra context doesn’t make the transcription better. It just repeats itself. Complexity with no payoff, cut.
What shipped
The production config, with the reasoning:
- Model:
ggml-large-v3-turbo-q5_0.bin. 547 MB instead of 3086 MB, ~15% faster, same effective quality. - VAD on, always. Stacked with q5_0 that’s ~28% faster than the f16 baseline, and it eliminates hallucinated text on silence.
- Real-time: 5s chunks minimum, 10s when the latency tolerance allows. Never below 5s.
- No chunk overlap. Not worth the dedup complexity.
- Post-recording transcription always runs full-file. Chunking exists only to serve the live display; once the recording is done, give Whisper all the context.
- VAD in the streaming path skips silent chunks entirely, saving the 1.6s warm-up cost per silent chunk (meetings have a lot of silence).
One architectural note the benchmarks don’t show: in my pipeline, audio capture publishes frames synchronously to all subscribers, so the transcription manager must never block on the FFI call. Frames go through an unbounded channel to a background task that does the buffering and the blocking Whisper calls. If your capture path and your inference path share a thread, no amount of chunk tuning saves you.
The transferable summary
If you’re shipping whisper.cpp large-v3-turbo on Apple Silicon:
- Skip CoreML, use Metal. Warm-run parity, and CoreML costs a 31s first-run compile.
- Quantize to q5_0. 6x smaller, ~15% faster, no meaningful quality loss.
- Run Silero VAD in front. ~28% total speedup combined with q5_0, and it fixes silence hallucinations, at the cost of occasional output changes worth knowing about.
- Respect the ~1.6s per-inference floor. 5s chunks minimum for streaming, 10s preferred, full-file for anything offline.
- Don’t build overlap logic. I did the experiment so you don’t have to.
Every number above came out of a benchmark run, not a changelog. That’s the part I’d push on hardest: one afternoon of measurement replaced the defaults (f16, no VAD, whatever chunk size the example uses) with a config that’s 28% faster, 6x smaller, and hallucinates less.
This is also the kind of work I take on contract: on-device AI, native macOS and iOS, pipelines that need numbers before opinions. If yours hasn’t had this pass yet, you know where to find me.