Running Whisper + Kokoro on a laptop for a real-time sales coach

A voice role-play app lives or dies on the gap between you finishing a sentence and the AI prospect answering. Round-trip a cloud STT service, then an LLM, then a cloud TTS service, and you're at three network hops of latency plus three vendors watching a founder practice their pitch. For ToCoach I wanted two of those three hops gone: speech recognition and speech synthesis run on the machine, and only the LLM call leaves it.

The models are smaller than people assume. Whisper base.en is ~150MB; Kokoro-82M is ~80MB. Both download automatically on first run and load through ONNX Runtime — Whisper via Transformers.js, Kokoro directly. The whole speech stack fits in less disk than a single node_modules folder, runs offline on a plane, and your fumbled discovery calls never leave the laptop.

The pipeline

MediaRecorder (mic)
  → Whisper base.en (ONNX, on-device)   ~transcript
  → Claude (the prospect persona)        ~reply text
  → Kokoro-82M (ONNX, on-device)         ~speech
  → Web Audio playback

Two implementation details do most of the latency work. First, sentence-level TTS: don't wait for the full LLM reply — synthesize and start playing the first sentence while the rest is still generating. That's what gets speech onset under a second; synthesizing the whole reply first would double or triple perceived latency for long answers. Second, tiered Whisper: tiny.en is available for weaker machines, base.en where the accuracy is worth the extra inference time.

The day I lost to a header

Transformers.js wants multithreaded ONNX inference, and multithreading in a browser context means SharedArrayBuffer. What nobody tells you: since Spectre, SharedArrayBuffer only exists on pages that are cross-origin isolated — served with Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp. My symptom was nothing so helpful as an error saying that. Inference just ran single-threaded and slow, and one library path threw a SharedArrayBuffer is not defined deep inside a worker where the stack trace pointed everywhere except the actual cause.

A full day of profiling model quantization levels later, the fix was two lines — in Electron, headers on the session:

session.defaultSession.webRequest.onHeadersReceived((d, cb) => cb({
  responseHeaders: { ...d.responseHeaders,
    "Cross-Origin-Opener-Policy": ["same-origin"],
    "Cross-Origin-Embedder-Policy": ["require-corp"] }
}));

If your in-browser inference is mysteriously slow, check crossOriginIsolated in the console before you touch anything else. It's a boolean. It would have saved me a day.

What I'd do differently

I'd budget for the audio plumbing, not the models. The models were the easy part — they're good now, and they're free. The time went into VAD thresholds, barge-in behavior, MediaRecorder chunk sizes, and the never-brick rule: every AI surface has a deterministic fallback, so a hung model call degrades to an error string, never a frozen session.