Voice latency, honestly measured

Part 4 of Jarvis: the architecture series by Jack Stovell · published 2026-08-17

Voice latency, honestly measured — architecture diagram

Voice latency, and where it actually lives.

Everyone demos a voice agent. Almost nobody publishes the latency distribution from real use, because it's not always flattering.

So, 847 real conversational turns, a month of talking to Jarvis. The model turn (LLM plus tool calls, start to finish) came out at p50 3.0s, p90 6.3s, p99 13.5s. Not a benchmark. Measured in the turn loop itself.

Here's the bit I didn't expect. Before the model even starts, there's an endpointing stage, a classifier deciding whether I'm done talking or mid-thought. It was tuned to hold when unsure. Sensible, in theory, you don't want to cut someone off mid-sentence.

Except ground-truth logging told a different story. 97.8% of those holds were needless, I had finished talking. The thing it was protecting against, cutting me off, happened 0.4% of the time.

The bias was solving a problem that basically didn't exist. And every needless hold is pure latency, a 745ms politeness tax on every turn.

Flipped the bias to answer promptly. Tax dropped to 496ms, cut-offs unchanged, checked across 4,277 endpointing decisions. Still 93% of the remaining holds are needless, so there's more to squeeze.

Also: 344 barge-ins in that window. I talk over him constantly — a barge-in halts the audio mid-sentence, and mid-thinking too, killing the in-flight model turn. Demos rarely show that.

Full anatomy and diagram: github.com/StovBuilds/jarvis-architecture

Measure the felt thing end to end. I was watching the model. The tax was hiding somewhere else entirely.

Measurement provenance (all pulled live 2026-08-12)

claude-voice-convo unit, window 2026-07-14 → 08-11 (journal retention start),

n=847. p50 2,996ms / p90 6,294ms / p99 13,482ms. First token p50 1,283ms.

Recent slice (since 08-01, n=237) is faster: p50 2.5s / p90 5.6s — not used in

the post to keep one honest window, but it's headed the right way.

2026-06-09 → 08-11 split at the flip commit 35c65e9 (2026-07-22): pre-flip

needless holds 1,358/1,388 (97.8%), cut-offs 4/910 (0.4%), mean hold across all

turns 745ms; post-flip 292/313 (93.3%), 1/288 (0.3%), 493ms. Median hold

1,221ms → 764ms. The flip + floor cuts shipped same-day from the 07-22 timing

analysis (docs/timing-analysis-2026-07-22.md); knobs confirmed live in .env

(hold 700, ceiling 2500, coalesce 1100, grace 400).

2026-08-12). The public README's old "85" was stale and got corrected in the same

commit — a session registers the subset its transport wires, so the catalog count

is the honest one.

Architecture write-up: https://github.com/StovBuilds/jarvis-architecture.

← Part 2: The bus: how 22 agents coordinate · All parts