Latency & interruption
What runs on the response path, and keeping barge-in responsive.
For a voice agent, two things dominate the felt experience: how fast the first audio comes back, and how fast the agent stops when the user talks. The runtime is built around protecting both. This page is what to keep in mind operationally.
What's on the response path
Only generation blocks the turn. Everything else runs around it:
- On the path: LLM streaming → sentence-wise TTS → forwarding audio to the transport. Text flushes to TTS as sentences complete, so audio starts before the full reply is written.
- Off the path: metrics, usage, telemetry spans, committed-event recording. These happen around the reply, never inside the generation loop.
The practical consequence: instrumentation and bookkeeping don't cost you latency. Model round-trips do — so first-audio time is dominated by your LLM and TTS providers.
Speculative generation
TurnOptions.PreemptiveGeneration (on by default) starts a reply from a final
STT transcript before the turn formally commits, and promotes it only if the
committed transcript and request snapshot are unchanged. PreemptiveTTS
(off by default) extends this to synthesis — audio is buffered and never played
until promotion. Together they hide model latency behind the endpointing delay.
Keeping barge-in responsive
Barge-in responsiveness is a threshold trade-off:
| Knob | Lower = | Higher = |
|---|---|---|
InterruptionMinDuration (500ms) | Faster to stop, more false stops | Slower to stop, fewer false stops |
InterruptionMinWords (0/off) | (STT only) stricter word gate | — |
MinEndpointingDelay (500ms) | Snappier turns, more early commits | More patient, higher latency |
Tune interruption thresholds first; false-interruption resume is the safety
net for the residual (a cough or backchannel pauses the reply and resumes it after
FalseInterruptionTimeout rather than killing it).
A tool that calls RunContext.WaitForPlayout holds the turn until its audio
finishes. It's the right tool for "say it, then act," but every use adds latency
to that turn — reserve it for genuine ordering needs.
Where the ML lives
VAD and silence detection run in Go, in-process. Heavier end-of-turn / interruption models run in an external sidecar over HTTP, so the agent process stays free of ONNX/CGo — but a slow or unreachable sidecar becomes a latency source of its own. Size and colocate it accordingly.
Related
- Turn detection & interruption
- Reference: TurnOptions · Telemetry