Speech & the turn
How a turn flows, and steering a single reply.
A turn is one user→agent exchange, in two halves: recognition (turn the user's audio into a committed user turn — see Turn detection) and generation (produce and speak the reply). Only generation rides the latency-critical path. For a voice agent latency is the product, so the runtime is built around two rules: get to first audio fast, and always be ready to stop.
The flow
Generation: produce and speak the reply
One reply is one SpeechHandle. The generation loop:
- Builds the chat context (current agent's instructions + shared history).
- Streams the LLM. Text flushes to TTS as sentences complete — audio starts before the full reply is written.
- Runs any tool calls the model emits, then does a follow-up round folding the tool output back into the reply.
- Forwards TTS audio to the transport and waits for playout.
Speech is scheduled one handle at a time, so replies never overlap. Say (fixed
text, no LLM) and GenerateReply both produce handles on the same queue;
session.Run wraps generate-and-wait for a single turn.
On the path vs off it
Only generation blocks the turn. Metrics, usage, telemetry spans, and committed-event recording happen around the reply, not inside the latency-critical loop — so the agent never pauses on bookkeeping.
A tool that calls RunContext.WaitForPlayout waits for this step's audio to
finish before continuing — useful for "say it, then act," but it holds up the
turn. Reserve it for when ordering against speech matters.
Steer a single reply
GenerateReply takes per-reply options that scope to one reply without
changing the agent.
Extra instructions layer on top of the agent's base instructions for this reply only:
h, _ := session.GenerateReply(agents.GenerateReplyOptions{
Instructions: "Keep this answer under 20 words.",
UserInput: "Summarise our refund policy.",
})
_ = h.WaitForPlayout(ctx)Other per-reply knobs: Tools (restrict to a named subset — see
Tools), ToolChoice, AllowInterruptions,
Priority, and InputModality (text vs audio instruction variant).
Next
- Chat context & history — what the loop builds context from.
- Full API: Reference → GenerateReplyOptions.