Concurrency & the turn loop

A common worry with a voice agent is: while it's fetching data for one request, is it stuck — deaf and silent until the fetch returns? For an AgentSession, the answer is no. A session is not a single blocking loop. It runs several things at once and only serializes the one thing that must be serialized: the agent talking over itself.

This page is the deep companion to Speech & the turn. That page covers the user-facing shape of a turn; this one shows the concurrency underneath it — what overlaps, what waits, and why.

The mental model (read this first)

Think of a session like a person on a phone call, not a program that reads a request and returns a response:

• Their ears are always open. They keep listening even while they're talking or looking something up. → Recognition runs continuously; the agent can always hear you.
• They only have one mouth. They finish one sentence before starting the next; they don't say two things at once. → The scheduler runs one reply at a time.
• When you cut in, they stop. They stop mid-sentence, drop what they were about to say, and even abandon the thing they were looking up. → Barge-in interrupts the reply and cancels the in-flight tool.
• They start thinking before you finish. They begin forming an answer as they hear you, and use it if you stop where they expected. → Preemptive generation.
• They fill dead air. "Hang on, let me check…" while they look something up. → The tool filler.

So the honest one-line answer to "does it block while fetching data?": the answer for that one turn waits for the fetch (that's just cause and effect), but the session does not go deaf or frozen — it keeps listening, can be interrupted, and can fill the silence. The only things that are strictly one-at-a-time are the agent's own replies and the tools inside a single turn.

Everything below is the same story with the actual moving parts drawn out.

The three concurrent activities

A single session runs three cooperating activities. They communicate by passing messages, never by sharing one call stack, so a slow step in one never freezes the others.

• Recognition never stops while a reply is being produced. New user speech is always being processed — which is what makes barge-in and preemptive generation possible.
• Generation runs on its own goroutine, one per reply turn.
• The scheduler hands out exactly one turn at a time, so the agent never speaks two of its own replies at once.

Across a fleet, sessions are fully independent — one per conversation — so a slow tool in one call never affects another.

A normal turn

The straight-line path: recognize the user, generate a reply, speak it while the model is still streaming.

Preemptive generation

The agent does not wait for the turn to be fully committed before it starts thinking. On the final transcript — before the turn detector has confirmed the user is done — it begins computing the reply speculatively, including precomputing speech. If the user keeps talking, that work is thrown away; if the turn commits, the in-flight work is promoted instead of restarted. This is what shaves latency off the first word.

Tools within a turn

This is the heart of the "does it block?" question. When the model asks for tools, they run sequentially inside the generation goroutine — one tool's result is awaited before the next call. While a tool is working, an optional tool filler (a low-priority utterance like "let me check that for you") covers the silence, and is interrupted the moment the tool returns. If the tools produce output, a follow-up model round folds the results into the reply — looping until the model stops asking for tools or the step cap is hit.

The final spoken answer for a turn still waits on the tool result — that is just cause and effect. What the filler and streaming buy you is that the caller is not sitting in dead silence while it happens.

Interruption (barge-in)

Because recognition never stops, the user can talk over the agent at any point — including while a tool fetch is in flight. When that happens, the current reply is interrupted: its context is cancelled (which aborts the in-flight LLM stream and the running tool, assuming the tool honors context cancellation), and any queued agent audio is flushed.

Interruption is gated: a handle created with interruptions disallowed is not interrupted, and a short cough or false trigger is filtered by minimum-duration and word-count thresholds (with a timer to resume speech that was paused on a false alarm). See Turn detection for the thresholds.

One reply at a time

Every utterance — a reply, a say, a filler, a sub-agent's speech — becomes a SpeechHandle on a priority queue. The scheduler authorizes exactly one at a time and waits for it to finish before starting the next. This is deliberate: it stops the agent from talking over itself. It does not block recognition, so user input and barge-in stay live the entire time.

Sub-agent takeover

A tool can hand the turn to a sub-agent with DelegateTask. The parent reply yields its scheduler slot so the sub-agent's speech can run, then the slot is handed back when the sub-agent finishes (last-in, first-out for nested takeovers). This is how a supervisor delegates without deadlocking the single scheduler.

See Tasks & sub-agents and the Supervisor pattern for how to build these.

The contract in one table

The short version: the session stays responsive while it works. The only things serialized are the agent's own reply turns (one at a time) and the tools within a single turn (in order). Everything on the input side keeps running.