Commit graph

2 commits

Author SHA1 Message Date
Brooklyn Nicholson
9859e1f7df fix(voice): speak the whole turn, not just its first bubble; idle-flush held narration
Two fixes for desktop hands-free voice:

- The live speech session bound to the first assistant bubble with text, so
  a tool-calling turn spoke only the opening narration and silently dropped
  every later interim AND the final answer. The conversation selector now
  aggregates all unspoken assistant bubbles in order (turn-scoped speech);
  auto-speak keeps its latest-reply-only behavior.

- The speak-stream WS producer blocked forever on the text queue, so a
  narration line with no trailing whitespace ("Let me check.") sat in the
  sentence chunker until end-of-turn — spoken long after the tool finished,
  with the UI stuck on "Preparing audio…". Mirror the CLI speaker's idle
  flush: sentence-terminated buffers flush after 0.5s of producer silence,
  anything else after ~2s; open <think> blocks are never flushed.
2026-07-23 01:53:37 -05:00
Brooklyn Nicholson
93e9061f15 feat(voice): desktop speech-stream sessions with barge-in capture
/api/audio/speak-stream WebSocket: one socket + one Web Audio clock per
reply. The renderer feeds raw LLM deltas as they arrive; the server
cuts sentences with the shared chunker and streams int16 PCM back while
generation continues — speech overlaps generation with no per-sentence
connection or synthesis gaps. Falls back to the POST data-URL path for
old backends / non-chunked providers.

Barge-in runs a MediaRecorder on the monitor's stream the whole time
playback is live (rotated while quiet to bound pre-roll); talking over
the agent cuts playback and the complete utterance goes straight to
transcription and submit. /api/audio/transcribe returns 200/"" for
no-speech results so quiet turns re-listen instead of toasting a 400.
2026-07-22 17:47:25 -05:00