docs(voice): streaming TTS + barge-in across surfaces

This commit is contained in:
Brooklyn Nicholson 2026-07-22 17:47:25 -05:00
parent 93e9061f15
commit 83456d0bc6

View file

@ -159,11 +159,20 @@ Both `silence_threshold` and `silence_duration` are configurable in `config.yaml
### Streaming TTS
When TTS is enabled, the agent speaks its reply **sentence-by-sentence** as it generates text — you don't wait for the full response:
When TTS is enabled, the agent speaks its reply **sentence-by-sentence** as it generates text — you don't wait for the full response. This works with **every TTS provider**:
1. Buffers text deltas into complete sentences (min 20 chars)
2. Strips markdown formatting and `<think>` blocks
3. Generates and plays audio per sentence in real-time
2. Strips markdown formatting, emoji, and `<think>` blocks
3. Plays audio per sentence in real-time — providers with a chunked PCM API (ElevenLabs, OpenAI) stream raw audio for the lowest time-to-first-word; every other provider (including the default Edge) synthesizes and plays each sentence as it completes
The same pipeline runs in the classic CLI, the TUI, and the desktop app. In a desktop voice conversation the reply text is fed **live** into a per-reply speech WebSocket as the model generates it, so speech overlaps generation — one socket and one audio clock per reply, no per-sentence connection gaps.
### Barge-in
You can interrupt the agent mid-speech:
- **Talk over it** — in continuous voice mode, a voice-activity monitor listens while the agent speaks and cuts playback the moment you start talking, then goes straight back to recording. The detector calibrates its noise floor against the playback itself, so speaker bleed doesn't self-trigger. Disable with `voice.barge_in: false` in `config.yaml`.
- **Type or press the record key** — sending a new message or hitting the push-to-talk key stops playback instantly on every surface.
### Hallucination Filter