mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-29 18:46:59 +00:00
3a4aa2f8e6
Add an opt-in streaming-audio adapter seam to BasePlatformAdapter so voice-capable gateway platforms (LiveKit, Discord voice, future adapters) can consume LLM output as streaming PCM audio before the full response completes, dropping perceived voice latency from ~2-3.5s to ~500-800ms. Adapter contract (gateway/platforms/base.py): - AudioFormat dataclass: declared sample_rate, channels, sample_width - StreamingTTSHandle: opaque handle with audible/aborted flags - supports_streaming_tts / begin_streaming_tts / write_streaming_tts / finish_streaming_tts / abort_streaming_tts - All default to unsupported/no-op so existing adapters are source-compatible - Per-turn _streaming_tts_completed_chats set suppresses duplicate whole-file auto-TTS when streaming succeeded; cleared after turn completion Gateway consumer (gateway/streaming_tts_consumer.py): - StreamingTTSConsumer: bridges sync agent deltas to async adapter audio sink - Uses existing SentenceChunker (no competing parser) - Thread-safe bounded queue; on_delta never blocks the agent worker thread - Resolves configured streaming provider via resolve_streaming_provider() - Serialises clause playback in order; flushes tail on completion - Pre-audio failure: completed=False (falls back to whole-file TTS) - Post-audio failure: completed=True, partial=True (no replay from start) - Abort is idempotent; late chunks silently dropped - Per-turn state isolated across concurrent chats Gateway integration (gateway/run.py): - message_type parameter threaded through _run_agent -> _run_agent_inner - StreamingTTSConsumer created when voice input + auto-TTS + provider active - Delta callback teed to both text stream consumer and TTS consumer - TTS-only delta callback installed when text streaming is off - finish() called from executor; wait_complete() in async context after - Barge-in aborts the consumer at all three interrupt detection points - Runner-level _send_voice_reply suppressed when streaming TTS completed Tests (tests/gateway/test_streaming_tts_consumer.py): - 15 focused tests: adapter defaults, lifecycle, ordered chunks, unsupported/No-streamer fallback, abort idempotency, late-chunk drop, pre/post-audio failure, concurrent-turn isolation, think-block suppression, queue backpressure Does not touch desktop/TUI code or add config flags. Plugin TTS provider stream() metadata gap (#47896) is explicitly out of scope — built-in ElevenLabs/OpenAI PCM streamers are the first consumers. Refs: #60671, #47896 |
||
|---|---|---|
| .. | ||
| qqbot | ||
| __init__.py | ||
| _http_client_limits.py | ||
| ADDING_A_PLATFORM.md | ||
| api_server.py | ||
| base.py | ||
| bluebubbles.py | ||
| helpers.py | ||
| msgraph_webhook.py | ||
| signal.py | ||
| signal_format.py | ||
| signal_rate_limit.py | ||
| webhook.py | ||
| webhook_filters.py | ||
| weixin.py | ||
| whatsapp_cloud.py | ||
| whatsapp_common.py | ||
| yuanbao.py | ||
| yuanbao_media.py | ||
| yuanbao_proto.py | ||
| yuanbao_sticker.py | ||