mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-29 18:46:59 +00:00
3a4aa2f8e6
Add an opt-in streaming-audio adapter seam to BasePlatformAdapter so voice-capable gateway platforms (LiveKit, Discord voice, future adapters) can consume LLM output as streaming PCM audio before the full response completes, dropping perceived voice latency from ~2-3.5s to ~500-800ms. Adapter contract (gateway/platforms/base.py): - AudioFormat dataclass: declared sample_rate, channels, sample_width - StreamingTTSHandle: opaque handle with audible/aborted flags - supports_streaming_tts / begin_streaming_tts / write_streaming_tts / finish_streaming_tts / abort_streaming_tts - All default to unsupported/no-op so existing adapters are source-compatible - Per-turn _streaming_tts_completed_chats set suppresses duplicate whole-file auto-TTS when streaming succeeded; cleared after turn completion Gateway consumer (gateway/streaming_tts_consumer.py): - StreamingTTSConsumer: bridges sync agent deltas to async adapter audio sink - Uses existing SentenceChunker (no competing parser) - Thread-safe bounded queue; on_delta never blocks the agent worker thread - Resolves configured streaming provider via resolve_streaming_provider() - Serialises clause playback in order; flushes tail on completion - Pre-audio failure: completed=False (falls back to whole-file TTS) - Post-audio failure: completed=True, partial=True (no replay from start) - Abort is idempotent; late chunks silently dropped - Per-turn state isolated across concurrent chats Gateway integration (gateway/run.py): - message_type parameter threaded through _run_agent -> _run_agent_inner - StreamingTTSConsumer created when voice input + auto-TTS + provider active - Delta callback teed to both text stream consumer and TTS consumer - TTS-only delta callback installed when text streaming is off - finish() called from executor; wait_complete() in async context after - Barge-in aborts the consumer at all three interrupt detection points - Runner-level _send_voice_reply suppressed when streaming TTS completed Tests (tests/gateway/test_streaming_tts_consumer.py): - 15 focused tests: adapter defaults, lifecycle, ordered chunks, unsupported/No-streamer fallback, abort idempotency, late-chunk drop, pre/post-audio failure, concurrent-turn isolation, think-block suppression, queue backpressure Does not touch desktop/TUI code or add config flags. Plugin TTS provider stream() metadata gap (#47896) is explicitly out of scope — built-in ElevenLabs/OpenAI PCM streamers are the first consumers. Refs: #60671, #47896 |
||
|---|---|---|
| .. | ||
| assets | ||
| builtin_hooks | ||
| platforms | ||
| relay | ||
| __init__.py | ||
| authz_mixin.py | ||
| cgroup_cleanup.py | ||
| channel_directory.py | ||
| code_skew.py | ||
| config.py | ||
| cwd_placeholder.py | ||
| dead_targets.py | ||
| delivery.py | ||
| delivery_ledger.py | ||
| display_config.py | ||
| drain_control.py | ||
| hooks.py | ||
| kanban_watchers.py | ||
| memory_monitor.py | ||
| message_timestamps.py | ||
| mirror.py | ||
| pairing.py | ||
| platform_registry.py | ||
| profile_routing.py | ||
| readiness.py | ||
| response_filters.py | ||
| restart.py | ||
| restart_loop_guard.py | ||
| rich_sent_store.py | ||
| run.py | ||
| runtime_footer.py | ||
| scale_to_zero.py | ||
| session.py | ||
| session_context.py | ||
| shutdown_flush.py | ||
| shutdown_forensics.py | ||
| shutdown_watchdog.py | ||
| slash_access.py | ||
| slash_commands.py | ||
| status.py | ||
| status_phrases.py | ||
| sticker_cache.py | ||
| stream_consumer.py | ||
| stream_dispatch.py | ||
| stream_events.py | ||
| streaming_tts_consumer.py | ||
| systemd_notify.py | ||
| turn_lease.py | ||
| wake.py | ||
| whatsapp_identity.py | ||