mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-31 19:16:29 +00:00
5081551f09
Replaces the half-duplex per-playback barge monitors with ONE listener that runs for the entire agent turn in continuous voice mode: armed at utterance-submit, disarmed when the turn is fully done (response + TTS finished). Fixes Teknium's live report that voice interruption never works: (a) not while the LLM is generating, (b) not while TTS plays. Root causes: - HALF-DUPLEX GAP: the barge monitor only spawned when TTS playback STARTED (cli.py streaming/whole-file paths, gateway _tts_stream_begin). During LLM generation there was NO microphone listener at all. - PLAYBACK DEAFNESS: the monitor calibrated its VAD noise floor WHILE the speaker was blasting TTS (speaker bleed baked into the floor), then multiplied it by 8x with a 1s strictly-consecutive block requirement — normal speech could rarely reach the trigger, and the 2s grace swallowed early interjections. New model — tools/voice_mode.full_duplex_listen(): - Pre-playback calibration: quiet-room noise floor established at turn start and HELD through playback (never recalibrated against bleed). - Phase-aware trigger: generation = floor x voice.barge_in_threshold_multiplier (new config, default 3.0, justified by synthetic-frame tests); playback = additionally clamped to a 1500-RMS minimum so bleed alone can't trip; 4000-RMS ceiling keeps speech always reachable. - Windowed-majority detection (>=80% of a 300ms window) instead of the strictly-consecutive counter that reset on intra-word energy dips. - Grace on playback ONSET only (voice.barge_in_grace_seconds, default down 2.0 -> 0.5) — suppresses the onset transient, not the mic. - Debug diagnostics at every decision point (calibrated floor, per-window RMS above 50% of trigger, trip/no-trip, grace suppressions) — always logger.debug, mirrored to stderr under HERMES_VOICE_DEBUG=1. Phase behavior (CLI cli.py + tui_gateway/server.py, same model): - generation: speech interrupts the in-flight turn via the SAME seam the typed/Ctrl+C interrupt uses (agent.interrupt()), cuts any pending TTS pipeline so the stale reply never plays, and submits the captured interjection (pre-roll capture, first syllable kept) as the next turn. - playback: cuts TTS (streaming pipeline stop + fallback speak stop events + file player) and submits the capture. - stop phrase honored in BOTH phases: mid-generation 'stop' interrupts the turn AND ends the voice chat (stop everything). - one listener instance spans generation -> playback (no re-arm race); double-arm refused (CLI _voice_fd_active / gateway _fd_listener_active). Gateway specifics: _arm_full_duplex_listener() at _run_prompt_submit turn start and inside _tts_stream_begin; _speak_text_with_barge registers its (stop, done) pair in _fd_speak_pipelines so fallback speaks are cut and tracked; _tts_stream_barge_in_monitor kept as a shim that arms the new listener. Desktop renderer owns its own mic path (voice-barge-in.ts) and is unaffected; if desktop backend-mic mode is used it inherits via the gateway. Tests: full_duplex_listen synthetic-RMS suite (speech-over-bleed trips, bleed alone doesn't, quiet floor held through playback, grace window, multiplier math 3x vs 8x, windowed-majority dips), CLI listener phase tests (generation interrupt seam, playback cut, lifecycle spans phases, double-arm, config forwarding, stop-phrase-mid-generation), gateway generation-phase interrupt + stop-phrase tests. Generation-interrupt test sabotage-verified. |
||
|---|---|---|
| .. | ||
| acp | ||
| acp_adapter | ||
| agent | ||
| ci | ||
| cli | ||
| computer_use | ||
| conformance | ||
| cron | ||
| dashboard | ||
| docker | ||
| e2e | ||
| fakes | ||
| fixtures/plugins/example-dashboard/dashboard | ||
| gateway | ||
| hermes_cli | ||
| hermes_state | ||
| honcho_plugin | ||
| integration | ||
| manual | ||
| monitoring | ||
| openviking_plugin | ||
| plugins | ||
| providers | ||
| run_agent | ||
| scripts | ||
| secret_sources | ||
| skills | ||
| state | ||
| stress | ||
| tools | ||
| tui_gateway | ||
| website | ||
| __init__.py | ||
| conftest.py | ||
| run_interrupt_test.py | ||
| test_account_usage.py | ||
| test_atomic_replace_symlinks.py | ||
| test_audio_playback_guard.py | ||
| test_background_review_list_shapes.py | ||
| test_background_review_session_isolation.py | ||
| test_base_url_hostname.py | ||
| test_batch_runner_checkpoint.py | ||
| test_bitwarden_secrets.py | ||
| test_cli_file_drop.py | ||
| test_cli_manual_compress.py | ||
| test_cli_skin_integration.py | ||
| test_code_skew.py | ||
| test_command_secret_source.py | ||
| test_conftest_wal_gate.py | ||
| test_copilot_initiator.py | ||
| test_credential_file_permissions.py | ||
| test_ctx_halving_fix.py | ||
| test_delegate_cascade_49148.py | ||
| test_dispatch_session_id.py | ||
| test_empty_model_fallback.py | ||
| test_empty_session_hygiene.py | ||
| test_env_loader_applied_homes.py | ||
| test_env_loader_op_bootstrap.py | ||
| test_env_loader_secret_sources.py | ||
| test_evidence_store.py | ||
| test_fast_safe_load.py | ||
| test_fts_cjk_bigram.py | ||
| test_gateway_streaming_nested_config.py | ||
| test_get_tool_definitions_cache_isolation.py | ||
| test_hermes_bootstrap.py | ||
| test_hermes_constants.py | ||
| test_hermes_home_profile_warning.py | ||
| test_hermes_logging.py | ||
| test_hermes_state.py | ||
| test_hermes_state_compression_locks.py | ||
| test_hermes_state_readonly_preflight.py | ||
| test_hermes_state_wal_fallback.py | ||
| test_honcho_client_concurrency.py | ||
| test_honcho_client_config.py | ||
| test_honcho_session_context.py | ||
| test_honcho_startup_fail_open.py | ||
| test_install_autostash_conflict_recovery.py | ||
| test_install_diverged_update.py | ||
| test_install_lockfile_churn.py | ||
| test_install_macos_launcher.py | ||
| test_install_no_initial_commit.py | ||
| test_install_ps1_ascii_only.py | ||
| test_install_ps1_native_stderr_eap.py | ||
| test_install_ps1_node_path_for_npm.py | ||
| test_install_ps1_python_fallback_venv.py | ||
| test_install_ps1_uv_powershell_host.py | ||
| test_install_ps1_web_server_syntax_probe.py | ||
| test_install_sh_acp_launcher.py | ||
| test_install_sh_bootstrap_marker.py | ||
| test_install_sh_browser_install.py | ||
| test_install_sh_install_method_stamp.py | ||
| test_install_sh_node_global_prefix.py | ||
| test_install_sh_pythonpath_sanitization.py | ||
| test_install_sh_root_fhs_uv_python_path.py | ||
| test_install_sh_setup_wizard_tty_probe.py | ||
| test_install_sh_symlink_stomp.py | ||
| test_install_sh_termux_network_prereqs.py | ||
| test_install_unmerged_index.py | ||
| test_ipv4_preference.py | ||
| test_iron_proxy.py | ||
| test_iron_proxy_cli.py | ||
| test_iron_proxy_e2e.py | ||
| test_lazy_session_regressions.py | ||
| test_lint_config.py | ||
| test_live_system_guard_self_test.py | ||
| test_mcp_serve.py | ||
| test_mini_swe_runner.py | ||
| test_minimax_model_validation.py | ||
| test_minimax_oauth.py | ||
| test_minisweagent_path.py | ||
| test_model_forces_max_completion_tokens.py | ||
| test_model_picker_scroll.py | ||
| test_model_tools.py | ||
| test_model_tools_async_bridge.py | ||
| test_ollama_num_ctx.py | ||
| test_onepassword_secrets.py | ||
| test_output_cap_parsing.py | ||
| test_packaging_build_guard.py | ||
| test_packaging_metadata.py | ||
| test_plugin_skills.py | ||
| test_plugin_utils.py | ||
| test_process_loop_event_loop_warning.py | ||
| test_profile_isolation_runtime.py | ||
| test_project_metadata.py | ||
| test_pty_keepalive_ws.py | ||
| test_pty_session.py | ||
| test_retry_utils.py | ||
| test_run_tests_parallel.py | ||
| test_sanitize_tool_error.py | ||
| test_search_slow_query_log.py | ||
| test_session_db_read_path_split.py | ||
| test_session_skill_previews.py | ||
| test_session_workspace_binding.py | ||
| test_slack_thread_require_mention.py | ||
| test_slash_worker_watchdog.py | ||
| test_sql_injection.py | ||
| test_sqlite_lock_safe_inspection.py | ||
| test_sqlite_wal_reset_gate.py | ||
| test_stale_utils_module_import.py | ||
| test_state_db_malformed_repair.py | ||
| test_subprocess_home_isolation.py | ||
| test_telegram_polling_progress_ptb.py | ||
| test_termux_all_extra_compat.py | ||
| test_timezone.py | ||
| test_tini_shim.py | ||
| test_toolset_distributions.py | ||
| test_toolsets.py | ||
| test_trajectory_compressor.py | ||
| test_trajectory_compressor_async.py | ||
| test_transform_llm_output_hook.py | ||
| test_transform_tool_result_hook.py | ||
| test_tui_entry_mcp_owner.py | ||
| test_tui_gateway_loop_noise.py | ||
| test_tui_gateway_queue_on_busy.py | ||
| test_tui_gateway_server.py | ||
| test_tui_gateway_ws.py | ||
| test_tui_mcp_late_refresh.py | ||
| test_utils_truthy_values.py | ||
| test_voice_max_recording_seconds.py | ||
| test_wal_checkpoint_strategy.py | ||
| test_web_server.py | ||
| test_web_server_sessiondb_eventloop.py | ||
| test_windows_subprocess_no_window_flags.py | ||
| test_yaml_indent_consistency_31999.py | ||
| test_yuanbao_integration.py | ||
| test_yuanbao_markdown.py | ||
| test_yuanbao_pipeline.py | ||
| test_yuanbao_proto.py | ||
| test_yuanbao_reconnect_set_active.py | ||
| test_yuanbao_shutdown.py | ||
| test_zeroed_state_db.py | ||