mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-29 18:46:59 +00:00
3c7cff843d
Follow-up to #69112. That PR hardened the shared gateway dispatch path in gateway/run.py against the *caller* being cancelled. A second, self-referential cancellation specific to PhotonAdapter sits one layer underneath it and survived that fix. PhotonAdapter is the only platform adapter that awaits _notify_fatal_error() inline, on the same task that detected the fault. Both _monitor_sidecar_health and _supervise_sidecar run as self._sidecar_health_task / self._sidecar_supervisor_task, and the notification routes into GatewayRunner._handle_adapter_fatal_error_impl, which tears the adapter down via _safe_adapter_disconnect -> disconnect(). disconnect() then cancels self._sidecar_health_task and awaits it -- which, when the health task is what raised the notification, means disconnect() cancels its own caller several plain-await frames up. disconnect()'s `task is not asyncio.current_task()` guard does not catch this. The current task where that guard evaluates is the wrapper _await_adapter_cleanup_with_timeout creates around disconnect() via asyncio.ensure_future, not the health task further up the chain, so the guard passes and the cancel lands. CancelledError stopped subclassing Exception in Python 3.8, so the `except Exception` that wrapped the inline notify call never saw it. The health task died silently mid-handoff: no log line, no "exception never retrieved" warning (cancellation is normal asyncio), and no retry. The platform stayed stranded until the gateway was restarted by hand. Fix: dispatch the notification onto a new task, the same pattern DiscordAdapter._handle_bot_task_done already uses for this reason. disconnect() can then cancel the health/supervisor task freely without that cancellation reaching the code still running the handoff, so the handoff always reaches the reconnect queue. Both Photon fatal call sites are converted: the health-poll path (observed wedging) and the sidecar-crash path (same shape, not yet observed). gateway/run.py is untouched. Observed twice on a self-hosted gateway, ~4h38m and ~52min of silent inbound outage, both cleared only by a manual restart, both post-dating #69112's merge. In each case the fatal log line appears and `queued for background reconnection` never does. Tests: new tests/plugins/platforms/photon/test_fatal_notify_self_cancel.py covers the self-cancellation (fails with CancelledError without this change), that the dispatch does not block its caller, that a failing notification warns rather than raising, and a source guard against reintroducing either inline await. Two assertions in test_overflow_recovery.py that drove these coroutines directly now drain pending tasks before asserting delivery, since the notification is deliberately no longer awaited inline. Prepared with agent assistance and reviewed before submission. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| browser | ||
| dashboard_auth | ||
| image_gen | ||
| memory | ||
| model_providers | ||
| platforms/photon | ||
| transcription | ||
| tts | ||
| video_gen | ||
| web | ||
| __init__.py | ||
| test_achievements_plugin.py | ||
| test_chronos_cron.py | ||
| test_chronos_verify.py | ||
| test_discord_runtime_failure.py | ||
| test_disk_cleanup_plugin.py | ||
| test_google_meet_audio.py | ||
| test_google_meet_node.py | ||
| test_google_meet_plugin.py | ||
| test_google_meet_realtime.py | ||
| test_hindsight_health_grace_timeout.py | ||
| test_hindsight_root_guard.py | ||
| test_kanban_attachments.py | ||
| test_kanban_dashboard_plugin.py | ||
| test_kanban_model_override.py | ||
| test_kanban_worker_runs.py | ||
| test_langfuse_plugin.py | ||
| test_nemo_relay_plugin.py | ||
| test_plugin_dashboard_auth_contract.py | ||
| test_raft_check_fn_silent.py | ||
| test_retaindb_plugin.py | ||
| test_security_guidance_plugin.py | ||
| test_teams_pipeline_plugin.py | ||