mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-29 18:46:59 +00:00
74939d2bfe
Fixes #73159. When the Photon sidecar exits unexpectedly, _supervise_sidecar() (running as self._sidecar_supervisor_task) correctly detects SIDECAR_CRASHED and calls self._notify_fatal_error(). The Gateway's fatal-error handler answers that by calling adapter.disconnect(), which calls _stop_sidecar() -- from INSIDE the very task that's currently executing this whole chain. _stop_sidecar()'s cleanup unconditionally cancelled self._sidecar_supervisor_task. Cancelling the currently-running task raises CancelledError at its own next await point (inside _notify_fatal_error() or _stop_sidecar() itself). Since asyncio.CancelledError inherits from BaseException (not Exception), the Gateway's `except Exception` guards around the fatal-error handler don't catch it -- the handler aborts before ever reaching the "queue platform for background reconnection" step. Photon then stays permanently in `retrying` state until the whole process is manually restarted, even though detection worked correctly and the underlying transient upstream outage had long since recovered. Fix: in _stop_sidecar()'s cleanup, check whether self._sidecar_supervisor_task is asyncio.current_task() before cancelling it. A task cannot legally cancel itself in any useful way anyway (the cancellation only takes effect at its own next await, which is exactly the corruption described above) -- when we're inside the supervisor's own call stack, it's already in the process of finishing on its own once _notify_fatal_error() returns, so skip the cancel and just clear the reference. This mirrors an existing precedent in the same file: disconnect() already guards its OTHER task-cancellation (self._sidecar_health_task) the same way (`if task is not asyncio.current_task()`), just not the supervisor task in _stop_sidecar(). Per the issue's own note that existing tests mock out _notify_fatal_error() entirely (so this integration chain was never exercised), added two tests that drive the REAL chain: one runs _supervise_sidecar() as an actual asyncio task with a real _notify_fatal_error() that calls the real disconnect() -> _stop_sidecar(), confirming the task completes without CancelledError and that the post-disconnect reconnect-queue step actually executes; a second confirms the OTHER call path (external cleanup, a different task) still correctly cancels a running supervisor exactly as before. Reverting only the adapter.py fix (keeping the new test) reproduces the exact CancelledError from the bug report, confirming this is a genuine regression test. 2 new tests pass; 113/113 in the full tests/plugins/platforms/photon/ directory (no regression to sidecar lifecycle, overflow recovery, or health-monitoring behavior). |
||
|---|---|---|
| .. | ||
| browser | ||
| dashboard_auth | ||
| image_gen | ||
| memory | ||
| model_providers | ||
| platforms/photon | ||
| transcription | ||
| tts | ||
| video_gen | ||
| web | ||
| __init__.py | ||
| test_achievements_plugin.py | ||
| test_chronos_cron.py | ||
| test_chronos_verify.py | ||
| test_discord_runtime_failure.py | ||
| test_disk_cleanup_plugin.py | ||
| test_google_meet_audio.py | ||
| test_google_meet_node.py | ||
| test_google_meet_plugin.py | ||
| test_google_meet_realtime.py | ||
| test_hindsight_health_grace_timeout.py | ||
| test_hindsight_root_guard.py | ||
| test_kanban_attachments.py | ||
| test_kanban_dashboard_plugin.py | ||
| test_kanban_model_override.py | ||
| test_kanban_worker_runs.py | ||
| test_langfuse_plugin.py | ||
| test_nemo_relay_plugin.py | ||
| test_plugin_dashboard_auth_contract.py | ||
| test_raft_check_fn_silent.py | ||
| test_retaindb_plugin.py | ||
| test_security_guidance_plugin.py | ||
| test_teams_pipeline_plugin.py | ||