mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-27 17:58:07 +00:00
4b039e9543
Fixes #71758. A platform adapter that dies on a transient upstream failure (marked retryable=True, e.g. photon's sidecar exiting when its upstream gRPC/CDN returns errors) is correctly queued into _failed_platforms for background reconnection. But the reconnect watcher task itself was spawned via a bare asyncio.create_task -- if an exception ever escaped its OUTER while-loop (not just the per-platform inner try/except), the watcher died silently: no log, no restart. _ensure_reconnect_watcher_running() already existed to respawn a dead watcher, but it's only called from _handle_adapter_fatal_error_impl() when a NEW platform's fatal error arrives. If the watcher dies while a platform is already sitting in the queue and no OTHER platform ever fails afterward, nothing ever notices the watcher is dead -- exactly matching the reported symptom: photon queued for reconnect, the gateway itself healthy (other platforms kept working, so nothing re-triggered the ensure-alive check), and the platform stayed dead for 17.5h until a manual restart, well after the transient upstream outage had recovered. Fix: spawn the reconnect watcher via the existing _spawn_supervised() task-level supervisor (already used for kanban_dispatcher_watcher, handoff_watcher, etc.) instead of a bare asyncio.create_task, at both the initial startup spawn and the manual-respawn path in _ensure_reconnect_watcher_running(). _spawn_supervised already provides exactly what's missing here: catches and logs any exception escaping the task, and auto-restarts with capped exponential backoff (healthy-run counter resets so a daemon that crashes occasionally over days is never permanently abandoned) -- self-healing independent of any new fatal-error event. Also hardened a related race: the watcher's per-platform loop looked up self._failed_platforms[platform] via direct indexing after snapshotting the keys with list(...). A platform removed concurrently between the snapshot and the lookup (e.g. a manual /platform resume, or a reconnect that succeeded via a different path) would raise an uncaught KeyError -- exactly the class of bug this fix's supervision now catches, but avoiding the crash-and-restart cycle entirely is better than relying on it. Changed to .get() with a skip-if-missing guard. 6 new tests pass (initial spawn uses _spawn_supervised, manual respawn uses _spawn_supervised, the core regression -- watcher self-heals after an uncaught exception with no new fatal-error event -- and the race-guard scenario); 52/52 in the full tests/gateway/test_platform_reconnect.py file (including the 4 pre-existing _ensure_reconnect_watcher_running tests, confirming no regression to that respawn-when-dead-or-missing behavior). |
||
|---|---|---|
| .. | ||
| assets | ||
| builtin_hooks | ||
| platforms | ||
| relay | ||
| __init__.py | ||
| authz_mixin.py | ||
| cgroup_cleanup.py | ||
| channel_directory.py | ||
| code_skew.py | ||
| config.py | ||
| cwd_placeholder.py | ||
| dead_targets.py | ||
| delivery.py | ||
| delivery_ledger.py | ||
| display_config.py | ||
| drain_control.py | ||
| hooks.py | ||
| kanban_watchers.py | ||
| memory_monitor.py | ||
| message_timestamps.py | ||
| mirror.py | ||
| pairing.py | ||
| platform_registry.py | ||
| profile_routing.py | ||
| readiness.py | ||
| response_filters.py | ||
| restart.py | ||
| restart_loop_guard.py | ||
| rich_sent_store.py | ||
| run.py | ||
| runtime_footer.py | ||
| scale_to_zero.py | ||
| session.py | ||
| session_context.py | ||
| shutdown_forensics.py | ||
| shutdown_watchdog.py | ||
| slash_access.py | ||
| slash_commands.py | ||
| status.py | ||
| status_phrases.py | ||
| sticker_cache.py | ||
| stream_consumer.py | ||
| stream_dispatch.py | ||
| stream_events.py | ||
| systemd_notify.py | ||
| turn_lease.py | ||
| wake.py | ||
| whatsapp_identity.py | ||