mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-31 19:16:29 +00:00
709dd3282f
spectrum-ts's live-stream consumer (consumeLive in spectrum-ts 3.x) only
reconnects when its inbound async iterator throws or ends. A half-open
("zombie") gRPC socket — where the TCP connection stays ESTABLISHED but the
peer is gone (NAT idle-timeout, network blip, laptop sleep) — makes the
iterator hang forever: no error, no end. The SDK exposes no gRPC keepalive
knob (createClient takes only {address, tls, token}; grpc.keepalive_time_ms
defaults to -1 = pings off), so the inbound stream silently dies and stays
dead until the gateway is restarted. Symptom: the agent's iMessage line goes
"online but deaf" — Photon's cloud-side fallback answers users with "the agent
isn't online right now" and inbound never reaches the gateway.
Fix, entirely in the code we own (no SDK fork):
- Sidecar gains a POST /probe endpoint that drives a cheap unary read
(space.getMessage on a synthetic id) over the SAME gRPC channel the inbound
stream uses. A live channel round-trips in ms (server returns not-found,
which is success for liveness); a zombie hangs. It sends nothing to any user
and creates no chat (space.get is local in shared/dedicated mode; only the
message read touches the wire).
- The adapter runs a presence watchdog: it probes on an interval, skips the
probe when natural inbound traffic already proved liveness within the
window, and after N consecutive failed probes respawns the sidecar — a fresh
Spectrum() re-subscribes the stream and re-registers presence. Successful
probes double as application-level keepalive, helping prevent the zombie
from forming at all. Respawn is lock-guarded against double-spawn and the
watchdog is torn down cleanly on disconnect.
Behavioural settings live in config.yaml (extra), bridged to env per the
.env-is-secrets-only convention:
probe_interval_seconds (60), probe_timeout_seconds (10),
probe_max_failures (3). A non-positive interval disables the watchdog.
Tests: tests/plugins/platforms/photon/test_presence_watchdog.py covers config
resolution, the disable switch, probe alive/dead(500)/timeout/no-client, the
core N-failures->one-respawn detection, success-resets-failures, stop-then-
start respawn ordering, and lock-guarding — all without spawning Node or
hitting the network.
Contributed by Vaibhav Sharma (X: @vabbyshabby).
|
||
|---|---|---|
| .. | ||
| test_auth.py | ||
| test_check_requirements_risks.py | ||
| test_fatal_notify_self_cancel.py | ||
| test_inbound.py | ||
| test_markdown.py | ||
| test_mention_gating.py | ||
| test_npm_error_log_regression.py | ||
| test_outbound_media.py | ||
| test_overflow_recovery.py | ||
| test_poll_clarify.py | ||
| test_presence_watchdog.py | ||
| test_reactions.py | ||
| test_rich_links.py | ||
| test_runtime_record.py | ||
| test_setup_access.py | ||
| test_sidecar_deps_stale.py | ||
| test_sidecar_lifecycle.py | ||
| test_spectrum_patch.py | ||
| test_streaming.py | ||
| test_url_send_path.py | ||