mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-24 05:41:40 +00:00
fix(gateway): keep running when platforms fail; add per-platform circuit breaker + /platform (#26600)
Stop the gateway from exiting (or systemd-restart-looping) when a single
messaging adapter fails at startup or runtime. A misconfigured WhatsApp
(npm install timeout, unpaired bridge, missing creds.json) used to take
the entire gateway down, killing cron jobs and any other connected
platforms with it.
Changes:
• Startup (gateway/run.py): when connected_count==0 but the only
errors are retryable, log a degraded-state warning and keep the
gateway alive instead of returning False. Reconnect watcher then
recovers platforms as their underlying problem clears.
• Runtime (gateway/run.py _handle_adapter_fatal_error): when the last
adapter goes down with a retryable error and is queued for
reconnection, stay alive instead of exit-with-failure. Previously
this triggered systemd Restart=on-failure, which created infinite
restart loops on persistent retryable failures (proxy outage,
repeated bridge crashes).
• Reconnect watcher (gateway/run.py _platform_reconnect_watcher):
replace the 20-attempt hard drop with a circuit-breaker pause.
After _PAUSE_AFTER_FAILURES (10) consecutive retryable failures, the
platform stays in _failed_platforms with paused=True so the watcher
skips it but the operator can still see and resume it. Non-retryable
errors still drop out of the queue immediately. Resolves #17063
(gateway giving up on Telegram after 20 attempts).
• WhatsApp preflight (gateway/platforms/whatsapp.py): refuse to start
the Node bridge when creds.json is missing. Sets a non-retryable
whatsapp_not_paired fatal error so the watcher drops it cleanly
with a single 'run hermes whatsapp' log line instead of paying the
30s bridge bootstrap timeout on every gateway start.
• WhatsApp setup ordering (hermes_cli/main.py cmd_whatsapp): only set
WHATSAPP_ENABLED=true once pairing actually succeeds. Previously
the wizard wrote the env var at step 2 (before npm install and QR
pairing), so any Ctrl+C left .env claiming WhatsApp was ready when
the bridge had no creds.json. Also propagate the env var when the
user keeps an existing pairing on a re-run.
• /platform slash command (hermes_cli/commands.py + gateway/run.py):
new gateway-only command for manual circuit-breaker control.
/platform list — show connected + failed/paused platforms
/platform pause <name> — silence a known-broken platform
/platform resume <name> — re-queue a paused platform
Tests:
• New: pause/resume helpers, /platform list|pause|resume command,
WhatsApp creds.json preflight, WhatsApp setup ordering.
• Updated: stale assertions that codified the old 'exit and let
systemd restart' behavior in test_runner_fatal_adapter.py,
test_runner_startup_failures.py, and test_platform_reconnect.py
(the 20-attempt give-up test became a circuit-breaker pause test).
5488 tests pass in tests/gateway/.
This commit is contained in:
parent
3b9368a0c4
commit
518f39557b
9 changed files with 745 additions and 62 deletions
|
|
@ -493,13 +493,45 @@ class WhatsAppAdapter(BasePlatformAdapter):
|
|||
"""
|
||||
if not check_whatsapp_requirements():
|
||||
logger.warning("[%s] Node.js not found. WhatsApp requires Node.js.", self.name)
|
||||
self._set_fatal_error(
|
||||
"whatsapp_node_missing",
|
||||
"Node.js is not installed — install Node.js and re-run `hermes gateway`.",
|
||||
retryable=False,
|
||||
)
|
||||
return False
|
||||
|
||||
bridge_path = Path(self._bridge_script)
|
||||
if not bridge_path.exists():
|
||||
logger.warning("[%s] Bridge script not found: %s", self.name, bridge_path)
|
||||
self._set_fatal_error(
|
||||
"whatsapp_bridge_missing",
|
||||
f"WhatsApp bridge script missing at {bridge_path}.",
|
||||
retryable=False,
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
# Pre-flight: skip the 30s bridge bootstrap entirely if the user
|
||||
# never finished pairing. Without creds.json the bridge prints
|
||||
# QR codes to its log file and never reaches status:connected,
|
||||
# so every gateway restart paid the 30s timeout + queued WhatsApp
|
||||
# for indefinite retries. Mark non-retryable so the user gets a
|
||||
# clear "run hermes whatsapp" message instead of the watcher
|
||||
# silently hammering an unconfigured platform.
|
||||
creds_path = self._session_path / "creds.json"
|
||||
if not creds_path.exists():
|
||||
logger.warning(
|
||||
"[%s] WhatsApp is enabled but not paired (no creds.json at %s). "
|
||||
"Run `hermes whatsapp` to pair, or remove WHATSAPP_ENABLED from "
|
||||
"your .env to disable.",
|
||||
self.name, creds_path,
|
||||
)
|
||||
self._set_fatal_error(
|
||||
"whatsapp_not_paired",
|
||||
"WhatsApp enabled but not paired — run `hermes whatsapp` to pair.",
|
||||
retryable=False,
|
||||
)
|
||||
return False
|
||||
|
||||
logger.info("[%s] Bridge found at %s", self.name, bridge_path)
|
||||
|
||||
# Acquire scoped lock to prevent duplicate sessions
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue