mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-29 06:31:32 +00:00
Long-running gateway processes that survive 'hermes update' keep pre-update modules cached in sys.modules. When new tool files on disk then try to 'from hermes_cli.config import cfg_get' (added in PR #17304), the import resolves against the stale module object and raises ImportError — hitting users on Matrix, Telegram, Feishu, and other platforms. Two defenses: 1. Gateway self-check (gateway/run.py). On __init__, snapshot the newest mtime across sentinel source files (hermes_cli/config.py, run_agent.py, gateway/run.py, etc.). On every inbound message, re-read those mtimes; if any is newer than boot time + 2s slack, request a graceful restart via the normal drain path and return a one-line ack to the user. Idempotent, works regardless of how the update happened (hermes update, manual git pull, installer). 2. Post-restart survivor sweep ('hermes update'). After the existing restart loop, sleep 3s, rescan for gateway PIDs we already tried to kill, and SIGKILL any survivors. The detached profile watchers and systemd then relaunch with fresh code instead of waiting out the 120s watcher timeout. Closes #17648.
This commit is contained in:
parent
77c0bc6b13
commit
f99676e315
3 changed files with 399 additions and 0 deletions
|
|
@ -7548,6 +7548,42 @@ def _cmd_update_impl(args, gateway_mode: bool):
|
|||
# No gateways were running — nothing to do
|
||||
pass
|
||||
|
||||
# --- Post-restart survivor sweep -----------------------------
|
||||
# Issue #17648: some gateways ignore SIGTERM (stuck drain,
|
||||
# blocked I/O, PID dead but zombie). The detached profile
|
||||
# watchers wait 120s for the old PID to exit — if it never
|
||||
# does, no respawn happens and the user keeps hitting
|
||||
# ImportError against a stale sys.modules. Give the
|
||||
# graceful paths a brief window to complete, then SIGKILL
|
||||
# any remaining pre-update PIDs so the watcher / service
|
||||
# manager can relaunch with fresh code.
|
||||
try:
|
||||
_time.sleep(3.0)
|
||||
_service_pids_after = _get_service_pids()
|
||||
_surviving = find_gateway_pids(
|
||||
exclude_pids=_service_pids_after, all_profiles=True,
|
||||
)
|
||||
# Scope to PIDs we already tried to kill during this
|
||||
# update (killed_pids). Anything new is a gateway that
|
||||
# started AFTER our restart attempt — respecting user
|
||||
# intent, we don't kill those.
|
||||
_stuck = [pid for pid in _surviving if pid in killed_pids]
|
||||
if _stuck:
|
||||
print()
|
||||
print(
|
||||
f" ⚠ {len(_stuck)} gateway process(es) ignored SIGTERM — force-killing"
|
||||
)
|
||||
for pid in _stuck:
|
||||
try:
|
||||
os.kill(pid, _signal.SIGKILL)
|
||||
except (ProcessLookupError, PermissionError):
|
||||
pass
|
||||
# Give the OS a beat to reap the processes so the
|
||||
# watchers see them exit and respawn.
|
||||
_time.sleep(1.5)
|
||||
except Exception as _sweep_exc:
|
||||
logger.debug("Post-restart survivor sweep failed: %s", _sweep_exc)
|
||||
|
||||
except Exception as e:
|
||||
logger.debug("Gateway restart during update failed: %s", e)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue