mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-31 19:16:29 +00:00
7739f5fc5b
Implements the trace-point suggestions from the Hermes Cloud health report. A live gateway plus a service supervisor (systemd Restart=, s6, Fly machine restart) respawning a second 'gateway run' produces a tight crash-loop: every losing starter refuses to claim the PID file / runtime lock and exits, while the original gateway stays up. A point-in-time /api/status probe reports the box as healthy (gateway_running: true) so the loop is invisible. Confirmed in prod on girt-numbat-4665: probe healthy while emitting ~24k 'Gateway already running' lines/24h, only detectable via a fragile ILIKE log scan. - gateway/status.py: add record_start_blocked(reason, pid) and count_recent_start_blocks(window). record_start_blocked emits a single canonical, exact-match log token 'gateway.start_blocked' with stable reason=/pid=/version= key=value fields (replacing the ILIKE '%already running%' scan and making loops attributable to a release cohort), and persists a self-pruning, capped ring of recent block timestamps under HERMES_HOME so a separate process can observe the loop. - gateway/run.py: instrument all four start-refusal sites (already_running, runtime_lock_held, pid_file_race, startup_race) with accurate reason tokens. Existing log messages are preserved unchanged (tests assert on them). - hermes_cli/web_server.py + gateway/platforms/api_server.py: surface gateway_restarts_5m on /api/status and /health/detailed, read from the shared ring, so a wedged-but-looping box stops reading as cleanly healthy. No outbound telemetry, no new env var, no third-party identifiers: this is local log + existing public liveness fields only. version is already exposed on /api/status, so the 'app_version cohort' suggestion is realized as the version= field on the structured log token rather than a Prometheus label (the repo ships no Prometheus exporter). Tests: 9 new in tests/gateway/test_status.py (record/count, windowing, self-prune, cap, corrupt-file degradation, canonical log token) and 2 in tests/hermes_cli/test_web_server.py (field present + reflects the ring). 0 new failures in the gateway suite. |
||
|---|---|---|
| .. | ||
| assets | ||
| builtin_hooks | ||
| platforms | ||
| relay | ||
| __init__.py | ||
| authz_mixin.py | ||
| channel_directory.py | ||
| code_skew.py | ||
| config.py | ||
| delivery.py | ||
| display_config.py | ||
| hooks.py | ||
| kanban_watchers.py | ||
| memory_monitor.py | ||
| message_timestamps.py | ||
| mirror.py | ||
| pairing.py | ||
| platform_registry.py | ||
| response_filters.py | ||
| restart.py | ||
| rich_sent_store.py | ||
| run.py | ||
| runtime_footer.py | ||
| session.py | ||
| session_context.py | ||
| shutdown_forensics.py | ||
| slash_access.py | ||
| slash_commands.py | ||
| status.py | ||
| sticker_cache.py | ||
| stream_consumer.py | ||
| stream_dispatch.py | ||
| stream_events.py | ||
| whatsapp_identity.py | ||