mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-20 15:33:54 +00:00
fix(container-boot): autostart a gateway stranded in 'draining' state
A gateway hard-killed while draining (a container/VM recreate SIGTERMs it before _stop_impl reaches its terminal-state persist) leaves gateway_state.json frozen at 'draining'. With no explicit desired_state to fall back to, container_boot read that transient value literally, found it not in _AUTOSTART_STATES, and left the gateway DOWN on every subsequent boot — dashboard up, messaging silently dark. Observed on a relay-opted-in staging instance (2026-06): the s6 gateway-default slot kept its 'down' marker across recreates and the gateway never came back. 'draining' is a transient sub-state of RUNNING (written by the drain watcher / scale-to-zero go-dormant path), never an operator stop and never a failed boot. Normalise it to 'running' in the gateway_state fallback so a stranded drain marker reads as the run-intent it represents. This extends gateway/run.py's #42675 handling (persist 'running' on an unexpected signal) to the case where the gateway died before persisting anything at all. 'starting'/'startup_failed' are deliberately NOT normalised — those mean a mid-boot death and must stay down to avoid the crash-loop the down-marker guard prevents. An explicit desired_state still wins verbatim, so an operator stop survives a transient 'draining' runtime value. Tests: draining named-profile + default-root autostart (both fail without the fix), plus a guard that an explicit desired_state=stopped still blocks a draining runtime.
This commit is contained in:
parent
d4c14011eb
commit
d3f2931b8c
2 changed files with 102 additions and 1 deletions
|
|
@ -38,6 +38,29 @@ log = logging.getLogger(__name__)
|
|||
# durable start/stop intent across pod/container recreation.
|
||||
_AUTOSTART_STATES = frozenset({"running"})
|
||||
|
||||
# Transient runtime sub-states of a RUNNING gateway. A gateway only ever
|
||||
# reaches these while it is up and serving — `draining` is written by the
|
||||
# drain watcher / scale-to-zero go-dormant path when an in-flight quiesce
|
||||
# begins (gateway/run.py). It is NOT an operator stop and NOT a failed boot.
|
||||
#
|
||||
# When a gateway is hard-killed *while draining* (a container/VM recreate
|
||||
# SIGTERMs it before `_stop_impl` reaches its terminal-state persist), the
|
||||
# last value left in gateway_state.json is `draining`. With no explicit
|
||||
# `desired_state` to fall back to, treating that literal value as the
|
||||
# autostart intent would leave the gateway DOWN on every subsequent boot —
|
||||
# the gateway never comes back, the dashboard is up but messaging stays dark
|
||||
# (observed on a relay-opted-in staging instance, 2026-06). Map these
|
||||
# transient sub-states to `running` so a stranded drain marker reads as the
|
||||
# run-intent it actually represents. This mirrors gateway/run.py's #42675
|
||||
# handling, which persists `running` (not the mid-shutdown `draining`) when an
|
||||
# unexpected signal tears the gateway down — extended here to the case where
|
||||
# the gateway died before it could persist anything at all.
|
||||
#
|
||||
# `starting` / `startup_failed` are deliberately NOT included: those mean the
|
||||
# gateway died mid-boot or failed to come up, so auto-restarting them would
|
||||
# reintroduce the crash-loop the down-marker guard exists to prevent.
|
||||
_TRANSIENT_RUNNING_STATES = frozenset({"draining"})
|
||||
|
||||
# Stale runtime files we sweep before recreating service slots. These
|
||||
# all hold container-namespaced state (PIDs, process tables) that's
|
||||
# garbage post-restart — a numerically-equal PID in the new container
|
||||
|
|
@ -324,6 +347,15 @@ def _read_desired_state(profile_dir: Path) -> str | None:
|
|||
that as a compatibility fallback so existing running/stopped profiles
|
||||
preserve their behavior until the next explicit start/stop.
|
||||
|
||||
When falling back to ``gateway_state`` (no explicit ``desired_state``),
|
||||
a transient running sub-state (``draining``) is normalised to ``running``
|
||||
— see ``_TRANSIENT_RUNNING_STATES``. A gateway hard-killed mid-drain
|
||||
leaves ``draining`` as its last persisted value; without this it would be
|
||||
treated as a non-autostart state and the gateway would stay DOWN forever.
|
||||
An explicit ``desired_state`` is always honoured verbatim (it is the
|
||||
operator's durable intent), so this normalisation only affects the
|
||||
legacy/transient fallback path.
|
||||
|
||||
Missing or unparseable files count as "no desired state" so we don't
|
||||
bork the whole reconciliation on a corrupt file.
|
||||
"""
|
||||
|
|
@ -335,7 +367,10 @@ def _read_desired_state(profile_dir: Path) -> str | None:
|
|||
desired_state = data.get("desired_state")
|
||||
if desired_state is not None:
|
||||
return desired_state
|
||||
return data.get("gateway_state")
|
||||
gateway_state = data.get("gateway_state")
|
||||
if gateway_state in _TRANSIENT_RUNNING_STATES:
|
||||
return "running"
|
||||
return gateway_state
|
||||
except (OSError, json.JSONDecodeError):
|
||||
log.warning(
|
||||
"could not read %s; treating as no prior state", state_file,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue