fix(gateway): auto-start after container restart via planned-stop marker (#42675) (#43236)

* fix(gateway): auto-start after container restart via planned-stop marker

On Docker (s6-overlay), the gateway runs as a dynamically-registered s6
service. When the container stops/restarts/upgrades, s6 sends the gateway
a plain SIGTERM. The shutdown path (_stop_impl) ended with an
unconditional _update_runtime_status("stopped"), persisting
gateway_state=stopped to the volume. container_boot.py reads that on the
next boot and only auto-starts gateways whose last state was "running"
(_AUTOSTART_STATES) — so after a routine `docker compose up
--force-recreate` the gateway stays down and messaging channels silently
go dark, with no error surfaced (issue #42675).

The codebase already distinguishes intentional stops from unexpected
signals via the planned-stop marker (write_planned_stop_marker /
consume_planned_stop_marker_for_self): `hermes gateway stop`,
systemd/launchd ExecStop, and Ctrl+C write a marker before signalling,
so the handler classifies them as planned. An unmarked SIGTERM
(container/s6 restart, OOM, bare kill) is signal-initiated.

This wires that existing classification through to the state persist,
rather than adding unreliable signal-source inference:

- run.py: GatewayRunner._signal_initiated_shutdown, set in
  shutdown_signal_handler's unmarked-signal branch. In _stop_impl, a
  signal-initiated (non-restart) teardown now persists "running" instead
  of "stopped" — preserving the operator's run-intent and overwriting the
  mid-shutdown "draining" marker so _AUTOSTART_STATES matches on reboot.
  Operator stops and restarts persist "stopped" as before.

- service_manager.py: S6ServiceManager.stop() now writes the planned-stop
  marker for the supervised PID (read from s6-svstat) before `s6-svc -d`,
  so an in-container `hermes gateway stop` is correctly classified as
  intentional (parity with the systemd/launchd/host stop paths, which
  already mark). Best-effort: a marker-write failure falls back to the
  safe signal-initiated path.

Tests: shutdown persist-decision table (signal→running, operator→stopped,
restart→stopped), s6 stop marker write + svstat PID parse + failure
tolerance. The signal→running and s6-marker tests fail without the
respective source change. Verified end-to-end against a container built
from this branch: an unmarked SIGTERM to the live gateway leaves
gateway_state=running (shutdown-context log confirms signal path);
existing real container-restart suite still green.

* docs(docker): clarify gateway autostart distinguishes operator-stop from container-kill

The per-profile-supervision section described the autostart-across-restart
contract as "running gateways come back, stopped stay stopped" without
spelling out what records 'stopped'. That contract was the source of
#42675 confusion: users expected a restart to bring the gateway back and
it didn't. With the write-side fix, only an explicit `hermes gateway stop`
records 'stopped'; container/s6 restart SIGTERMs (incl. image upgrades and
unexpected exits) leave the state 'running' so the gateway auto-starts.
Make that distinction explicit in both the multi-profile and
per-profile-supervision sections.

* test(docker): real-restart autostart E2E for #42675

Adds test_live_gateway_autostarts_after_real_restart_without_manual_state_stamp:
a live s6-supervised gateway is killed by an actual `docker restart`
SIGTERM (no manual gateway_state stamp, no planned-stop marker) and must
auto-start on the next boot. Exercises the WRITE side of the fix that the
existing stamp-based tests bypass.

Verified to FAIL against an origin/main image (reconciler logs
prior_state=stopped action=registered — the #42675 bug) and PASS against
the fixed image (prior_state=running action=started).
This commit is contained in:
Ben Barclay 2026-06-10 14:01:34 +10:00 committed by GitHub
parent b4170f3ac2
commit 5cf6e28a2f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 363 additions and 3 deletions

View file

@ -739,13 +739,55 @@ class S6ServiceManager:
"""
self._run_svc("-u", "start", name)
def _supervised_pid(self, name: str) -> int | None:
"""Return the PID of the supervised gateway process, or None.
Parses ``s6-svstat`` output (``up (pid NNNN) ...``). Used to
mark an operator-initiated stop with the planned-stop marker so
the gateway's shutdown handler classifies the incoming SIGTERM
as intentional rather than an unexpected kill (issue #42675).
Best-effort: any parse/exec failure returns None.
"""
import subprocess
try:
result = subprocess.run(
[f"{_S6_BIN_DIR}/s6-svstat", str(self.scandir / name)],
capture_output=True, text=True, timeout=5,
)
except (OSError, subprocess.SubprocessError):
return None
if result.returncode != 0:
return None
m = re.search(r"\(pid (\d+)\)", result.stdout)
return int(m.group(1)) if m else None
def stop(self, name: str) -> None:
"""Bring down a registered service (``s6-svc -d``).
Writes a planned-stop marker naming the supervised gateway PID
BEFORE sending the down command, so the gateway's shutdown
handler recognises this SIGTERM as an operator-initiated stop
and persists ``gateway_state=stopped`` (respecting the explicit
intent). Without the marker, an intentional ``hermes gateway
stop`` is indistinguishable from the container/s6 SIGTERM sent on
``docker restart``; the latter must NOT persist ``stopped`` or
container_boot refuses to auto-start on the next boot (#42675).
The marker write is best-effort a failure only means the stop
is treated as signal-initiated, which is the safe fallback.
Raises:
GatewayNotRegisteredError: no service directory for ``name``.
S6CommandError: s6-svc exited non-zero for any other reason.
"""
pid = self._supervised_pid(name)
if pid is not None:
try:
from gateway.status import write_planned_stop_marker
write_planned_stop_marker(pid)
except Exception:
pass
self._run_svc("-d", "stop", name)
def restart(self, name: str) -> None: