Salvaged from #57016 by @lEWFkRAD:
- cli.py: handle file:///C:/... drive-letter URIs on nt (strip the
leading slash urlparse leaves); join Termux example paths with literal
forward slashes so hints stay POSIX on Windows.
- gateway/status.py + hermes_cli/gateway.py: normalize backslashes to
forward slashes before the HERMES_HOME substring match so separator
style cannot defeat profile ownership detection.
- hermes_cli/banner.py: cprint degrades to plain print when
prompt_toolkit has no console (NoConsoleScreenBufferError on
redirected/absent Windows stdout).
- hermes_cli/browser_connect.py: posixpath.join for WSL /mnt/c/... bases
(os.path.join would emit backslashes on nt).
- Test hardening: symlink skip-guards, USERPROFILE alongside HOME for
ntpath.expanduser, SIGKILL absence skipif fixed via monkeypatch,
drive-letter URI / separator-normalization / banner-fallback coverage.
Dropped from the original PR: tests/cli/conftest.py fixture and the
AppSession _output monkeypatch — main's merged tests/cli/conftest.py
already handles that prompt_toolkit pollution.
The file-local telegram mock in test_dm_topics.py installed unconditionally
(no __file__ guard), registered a separate string-valued telegram.constants
module, and force-popped the adapter — poisoning the session for any later
telegram test in the same process (assert 'MARKDOWN_V2' in "'MarkdownV2'").
Fix at the source:
- conftest: _FakeEnumMember(str) with PTB-faithful str()==value and
repr()==<ChatType.X: 'x'>, satisfying both repr assertions and the
adapter's str(chat.type) normalization; the same object is bound to
mod.ParseMode and mod.constants.ParseMode.
- test_dm_topics.py: delete the divergent local mock installer; import the
shared conftest one.
- release.py: mailmap entry for the author.
Verified: the 5-failure cluster repro (dm_topics + slash_confirm +
approval_buttons + model_picker + network_reconnect + telegram_format in one
process) goes 83/83 green (3x); full tests/gateway single-process run drops
10 -> 5 failed, the remainder being pre-existing discord order-dep failures
out of scope here.
Salvaged from #68873. Credit to @liuhao1024 for the earliest root-cause
diagnosis of this str-enum mock class in PR #33875, two months earlier.
Fixes the telegram-mock order-dependent flake cluster.
Salvaged from PR #73982 by Alexander Russell (@AlexxRussell) — the
collector half only. _collect_history_media_paths used only
_TOOL_MEDIA_RE, which misses quoted and spaced paths that the delivery
pipeline's extract_media grammar accepts; run text content through the
same extractor so the surviving dedup consumers (auto-append lane and
bare-path filter) see every path that could actually have been
delivered.
The PR's other halves (post-stream dedup snapshot plumbing, canonical
path comparison in _deliver_media_from_response, queued-followup
snapshot union) are moot after #74495 removed the post-stream history
filter entirely.
DEFAULT_DB_PATH in hermes_state.py is computed at import time, freezing
the developer's real ~/.hermes even when a test fixture (or runtime
profile switch) later redirects HERMES_HOME. Any default SessionDB() —
e.g. gateway SessionStore — then opened the real state.db.
Add _default_db_path(): resolves get_hermes_home() fresh at call time,
while a deliberately re-pointed DEFAULT_DB_PATH (the established
monkeypatch escape hatch) still wins via an import-time snapshot
comparison, preserving existing test behavior. SessionDB.__init__ and
session_search's requirement check now use the resolver; explicit
db_path arguments are untouched.
Reimplemented from PR #11875 by @JorkeyLiu (original diff predates the
hermes_state rewrite); regression test ported and modernized.
tui_gateway/server.py freezes _hermes_home at import time, so
_launch_configured_cwd() reads the developer's real config.yaml even
under the per-test HERMES_HOME redirect. Any absolute terminal.cwd in
the real config made _completion_cwd() ignore monkeypatch.chdir and
broke the completion tests on a pristine checkout.
Patch _launch_configured_cwd to None in the autouse _reset_fuzzy_cache
fixture and add a regression test pinning that _completion_cwd resolves
via os.getcwd() under tests.
Salvaged from PR #70148 by @smfworks (rebased onto the pruned suite).
The restart-routing, systemd-support, and subprocess-HOME tests asserted
branch behavior but left part of the real probe surface unmocked, so they
fail when the suite itself runs inside a container (self-hosted CI) or a
launchd-descended shell:
- /restart routing tests: the handler also consults the real /.dockerenv —
extract the inline probe to gateway.restart.is_container_restart_context()
(patchable seam, no behavior change) and pin it False; scrub ALL four
supervisor env markers (ambient XPC_SERVICE_NAME on macOS flipped one).
- supports_systemd_services tests: pin shutil.which('systemctl') and
is_container() so the test asserts the branch, not the host.
- copilot ACP real-HOME test: pin is_container() (auto mode prefers profile
home in containers) and scrub ambient HERMES_REAL_HOME/TERMINAL_HOME_MODE.
97 tests green on macOS dev box AND inside a docker CI runner container.
Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
Manual port of @jethac's #63522 onto the pruned tree: 3 of the original
6 fixed-budget poll sites survive (3-concurrent-agents wait, session-A/B
routing wait, two-session queue wait). Replaced each 2.5-5s hard-ceiling
poll loop with the PR's _wait_until(predicate) helper — generous 30s
ceiling reached only on genuine failure, instant return on the green
path, assert with message instead of silent fall-through.
Dropped: the 3 hunks targeting pruned code, and the unrelated
scripts/release.py mailmap hunk (AUTHOR_MAP is frozen; contributor
mapping handled via contributors/emails/).
Companion to the cherry-picked #74158 fix (non-streaming path):
- gateway/run.py: remove the identical history-dedup filter from
_deliver_media_from_response — the post-stream rescan is explicit-only
by design (#20834), so every MEDIA tag it finds is a deliberate
attachment request; drop the now-unused history_media_paths parameter
and its call-site plumbing.
- gateway/platforms/base.py: log suppressed bare local file paths on the
surviving local_files history dedup (#73771 observability ask).
- tests: focused regression file covering explicit resend delivery on
both lanes, current-turn tool-echo poisoning, surviving bare-path
dedup + its log line, and the upstream auto-append dedup invariant.
Fixes#73771
Two relay-lane bugs from live Discord staging testing (2026-07-29):
1. TTS audio never attached over relay (any platform, any lane).
_history_media_paths_for_session excluded only the trailing assistant
entry from the persisted transcript when building the delivered-media
dedup set. The agent persists rows as it produces them, so THIS turn's
text_to_speech tool result (media_tag JSON) was already in the
transcript at delivery time — the fresh TTS path deduped against
itself and extract_media's attachment was silently stripped
(response_delivery_dropped for a MEDIA-tag-only reply; fly logs show
the exact signature). Fix: exclude everything from the last USER
message onward (the current turn); prior-turn dedup unchanged.
Affects every platform adapter (native + relay) on the non-streaming
delivery path — the streaming path passes explicit history and was
unaffected.
2. Connector-auto-created threads never got the LLM session-title
rename. The title fires on the FIRST exchange, whose source is the
PARENT channel event — the thread didn't exist at ingest, so the
Phase 4 auto-thread markers can't be present and
_is_discord_auto_thread_lane never matches on the relay title turn
(initial titles worked; semantic renames never happened; staging
telemetry shows zero thread_rename ops ever sent). Fix: consume the
connector's new send-result feedback (paired gateway-gateway PR —
contract §SendResult thread_id/auto_thread_name, additive):
RelayAdapter.send() caches (thread_id, initial_name) per chat
(bounded 256), run.py's title-callback registration + rename lane
read it back and pass initial_name as only_if_current_name so the
human-rename-wins guard holds on the relay lane too. Native marker
path unchanged; connectors that don't stamp the fields degrade to
exactly the old behavior.
Tests: 3 new (send-result feedback capture, absence, bound) in
test_relay_threads.py; 3 new in test_history_media_current_turn.py
(current-turn TTS not deduped, prior-turn still deduped, no-user-row
fallback). Relay suite 144 passed.
Telegram clients split messages above 4096 chars into multiple updates. A
long '/queue <prompt>' paste arrives as a COMMAND chunk near the limit plus
plain TEXT continuation chunk(s). _handle_command dispatched the command
chunk immediately, so the continuation landed as a separate plain message
that interrupted the running agent instead of being queued.
Near-limit (>= _SPLIT_THRESHOLD) command chunks now route through the same
text-batching pipeline used for split plain-text messages, merging the
continuation before dispatch. Short commands (/stop, /approve, ...) keep the
immediate path and are never delayed.
The gateway keeps one PairingStore per served profile, but every
`/api/pairing` endpoint built the global one. An operator managing a named
profile saw the wrong pending list, and approving wrote a grant into a
whitelist their running gateway never consults — the user stays locked out
while the UI shows them as approved.
`_pairing_store(profile)` now resolves per profile and validates the name
(400/404 on an unknown one). No `_profile_scope` needed: PairingStore
resolves the profile's home itself, so nothing process-global is swapped
across an await.
Both GUIs had to change to match. The listing rides the query param — for
the dashboard that meant deleting `pairing` from the "machine-global, must
NOT be rewritten" exclusion list, a comment this change makes false. The
mutating endpoints read the profile off the BODY, which no query-param
rewrite reaches, so approve/revoke send it explicitly on both surfaces.
The per-profile pairing isolation added self._dir and scoped every
per-file path helper (_pending_path, _approved_path) to it, but
_all_platforms still enumerated the module-global PAIRING_DIR. For a
profile-scoped PairingStore, list_approved/list_pending/clear_pending
therefore operated on the GLOBAL platform set while loading each
platform's file from the PROFILE dir — so list_approved() returned []
for a user that is_approved() confirmed as approved, a silent divergence
between the authz surface and the list/inspect/clear surface.
Route discovery through self._dir. Byte-identical for the global store
(self._dir == PAIRING_DIR when no profile is set); only the buggy
profile-scoped case changes. self._dir is guaranteed to exist (__init__
mkdirs it).
Follow-up hardening on the request-id grant path.
approve_request took the same lockout treatment as approve_code: gated by
it, and recording a miss toward it. But the two paths defend different
things. The lockout exists to stop guessing at the 8-char code space over a
messaging channel; a request id is only ever obtained by an admin already
authenticated to the store, so a miss means the row they clicked went stale.
Counting those let a handful of clicks on a stale list lock the operator out
of `hermes pairing approve` for an hour — the GUI DoSing the CLI.
Also drops the `code`/`code_hash_prefix` compat fields from list_pending.
The hash prefix is what admin surfaces mistook for an approvable code in the
first place, and re-exporting the request id under the old `code` key just
preserves the ambiguity; both consumers in the tree read `request_id` now.
The 16-hex sniffing that had been copy-pasted into the CLI and the endpoint
(where a chained conditional consulted it against the wrong field) moves to
one owner, PairingStore.looks_like_request_id.
The endpoint no longer reports a 429 on the request-id path, where lockout
can't apply — a stale id surfaced as a bogus "locked out" while the platform
sat locked for something else entirely.
approve_code()'s success path never cleared _failures:{platform}. The
counter is incremented on every non-matching code, persisted in
_rate_limits.json, and only ever reset to 0 when it reaches
MAX_FAILED_ATTEMPTS (firing the lockout). So it counts failures over the
gateway's entire lifetime, not consecutive ones.
An owner who mistypes a pairing code on a handful of separate occasions
— each time immediately retyping it correctly and successfully pairing —
accumulates those isolated typos. A later single fresh typo then hits
MAX_FAILED_ATTEMPTS and locks the whole platform out for an hour, at
which point _is_locked_out gates approve_code and even the *correct*
code is rejected.
Reset the counter on a successful approval, matching standard
brute-force-guard semantics (the counter tracks consecutive failures).
This does not weaken protection: an attacker cannot produce a success
without a valid code, and 5 consecutive wrong attempts still lock out.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
The earlier scrub covered adapter.py; nine internal QA-campaign tracker IDs
remained in the two new test files, including a module docstring and an
assertion message. They mean nothing to a future reader — describe the
behavior instead. Comments only, no assertion changes.
Both relay Slack knobs read their value through bool(), while the native
adapter they mirror uses str(raw).strip().lower() in {"1","true","yes","on"}.
A YAML-quoted string diverges:
dm_top_level_threads_as_sessions: "false" → relay True, native False
Non-empty strings are truthy, so the escape hatch is silently ignored in
exactly the shape an operator writes to switch it OFF. reply_in_thread has the
same defect and gates reply placement, session keying and run.py's progress
resolver, so one quoted "false" misfires three ways.
Route both through a shared _coerce_flag mirroring native's predicate. Real
booleans pass through untouched; None falls back to the default. Contract §8
documents the accepted spellings.
Tests: both knobs parametrized over the true/false spellings native accepts,
plus the absent-key default.
The DM thread-anchor contract was resolved only in send(). _send_media() —
backing send_image, send_image_file, send_voice, send_video and send_document
— passed reply_to straight to the frame and never touched metadata, so
attachments egressing through the same connector-side Slack sender got both
failure shapes this branch set out to remove:
flat mode → reply_to survives, the image threads UNDER the user's DM
message (the original reported symptom)
thread mode → no metadata.thread_id, and threadTs() never reads reply_to,
so the image lands in the home channel instead of the
per-message thread
Both are reachable: gateway/run.py delivers agent artifacts through
send_voice/send_document.
Extract the three steps that must always happen together (mode gate, mirrored
reply_to_message_id strip, metadata promotion) into
_apply_slack_thread_anchor and route BOTH lanes through it, so text and media
cannot drift again. The media lane copies caller metadata rather than mutating
it — these helpers are called in loops with a shared mapping.
Also fold send_typing/stop_typing's duplicated status-anchor blocks into
_with_status_thread_anchor. They had already drifted (stop_typing omitted the
platform check) and the clear must target the thread the heartbeat set or the
status line sticks until Slack's own timeout.
Tests: media lane pinned in both modes plus the channel and
no-caller-mutation cases; verified as real by reverting the fix and watching
them fail.
The gateway's /reset cleanup path called shutdown_memory_provider without
first draining the memory manager's serialized background write worker.
shutdown_all only gives that worker a bounded (~5s) drain and abandons
whatever is still queued past it, so a /reset could silently drop writes
the session had already handed off -- the next session then loaded
stale MEMORY.md.
The CLI exit path already drains via MemoryManager.flush_pending before
shutdown; this PR pins the same contract on the gateway cleanup path.
Cleanup now calls agent._memory_manager.flush_pending(timeout=10) before
the existing shutdown_memory_provider step. The flush is best-effort:
a flush failure must never block teardown, so it is wrapped in
try/except and the existing shutdown path remains the fallback.
Closes#73297
Follow-up for salvaged PRs #73400 + #73372:
- Move _preserve_agent_history_on_shutdown logic into
gateway/shutdown_flush.py as flush_agent_history_to_file()
- Reuse the hardened _write_payload (atomic writes, fsync, private
permissions, UUID filenames) from #73372 instead of the plain
open()/write() from #73400
- Replace os.environ.get('HERMES_HOME') with get_hermes_home() for
profile-safe path resolution
- Update tests to test the real function directly
- Net: removes 37 lines from gateway/run.py, unifies both fix paths
in a single module with consistent atomic-write guarantees
The previous attempt (#73171) snapshotted GatewayRunner._pending_messages,
which on current main has no writers (commit f6736ced8 removed its write
path; interrupt delivery uses adapter._pending_messages instead). The live
container is the per-agent agent._session_messages, flushed via
_flush_messages_to_session_db. When that flush raises (FTS/SQLite corruption,
the disk=0/memory=N state from #72680), the in-memory transcript is lost when
the process exits.
Retarget the preservation to the real path: in _finalize_shutdown_agents, wrap
the _flush call; on exception, dump agent._session_messages to an external JSON
recovery snapshot under $HERMES_HOME/shutdown-recovery/ (tagged issue=#72680)
so an operator can salvage it after repairing state.db. The dump is fully
guarded (non-fatal) so shutdown never blocks on a best-effort backup.
This directly addresses the reviewer note on #73171: retarget to the actual
cached-agent history (agent._session_messages) and prove a stale DB + shutdown
leaves a recoverable transcript.
Regression tests: tests/gateway/test_session_messages_shutdown_preserve.py
- flush raises -> recovery file written with session_id + messages
- healthy flush -> no recovery file
- write error -> non-fatal, no raise
Fixes#72680
Replace 20 lines of manual os.open/O_EXCL/fdopen/fsync/os.replace with the
existing atomic_json_write() from utils.py, which is already used by 6+
modules and handles temp-file creation, fsync, atomic replace, mode
control, and owner preservation. The only novel helper (_fsync_directory)
is retained — atomic_json_write does not do directory fsync.
Update test_flush_write_failure_leaves_no_recovery_file to monkeypatch
utils.os.replace (the new call path) instead of gateway.shutdown_flush.os.replace.
test_default_bind_serves_both_families asserts an IPv6 listening socket
exists, which is the environment's capability, not the adapter's — CI
runners with IPv6 disabled would flake. Probe an actual ::1 bind (not
just socket.has_ipv6, a compile-time constant) and skip cleanly when
the host has no usable IPv6 stack.
The LINE adapter's webhook server defaulted to host="0.0.0.0", which
binds IPv4 ONLY. On IPv6-only private networks — notably Fly.io 6PN,
where the hosted edge router reverse-proxies LINE ingest to
<app>.internal:8646 over an fdaa: IPv6 address — nothing is listening
on the dialed address: connection refused → customer-visible 502 when
LINE's console verifies the webhook (NS-603).
This is the same bug the generic webhook adapter fixed in d542894ad;
the LINE adapter was never updated to match. Fix mirrors that commit:
- DEFAULT_HOST = None → asyncio/aiohttp create_server binds one socket
per address family (v4 + v6), regardless of the bindv6only sysctl.
"::" is NOT a valid substitute — Fly machines set bindv6only=1, so
it would yield an IPv6-only socket and break IPv4 loopback probes.
- Empty-string host collapses to None; LINE_HOST/extra.host still pin
a specific bind address.
- reuse_address=False scoped to macOS only (BSD wildcard-socket
traffic-splitting footgun), mirroring 9420ad946.
- The three outbound-media guards compared webhook_host == "0.0.0.0"
by string equality; extracted to _missing_public_url() which treats
None/0.0.0.0/::/"" as "no fetchable hostname" so the LINE_PUBLIC_URL
requirement still fires under the new default. _media_url() falls
back to 127.0.0.1 instead of interpolating 'None' into URLs.
Tests: dual-stack default decision table (default None, empty→None,
pinned preserved, LINE_HOST override), behavioural both-families bind
proof via runner.addresses, and the media public-URL guard matrix.
88 passed, ruff clean.
Companion router fix (hermes-agent-router) routes /webhooks/line and
/line/webhook|/line/media to :8646 over 6PN; both are needed for
end-to-end hosted LINE delivery.
Fixes NS-603
The ownership guard let a sentinel with pid=None pass as self-owned, so
an exiting life could clobber evidence of unknown provenance with a
clean-exit claim. Tighten: only rewrite when the sentinel pid matches
os.getpid() exactly; pid=None (or malformed) is left untouched. Adds
tests for the pid=None no-op and the own-pid rewrite paths.
Hosted agents that die uncleanly (kernel OOM kill, SIGKILL, whole-VM
death) leave no trace: shutdown_forensics only covers graceful signals,
gateway-exit-diag.log only covers exit paths that actually run, and the
VM reboot wipes dmesg before anyone can capture it. NS-608 (BlueAtlas
hourly crash cycle, July 12-15) took days of manual log correlation to
classify because nothing recorded 'the previous life ended violently'.
Add gateway/lifecycle_ledger.py — a sentinel state machine persisted to
<HERMES_HOME>/state/gateway.lifecycle.json:
- start_gateway() claims the sentinel (phase=running) right after the
PID-file/runtime-lock claim, and reports any prior life that never
reached an exit path as gateway.previous_unclean_exit in
gateway-exit-diag.log + a WARNING log line.
- Every exit funnel marks the sentinel exited with a reason:
_exit_after_graceful_shutdown (graceful_shutdown), the shutdown
watchdog (shutdown_watchdog), and the loop-liveness watchdog
(loop_liveness_watchdog).
- Ownership-guarded for --replace takeovers: a live matching owner is
never reported dead, and the old life cannot clobber the
replacement's freshly claimed sentinel on its way out.
The 30s loop heartbeat now embeds a cheap /proc memory sample (own RSS,
MemAvailable, swap used) so every unclean-death report carries a
'memory N seconds before death' snapshot; the detector flags
suspected_oom when the last sample shows <64MiB or <5% available.
container-boot.log lines gain prior_exit=clean|unclean|unknown per
profile, stamping unclean container deaths into the volume-persisted
boot log where support can grep for them.
Tests: tests/gateway/test_lifecycle_ledger.py (16 cases) + 4 new
container-boot annotation cases. Existing watchdog/forensics/boot
suites all green; ruff clean.
