Addresses @egilewski's review: the parent-directory check still deleted
checkpoint history for the most common unmount layout.
Detaching storage removes the parent outright in some layouts
(`/Volumes/Ext/proj` on macOS, `/media/<user>/<label>/proj`), which the first
commit handles. But in the classic static layout — `/mnt/volume/proj`, an
fstab entry, a container bind-mount — unmounting removes the contents and
leaves the mount point behind as an empty directory. `parent.is_dir()` is then
true, the project is absent, and the startup sweep deletes its ref, index and
metadata: exactly the case this PR set out to protect.
Reproduced against the real predicate before this commit:
mount root vanished (macOS) -> False ok
empty surviving mount point -> True <-- history deleted
really deleted (siblings) -> True ok
An empty parent carries no information: it looks identical whether the volume
was detached or the project was deleted. So require the parent to actually say
something — it holds some other entry (we observed a populated directory that
does not contain the project), or it is itself a live mount point (the volume
is attached right now and demonstrably does not hold the project).
The cost is that a project deleted out of an otherwise-empty parent is no
longer reclaimed by the orphan rule. It is not leaked: the retention rule
reads `last_touch` rather than probing the filesystem and still collects it,
so reclamation is deferred, not lost. That is the right direction for a
predicate whose false positive destroys a user's restore points unattended.
`_dir_has_any_entry` stops at the first entry via `os.scandir` instead of
materializing a listing, since a project root can hold a large tree.
tests/tools/test_checkpoint_manager.py: `test_surviving_empty_mountpoint_
keeps_its_checkpoints` pins the reviewed case, and `test_empty_parent_project_
is_still_reclaimed_by_retention` pins the deferral above so the safety valve
cannot silently regress into a leak. Both fail on the previous commit. The
real-orphan control now seeds a sibling so it exercises a populated parent
rather than the ambiguous empty one. 80 passed in the checkpoint suite; the 2
remaining failures (`TestGitEnvIsolation`, `TestClearFunctions`) fail
identically on clean main.
Orphan pruning decides a project is gone from a single probe:
if delete_orphans and (not workdir or not Path(workdir).exists()):
reason = "orphan"
then deletes its ref, index, and metadata — the project's entire checkpoint
history. `Path.exists()` is False for a deleted directory, but it is equally
False for one whose storage is not attached right now: an unplugged external
drive, a share behind a downed VPN, a bind-mount absent from this container,
an offline Windows mapped drive. The project is fine; only our view of it is.
This is not an opt-in maintenance command. `maybe_auto_prune_checkpoints`
runs unattended at startup from both `cli.py` and `gateway/run.py`, with
`delete_orphans=True` by default. So starting Hermes once while the drive is
unplugged silently destroys the restore points for every project on it — the
one thing checkpoints exist to provide, and there is nothing to restore from
afterwards.
Reproduced against the real store: a project registered under an unmounted
path and one on local disk, then a startup prune —
prune: {'scanned': 2, 'deleted_orphan': 1}
unreachable project index still on disk: False
The legacy pre-v2 branch has the same flaw plus a second one: a
`HERMES_WORKDIR` marker that exists but cannot be read leaves `workdir = None`,
which the same condition treats as an orphan. Failing to read a file is not
evidence that a project was deleted.
Require corroboration before deleting: the workdir's parent must be present,
so its absence is something we actually observed. A missing parent means the
volume is not there and we know nothing, so the entry is left alone — and an
unreadable marker never deletes at all. Genuinely abandoned projects are still
reclaimed, both by the unchanged orphan path (parent present, project gone)
and by the retention/stale rule, which runs off `last_touch` rather than a
filesystem probe.
tests/tools/test_checkpoint_manager.py: a project whose whole mount disappears
keeps its history; controls prove a genuinely deleted project is still pruned
and a live project is untouched. The data-loss test fails on main; both
controls pass there. 81 passed across the checkpoint suites (2 failures in
test_checkpoint_manager.py are pre-existing and fail identically on clean
main).
The desktop and the Hermes backend it talks to version independently — a
remote VM running an older build still serves display_metadata as JSON text.
Indexing into that string with `in` threw and failed the whole resume, so
narrow the type to admit a string and parse it before reading task_count.
Falling back to the generic label keeps a delegation event renderable even
when the metadata is unusable.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: Studio729 <Studio729@users.noreply.github.com>
export_session() reads through get_messages(), so before the read fix an
already-serialized string went straight back into _insert_message_rows() and
got re-dumped — an export/import round trip permanently corrupted the row.
Guard the three write paths the same way tool_calls already is: parse a
string argument before storing it, and drop metadata that isn't an object
rather than persisting something no reader can use.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: aml1973 <aml1973@users.noreply.github.com>
get_messages(), get_messages_around() and get_anchored_view() returned the
raw display_metadata column instead of the dict every caller expects. The
desktop paints a resumed transcript from the REST prefetch, which reads
through get_messages(), so any session holding an async_delegation_complete
event failed resume with "Cannot use 'in' operator to search for
'task_count'" — on every such session, not just corrupted ones.
Route all four read paths through one shared codec that also unwraps rows
carrying a second JSON layer, so sessions already broken on disk recover on
read rather than needing a migration.
Co-authored-by: Studio729 <Studio729@users.noreply.github.com>
Co-authored-by: aml1973 <aml1973@users.noreply.github.com>
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Desktop boot polls /api/status, so readiness waits on gateway config and a
cold plugin import tree. On Windows that regularly outlives the probe and
Desktop kills a backend that is already listening, respawns it, and re-pays
the same import cost — the reported crash loop.
Probe /api/health instead, falling back to /api/status only for the
missing-route shapes the fetch helpers emit (404, or HTML from the SPA), so
an older remote backend still connects. Timeouts and server errors keep
polling health rather than dropping to the heavyweight route.
A cheap route is not enough on its own. Warming the gateway import holds the
GIL, so the event loop can stall for tens of seconds and starve /api/health
too. At the default 15s socket timeout only three attempts fit in the 45s
budget; give each probe 5s so the loop keeps retrying across the stall.
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: DESXIE <78300229+DESXIE@users.noreply.github.com>
Co-authored-by: frohsinnllc <231045016+frohsinnllc@users.noreply.github.com>
/api/status is the only public liveness route, and its handler loads the
gateway config, probes gateway health, and counts sessions before it can
answer. That work is wrong for a readiness probe: a caller that only needs
to know the process is up pays for a cold plugin import tree.
Add /api/health, which returns process liveness, version, and the auth-gate
shape and touches nothing else.
The create dialog used the settings-surface ListRow/ToggleRow inside a
modal, which read differently from every other form dialog, and the detail
header carried an enabled/disabled pill that rendered as a stray dash.
Switch the form to the shared Field primitive (+ Switch) and remove the
pill.
Blueprints lived behind a separate Jobs/Blueprints tab with its own card
gallery — a bespoke surface no other overlay uses. Remove the tab and make
blueprints a "Start from" dropdown at the top of the New Job dialog
(default "Custom" = the manual editor); picking one swaps the form for that
blueprint's typed slots. Also promote the detail-view "Trigger now" button
to a primary action and adopt the shared Field primitive.
Overlay panes each set their own top padding, so the Settings sidebar and
Panel headers sat at different heights than System/Agents and the close X
(the #67759 regression). Hoist the shared beside-the-X clearance into
OVERLAY_TOP_CLEARANCE, keep the taller pad only on OverlayMain (which sits
under the X), tighten OverlayMain's gutters, and drop the one-off Settings
override. Also give PanelAction a `primary` variant so a detail header can
promote its main action to a filled button.
Dialog forms each hand-rolled their own label+control+hint stack (or
borrowed the settings-surface ListRow), so gaps and hint styling drifted
between the profile, cron, and webhook dialogs. Add a single Field /
FieldHint primitive for label-over-control dialog fields and adopt it in
the create/rename profile dialogs as the first consumers.
The cherry-pick auto-resolution + AST sweep applied plain utf-8 at three
.env reader sites where the salvaged PRs (#62617, #62123) deliberately
use utf-8-sig — a Notepad BOM must not hide/duplicate the first key.
Restore the contract (tests pin it).
AST-driven pass over every Path.read_text()/write_text() without an
explicit encoding= across non-test code: 71 sites in 34 files
(skills_hub, hermes_cli/main+profiles+service_manager+container_boot,
mem0/hindsight/honcho plugins, achievements dashboard, release/CI
scripts, productivity+comfyui skill helpers, agent/*). Verified zero
positional-encoding collisions before insertion; per-file compile()
check after.
Adds a check-windows-footguns rule flagging bare single-line
read_text/write_text (multi-line forms stay covered by the AST guard
test from #38985). Together with the salvaged contributor commits this
retires the ~169-site bare file-I/O class (#37423's long tail).
The bundled office skills (#68595) read user documents and agent-authored
payloads with the locale-default codec:
- docx/powerpoint validators/base.py opened OOXML part XML in text mode
before handing it to lxml. On Windows (cp1251/GBK) the bytes decode to
mojibake that lxml then parses, so validation runs against silently
corrupted document text; on locales where the UTF-8 bytes don't decode
the validator crashes with UnicodeDecodeError instead of validating.
Opening as bytes lets lxml honor the encoding declared in the XML prolog.
- The pdf form scripts (fill_fillable_fields, fill_pdf_form_with_annotations,
create_validation_image, check_bounding_boxes) read the fields JSON the
agent authors — UTF-8 by construction — with the locale codec, so
non-ASCII form values (any Cyrillic/CJK/accented input) either crash or
get written into the user's PDF as mojibake. The json.dump writers use
ensure_ascii=True and were already safe; only the readers needed pinning.
Adds a contract test asserting every document/payload reader is
locale-independent, plus a live regression test that runs
check_bounding_boxes.py on a non-ASCII fields.json under a forced
non-UTF-8 locale — it fails without the fix on both POSIX (C locale)
and Windows (cp1251 chokes on the 0x98 byte of U+2018).
_setup_worktree read both files with the locale default encoding. On a
cp1251/GBK Windows machine a UTF-8 include list either decodes to
mojibake paths (non-ASCII entries silently not copied) or raises
UnicodeDecodeError, which the enclosing handler logs at DEBUG and
swallows — no include is copied at all, so the worktree starts without
.env/keys and the agent breaks invisibly. A Notepad BOM likewise glues
to the first include entry on every platform, and to the first
.gitignore line, defeating the '.worktrees/' membership check and
appending a duplicate entry on each run.
Read both files with utf-8-sig + errors=replace, matching the canonical
.env readers in hermes_cli/config.py (utf-8-sig because Notepad adds a
BOM) and the UTF-8 append this same block already performs on
.gitignore.
Regression tests exercise the real cli._setup_worktree: the two BOM
tests fail without the fix on any platform, the non-ASCII include test
additionally reproduces the Windows locale failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_read_hub_installed_names() reads ~/.hermes/skills/.hub/lock.json with a
strict utf-8 decode. Hub skill descriptions can carry Windows-1252
typographic bytes (em-dash 0x97, smart quotes, bullets) as single high
bytes; read_text(encoding="utf-8") then raises UnicodeDecodeError, which
is a ValueError sibling not caught by the function's
except (OSError, json.JSONDecodeError). It escapes and 500s the whole
/api/skills endpoint, blanking the desktop Skills panel.
Decode with errors="replace" so the offending byte degrades to U+FFFD
and the structurally valid JSON — and every other skill — stays readable.
Fixes#68053
`_render_distribution_plan` reads the target profile's `.env` to decide
whether a required env var is already set (so it doesn't nag the user),
using `Path.read_text()` with no encoding. Two bugs:
1. `Path.read_text()` defaults to the system locale (cp1251/GBK on Windows),
which raises `UnicodeDecodeError` on any non-ASCII byte. The surrounding
`except OSError` does NOT catch that — `UnicodeDecodeError` is a
`ValueError` — so a mis-encoded `.env` aborts the entire install preview.
2. Even on a UTF-8 locale, a Notepad-added BOM prefixes the first key
(`KEY`), so the very first required env var is mis-reported as
"needs setting" when it is actually present.
`.env` is written as UTF-8 everywhere in the codebase. Read it as
`utf-8-sig` (tolerates the BOM) and also catch `UnicodeDecodeError` so a
genuinely un-decodable file skips the pre-check instead of crashing.
Regression tests: a BOM-prefixed `.env` whose first key must still read as
"set", and an invalid-UTF-8 `.env` that must not abort the preview.
Follow-up to review feedback:
- mem0 _prompt_api_key read .env with the locale default, so a Notepad
BOM hid the first key from the masked current-value lookup; read it
with utf-8-sig + errors=replace like the canonical readers in
hermes_cli/config.py.
- hindsight _load_simple_env used plain utf-8; it also parses the Hermes
.env during post_setup, where a BOM stuck to the first key. Switch to
utf-8-sig + errors=replace.
- Add hindsight regressions: BOM key matching in _load_simple_env and in
the cloud post_setup writer, plus non-ASCII round-trip preservation,
and a mem0 regression for the BOM'd masked-key lookup. The BOM tests
fail without the fix on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mem0 and hindsight memory-provider setup routines round-trip the
user's ~/.hermes/.env: they read existing lines, update the keys they
manage, and rewrite the whole file preserving every other line verbatim.
Both used env_path.read_text() / write_text() with no encoding.
read_text()/write_text() with no encoding fall back to the system locale
(cp1252/GBK on Windows), so on a non-UTF-8 host the preserved lines get
mangled or the call crashes on any non-ASCII value, and — because the
reader never strips a BOM — a Notepad-edited .env makes the first key
fail the in-place match and get duplicated instead of updated.
Match the canonical .env readers in hermes_cli/config.py: read with
encoding='utf-8-sig' (BOM-tolerant) and write with encoding='utf-8'.
mem0/_setup.py already pins utf-8 for mem0.json, so this just aligns the
.env path in the same file. Fixes both memory plugins in one class fix.
Adds regression tests: a BOM'd .env updates the first key in place
(locale-independent, fails without the fix) and non-ASCII existing lines
survive the round-trip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On Windows with CJK locales (e.g. Chinese/GBK), pathlib.Path.read_text()
defaults to the system encoding instead of UTF-8, causing UnicodeDecodeError
when reading .env or .json config files that contain non-ASCII characters.
Explicitly pass encoding='utf-8' to all read_text() and write_text() calls
in the hindsight memory provider plugin.
Addresses hermes-sweeper review on PR #54866: the installer runs
tools/skills_sync.py as a child python.exe whose PYTHONIOENCODING /
PYTHONUTF8 the scoped install.ps1 block sets, but there was no
regression test for this child-Python UTF-8 path. The existing
test_child_process_inherits_utf8_mode covers a different (bootstrap
entry-point) flow.
Add TestSkillsSyncUtf8Guard: three subprocess tests that import
skills_sync (triggering its import-time stdout/stderr reconfigure)
and assert the checkmark/up-arrow glyphs the script prints at
tools/skills_sync.py:596,675 emit valid UTF-8 and exit 0 even when
the child env is left unset or explicitly hostile (gbk). A third
test proves the guard is load-bearing by reproducing the crash
without it.
Also keep the new install.ps1 comment ASCII-only (the checkmark
spelled out as U+2713) per the file's PS 5.1 parser-compatibility
contract at scripts/install.ps1:79-80; the literal glyph in the
comment violated that contract.
On Windows with a non-UTF-8 system locale (e.g. CP936/GBK on zh-CN),
Python defaults stdout/stderr to the active codepage. tools/skills_sync.py
prints glyphs such as checkmark (U+2713) and up-arrow (U+2191) that GBK
cannot encode, raising UnicodeEncodeError mid-run.
The installer (scripts/install.ps1) captures this script's stdout and the
Rust bootstrap parses it as UTF-8 expecting a JSON result frame. A GBK
byte stream (or the traceback it triggers) surfaces as:
WARN stdout read error: stream did not contain valid UTF-8
stage=config-templates state=Failed
error=install.ps1 -Stage config-templates produced no JSON result frame
(exit=Some(0))
i.e. the stage fails even though the script exits 0. install.ps1 already
sets [Console]::OutputEncoding = UTF8, but that does not propagate to the
python.exe child (Python reads PYTHONIOENCODING / locale, not the console
encoding).
Fix in two places for defense in depth:
- tools/skills_sync.py: reconfigure sys.stdout/stderr to UTF-8 at import so
output is valid UTF-8 regardless of caller or active codepage.
- scripts/install.ps1: set PYTHONIOENCODING=utf-8 and PYTHONUTF8=1 (scoped
to the call, restored afterwards) around the skills_sync.py invocation.
Path.read_text() without an explicit encoding uses the platform's
default encoding. On Windows this is typically cp1252 or mbcs, which
causes UnicodeDecodeError or silent data corruption when reading
UTF-8 content (JSON files, user text, config with non-ASCII chars).
This is the read-side companion to the write_text() encoding fix.
Fixed the most critical locations that read JSON data, user content,
and config files across 14 files with 31 call sites.
Pattern: .read_text() → .read_text(encoding='utf-8')
json.loads(path.read_text()) → json.loads(path.read_text(encoding='utf-8'))
Path.read_text() and Path.write_text() without explicit encoding
default to the system locale encoding. On Windows this is typically
cp1252, which causes UnicodeDecodeError for UTF-8 content (JSON
configs, user data, service scripts).
Add encoding="utf-8" to all read_text() and write_text() calls
across 8 CLI files, matching the pattern established in PR #50534
(security_audit_startup.py) and ruff rule PLW1514.
Fixed files:
- main.py: 4 read_text calls
- auth.py: 3 read_text calls
- banner.py: 1 read_text + 1 write_text
- service_manager.py: 1 read_text + 4 write_text
- container_boot.py: 1 read_text + 4 write_text
- doctor.py: 3 read_text calls
- uninstall.py: 2 read_text calls
- gateway.py: 1 write_text call
Several file-I/O call sites still use open() / Path.read_text() /
Path.write_text() without an explicit encoding, so they fall back to
the platform default. On Windows CN/JP/KR locales (GBK/CP932/CP949)
any non-ASCII byte in a config/state/user-content file raises
UnicodeDecodeError or UnicodeEncodeError and crashes the caller.
to the remaining hot paths:
- agent/copilot_acp_client.py: fs/read_text_file and fs/write_text_file
(Copilot's read_file / write_file tools,
directly reported in #18637 bug 2)
- agent/model_metadata.py: context-length YAML cache load + two
save sites (context probing is on the
call path of every model invocation)
- agent/nous_rate_guard.py: cross-session rate-limit JSON state
(read + atomic write via os.fdopen)
- cron/scheduler.py: user config.yaml read in run_job
- gateway/delivery.py: cron output writes for AI-generated
content, very likely non-ASCII
yaml.dump call sites also gain allow_unicode=True so the emitted
YAML preserves non-ASCII chars as-is instead of emitting \u escape
sequences.
Adds regression tests that monkeypatch builtins.open / Path.read_text
/ Path.write_text to simulate a GBK locale: each test raises
UnicodeDecodeError / UnicodeEncodeError unless the caller explicitly
passes encoding='utf-8'. Verified that the tests fail on main and
pass with this change, on Linux as well as on Windows.
Refs #18637
The three refresh tests asserted the replaced shared client gets
close()d — the exact cross-thread close#70773 removes. They now pin
the new contract: close() is NOT called from the refresh path; the
old client is retired (sockets shutdown, FD release deferred to GC).
Widen the #70773 fix beyond the three in-request cleanup sites removed in
the cherry-picked commit: every remaining path that swaps out the shared
OpenAI client could still hard-close its pool from a thread that doesn't
own the in-flight sockets (credential rotation/refresh on the turn thread,
dead-connection cleanup, gateway cache eviction, transport recovery) —
the same FD-recycle corruption vector, just rarer.
Add AIAgent._retire_shared_openai_client(): shutdown(SHUT_RDWR) all pooled
sockets (FD-safe from any thread, unblocks in-flight readers) but never
call client.close() — FD release is deferred to GC, which cannot run until
every borrowing thread has unwound its SSL BIO. Refcounting is the
ownership handshake; with no borrowers the FDs are released immediately.
Wired into:
- _replace_primary_openai_client (rotation/refresh/dead-conn cleanup)
- try_recover_primary_transport (primary_recovery)
- release_clients (gateway cache_evict)
agent.close() keeps the hard close: full teardown is a real session
boundary where no request may be in flight.
Tests: new tests/run_agent/test_70773_shared_client_fd_corruption.py
covers the three watchdog/retry sites plus retire semantics; existing
close-assertions updated to pin retire-not-close.
The streaming stale watchdog was calling
_replace_primary_openai_client() from its polling thread, which closes
the shared client's connection pool. Worker threads from previous
stale-killed attempts may still be unwinding their SSL BIOs, causing
TLS application-data to overwrite SQLite file headers via FD reuse.
This is the same corruption vector documented in #67142 for Anthropic,
where the fix was to never close the shared client from a non-owner
thread. Apply the same pattern to the OpenAI-wire path:
- Stale stream watchdog: skip shared client replacement
- Mid-tool-retry cleanup: skip shared client replacement
- Stream retry cleanup: skip shared client replacement
The request-local client is already closed via _close_request_client_once.
The shared client is replaced lazily by _ensure_primary_openai_client
on the next request, which runs on the owning thread.
Closes#70773.
CI's check:lint failed on two perfectionist rule violations introduced by the
new test file: type import ordering and missing blank line between the
parent-relative and same-directory import groups. No behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two React passive-effect timing bugs let a session switch land in the wrong
chat: activeSessionIdRef/selectedStoredSessionIdRef (use-session-state-cache)
and the composer's attachment-scope swap (use-composer-draft) both mirrored
their source props via useEffect, which fires one commit AFTER the new
session's view has already painted — a synchronous read/submit in that window
observed the outgoing session's ids/attachments.
- use-session-state-cache.ts: mirror the session refs synchronously during
render instead of a useEffect, guarded to fire only when the prop itself
changed (not unconditionally) so an imperative pin from submit.ts /
use-session-actions (e.g. a freshly resumed runtime id, intentionally not
synced to the source atom) survives an unrelated re-render.
- use-composer-draft.ts: the per-thread attachment-scope-swap effect is now a
useLayoutEffect, closing the window before paint.
- submit.ts / session-context-drift.ts: add a 3rd drift prong comparing the
composer's loaded scope (SubmitTextOptions.composerScope) against the
submit target, resolved into the same lineage-root domain
(resolveComposerSessionKey) the composer itself uses — comparing against
the raw tip id would false-positive-abort every submit into any session
that has ever auto-compressed.
- routes.ts / chat/index.tsx: the primary composer's durable scope key now
prefers the route over a possibly-stale store selection
(primaryRouteSelectedSessionId).
- use-composer-draft.ts: redacted [composer-rehydrate] diagnostic log
(counts/kinds/scope only, never raw refs) for future reports in this class.
- chat-runtime.ts: normalize attachment id values (url/path) before hashing
so a re-attach with a trailing slash or backslash path dedupes correctly.
16 files, 286 tests across the touched/dependent suites (17 files) green,
including new regression coverage for each fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Teardown-path tests build bare runners via object.__new__ without
the liveness-guard machinery; the unguarded call raised
AttributeError in 8 tests. Same guard pattern as the start path.
Follow-up to the salvaged #69164 commits: policy forbids introducing new
HERMES_* environment variables, so the four watchdog env knobs
(HERMES_GATEWAY_LOOP_WATCHDOG / _INTERVAL / _TIMEOUT / _STRIKES) are
replaced with a single config.yaml boolean:
gateway:
loop_watchdog: true # default; false disables both guards
- gateway/config.py: new GatewayConfig.loop_watchdog field (default True),
parsed from top-level or nested gateway: form, round-trips via
to_dict/from_dict.
- gateway/run.py: _start_loop_liveness_guards() checks config.loop_watchdog
before arming the floor timer + watchdog (getattr-guarded for bare
object.__new__ runners).
- gateway/shutdown_watchdog.py: start_loop_liveness_watchdog() no longer
reads the environment; probe interval/timeout/strikes are module
constants (30s/10s/3 — ~90s to restart, matching the systemd watchdog
layer's posture).
- hermes_cli/config.py: documented gateway.loop_watchdog default so
'hermes config set gateway.loop_watchdog false' validates.
- tests: env-knob tests replaced with config-gate + round-trip tests;
the final-strike boundary test injects its probe via max_strikes
directly instead of patching the removed env helper.
- A stop() landing while the final diagnostics (critical log,
traceback dump) are executing could still reach os._exit(75) after
the pre-diagnostic check. Add a third stop_event recheck immediately
before the hard exit: diagnostics may complete, but a disarmed
watchdog never exits.
- Deterministic regressions for both windows (stop triggered from
inside logger.critical and from inside faulthandler.dump_traceback);
mutation-verified (removing the check turns both red). Frozen-loop
semantics unchanged.
Addresses the second round of the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Re-check stop_event after a missed probe (before the strike
increment) and again on entering the final-strike branch (before the
critical log, dump, and hard exit), so a normal stop() landing
between the last timeout check and the exit path can no longer be
misclassified as a freeze and trigger a supervisor restart.
- Deterministic boundary tests pin both re-checks independently
(mutation-verified: removing either check turns its own test red);
frozen-loop semantics are unchanged.
Addresses the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A self-rescheduling 5s call_later floor timer, armed before any
adapter connects, guarantees the selector always has a finite
timeout, so the existing async defenses (polling heartbeat, timeout
guards) regain a chance to run after a zero-pending-timer stall.
- A resident daemon-thread liveness watchdog probes the loop via
call_soon_threadsafe every 30s; after 3 consecutive 10s-timeout
misses (~120s of total unresponsiveness) it dumps all thread
tracebacks and exits with the established
GATEWAY_SERVICE_RESTART_EXIT_CODE (75) so a supervisor restarts the
gateway - async-level recovery cannot run on a frozen loop.
- stop() disarms both guards before any teardown await so a busy
shutdown is never misjudged as a freeze.
HERMES_GATEWAY_LOOP_WATCHDOG=0 disables; _INTERVAL/_TIMEOUT/_STRIKES
tune the thresholds.
Fixes#69089
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
signal.SIGUSR2 and faulthandler.register() don't exist on Windows;
the bare reference raised AttributeError at import time per the
windows-footgun checker. faulthandler.enable() still covers
fatal-error dumps on all platforms.
When connected_count == 0 and at least one platform failed with a
non-retryable error, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE
(78) even if OTHER platforms failed for merely transient reasons.
Real-world shape (NS-609, hosted instance): WhatsApp enabled but never
paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during
polling startup (retryable) => exit 78 => the gateway either goes
permanently down (supervisors honoring the exit-78 contract via
RestartPreventExitStatus / the s6 finish->125 translation from #51228) or
crash-loops (anything else). Either way Telegram never gets its retry and
the dashboard drops with every exit, so a single unpaired platform plus
one network blip disconnected every channel on the instance.
Now exit 78 is reserved for the case where ALL startup failures are
non-retryable (true config error, nothing to wait for). With mixed
failures the gateway stays alive in degraded state: the reconnect watcher
recovers the retryable platforms and the misconfigured ones stay
fatal-parked and visible in runtime status.
Three-part fix for the gateway going silently deaf after a retryable
fatal adapter error (e.g. httpx.ConnectError on Telegram):
1. **Detach-on-timeout in _connect_adapter_with_timeout** — Replaced
plain asyncio.wait_for with the task-detach pattern used by
_await_adapter_cleanup_with_timeout. asyncio.wait_for cancels the
overdue task but then waits for it to exit, so a connect() that
catches CancelledError can block recovery forever. The detach
pattern releases the runner at the deadline via
consume_detached_task_result.
2. **Ensure reconnect watcher always runs after escalation** — Added
_ensure_reconnect_watcher_running(), called after queueing a
retryable fatal error. If the reconnect watcher task has died
(exhausted restart budget, terminal exception), it is respawned
so queued platforms are never permanently stranded.
3. **Faulthandler at gateway startup** — Enabled faulthandler +
SIGUSR2 dump to a rotating file under HERMES_HOME/logs/ for
post-mortem diagnosis of future event-loop freezes.
Tests added for _ensure_reconnect_watcher_running (alive, dead,
not-started, not-running), fatal-error integration (retryable calls
ensure, non-retryable does not), and _connect_adapter_with_timeout
(timeout raises, success returns).
/api/status?profile= now passes pid_path=/path=/expected_home= to the
PID and runtime-status readers; the profile-unification fakes had
zero-arg signatures and raised TypeError. Plain /api/status call shapes
are unchanged (pinned by the existing zero-arg tests in
test_web_server.py).
Follow-up for the salvaged #70498 fix: replace the original PR's
mock-signature churn (28 lambda **kw edits, needed only because it changed
the no-profile call shape) with two targeted regression tests:
- ?profile=<name> must pass the profile's gateway.pid / gateway_state.json
paths and expected_home to the gateway status readers (HOME-anchored
per-profile state under ~/.hermes/profiles/<name>/)
- ?profile=<unknown> must 404 via _resolve_profile_dir
The production change keeps plain /api/status on the exact zero-arg calls,
so every pre-existing test passes unmodified.
