Leg 2 of #63078: prompt.submit returns {"status":"streaming"} immediately and
runs _start_agent_build + _wait_agent(timeout=30s) behind it. The deferred
build (MCP discovery with per-server retry backoff, synchronous model-metadata
HTTP, skills scanning) routinely outlives 30s on cold starts; on timeout
run_after_agent_ready emitted an error EVENT and returned without ever calling
_run_prompt_submit — the user's first message was permanently discarded while
the build finished successfully in the background. The desktop's optimistic
row eventually cleared with no visible error: the blank first session.
New _wait_agent_for_prompt replaces the flat cliff for the deferred prompt
path only (_sess()'s RPC-blocking _wait_agent keeps its 30s contract):
- The pending prompt stays attached to the (already off-RPC) run thread and
is delivered the moment the still-running build completes — a slow build is
no longer message loss.
- The wait runs in 5s slices so a cancel (session.interrupt / churn) is
honored promptly; the cancelled path returns None and defers to the
caller's cancel branch (the #65567 emit) for user-visible messaging.
- Past 30s the client gets ONE keyed notification.show ('Still starting the
agent…', key=agent-build-slow, desktop toast / TUI status bar), cleared on
delivery — patient, never silent.
- Permanent failure only when the build itself fails: agent_error set at
ready, the build thread died without signalling ready (fail fast via the
new _agent_build_thread handle instead of sitting out the cap on a corpse),
or the bounded cap expired on a genuinely hung build. The cap defaults to
600s and is tunable via agent.build_wait_timeout in config.yaml (no new
env vars); the error message states the message was not sent.
Tests: slow-build delivery with zero error events; the keyed progress notice
shown once and cleared; build-failure surfacing exactly one error event with
the real reason; dead-thread fail-fast; cancel honored mid-wait; config
override + fallback semantics; cap expiry message. The compute-host fallback
test stubs the new waiter alongside _wait_agent.
run_after_agent_ready() silently returned when _turn_cancel_requested or
running=False was set during lazy agent startup, leaving the Desktop with
a {"status":"streaming"} reply that never produced a message.start or
error event. The _wait_agent error branch 6 lines above already emits;
this mirrors it so the client can surface feedback instead of hanging.
This is the server-side half of #63078 — the client-side drift guard is
addressed by #64327, but even with that fix the server-side cancel race
still silently dropped the turn.
Adds two regression tests that capture _emit (the existing sibling test
mocked it to a no-op, so it could not catch the silent drop).
Closes#63078
The post-update state.db integrity guard called verify_sqlite_integrity()
with max_bytes=0, which disables the size ceiling and forces a full
PRAGMA integrity_check. That pragma walks every b-tree page in the file,
so its cost scales with database size — measured on a real 30 GB state.db:
143.5s with a cold page cache (worse under an update's memory pressure),
with zero output on screen. The update looks hung right after
"✓ Code updated!" and a CPU sits pegged.
Multi-GB session databases are normal for heavy users, so a
size-unbounded check is never an acceptable default on the update path.
- verify_sqlite_integrity(): max_bytes now defaults to
DEFAULT_INTEGRITY_CHECK_MAX_BYTES (2 GiB) instead of 0. max_bytes=0
remains the explicit opt-in for a full scan.
- The oversized path no longer degrades to a header-only check: it adds a
constant-time structural probe (read-only open + schema_version +
sqlite_master read) so the malformed-schema class is still caught, not
just the #68474 zeroed-file signature.
- Drop the explicit max_bytes=0 at the post-update guard and in
copy_db_and_verify() so both inherit the bounded default.
Measured on the reporter's real 30 GB state.db: 143.5s cold → 0.001s,
still valid=True. Corruption detection verified at multi-GB scale for
both classes (zeroed header, malformed schema) — both still fail closed.
Tests: default-is-bounded invariant, oversized probe catches malformed
schema, max_bytes=0 still forces the full check.
A real APPLICATION_COMMAND interaction forwarded over the relay arrived
slash-less: _discord_interaction_to_event set text = data['name'] ("new",
not "/new"), MessageType.TEXT, and dropped options entirely — so a
registered /new dispatched as plain chat instead of a command
(MessageEvent.is_command() is text.startswith("/")).
Port the connector's Slack slash-command precedent (normalizeSlackCommand
builds `${command} ${args}`.trim() with a leading slash and explicit
command type): for type-2 interactions build "/" + name, append rendered
options space-separated (scalar options contribute their value, matching
the native adapter's f"/model {name}" shape; SUB_COMMAND/
SUB_COMMAND_GROUP contribute their name then recurse into nested
options), and set MessageType.COMMAND. Type-3 (custom_id) and other
interaction types are unchanged. This implements the interaction->command
sub-design previously flagged as deferred in the _on_passthrough
docstring.
Companion connector fix in gateway-gateway: fix(relay): strip own-mention
prefix so addressed slash commands dispatch.
Session previews are the first 60 characters of the first user message, so
persisting the @image: directives ahead of the caption labelled the session
with a truncated file path in the sidebar, session switcher, and command
palette. Clients lift the refs out of the body line by line, so moving them
after the caption changes nothing about how the turn renders.
A turn routed to a model that takes pixels directly sends `content` as a
parts list, and the session store deliberately ignores a plain-string
persist override for a list payload — a text override must not erase a
turn's image summary. So the override was dropped for every user on a
vision-capable main model, and the durable row kept only the caption plus a
literal `[Image attached at: ...]` / `[screenshot]`, which the renderer
cannot turn back into an image. Only vision-preprocessed (text-mode) turns
were actually fixed.
Mirror the shape instead: swap the text part for the `@image:` ref form and
keep the image parts, so the model still has the pixels for the rest of the
session, and drop the `[screenshot]` stand-in on the way into the bubble
when a ref was lifted from the same message.
The unquoted alternative in the directive pattern is `\S+`, so a ref built
by string interpolation truncates at the first space and strands the tail
as loose text next to a broken thumbnail. Composer images live in the app's
userData dir, which on macOS is `~/Library/Application Support/<App>/` — so
every pasted or dropped image hit this.
Adds format_reference_value next to REFERENCE_PATTERN, mirroring
formatRefValue in the desktop's directive-text.tsx, and covers the
round-trip through the parser.
The desktop gateway passed the vision-enriched, model-only message text
(carrying an `image_url:<path>` hint) straight into run_conversation as
the persisted user turn. The renderer only parses `@image:<path>`, so it
could not rebuild the attachment from history: after a restart the image
was gone and only the caption survived, and on a live session switch the
warm cache disagreed with the authoritative text and the frontend
"rescued" the image by appending it after the caption.
run_conversation already supports persist_user_message for exactly this
"what the model sees" vs "what gets stored" split; it was simply never
wired up for the attachment path.
Bug-class coverage for both fixes: the full catalogue survives Save, context
lengths are preserved, the key never lands in config.yaml on either write
path, blank clears it, a pre-fix plaintext key migrates while a ${VAR}
template is left alone, two endpoints on one host keep separate credentials,
and an IP-derived name is still a valid POSIX env var.
The two delete tests asserted on the plaintext mirror; they now assert the
same invariants against the credential reference.
The desktop self-update chain (Desktop -> hermes-setup --update ->
hermes update -> hermes desktop --build-only -> relaunch) rebuilds
Hermes.exe on the user's machine and declared success on bare file
EXISTENCE. A truncated PE (corrupt cached Electron zip / interrupted
extraction or rcedit rewrite / full disk) or a wrong-architecture
unpacked tree therefore shipped as the 'updated' app, which Windows
refuses to load with 'This app can't run on your computer'
(此应用无法在你的电脑上运行) — and the previous working build had
already been wiped by before-pack.mjs, leaving nothing to fall back to.
Fix, in three parts:
- hermes_cli/main.py: post-build integrity gate on Windows
(_ensure_desktop_exe_launchable). Parses the PE header of the freshly
built Hermes.exe — MZ/PE magic, section-table completeness vs file
size (catches truncation), and COFF machine vs the host arch (catches
arm64/x64 mixups). On failure it purges the (likely corrupt) cached
Electron zip, invalidates the content-hash build stamp so the
updater's retry-once genuinely re-downloads and rebuilds, restores
the previous build from the .bak tree when one exists (keeping the
corrupt tree as .corrupt for diagnostics), tells the user the update
was aborted and their old version kept, and exits nonzero.
_desktop_packaged_executable also now prefers a host-loadable PE over
pure newest-mtime when multiple win-*-unpacked trees coexist.
- apps/desktop/scripts/before-pack.mjs: on win32, the previous unpacked
tree is preserved as <appOutDir>.bak (only when it holds the product
exe — partial/corrupt trees still get the plain wipe) instead of
being destroyed, providing the rollback material for the gate above.
Non-Windows behavior is unchanged.
- Behavior-contract tests: tests/hermes_cli/test_desktop_exe_integrity.py
(23 tests — synthetic PE fixtures for truncation/non-PE/arch-mismatch,
rollback semantics, and the build-only exit contract) and 6 new vitest
cases in before-pack.test.mjs for the .bak preservation rules.
Progresses #69179
The strict cold-start readiness gate (#67498) means adapter.connect() no
longer returns True until the mocked start_polling records a successful
getUpdates round trip for its generation. Update the conflict-suite
Application mocks accordingly:
- fake_start_polling side effects call
adapter._record_polling_progress(adapter._polling_generation) on the
initial connect (retry generations intentionally do NOT auto-progress
where a test asserts the conflict count survives an unproven retry).
- _build_polling_app takes the adapter so its start_polling mock can
record progress.
Without this, the cold connects in these tests wait out the full 60s
readiness deadline and fail — which is exactly the fail-closed behavior
the gate is supposed to provide when polling shows no progress.
Follow-up hardening for the salvaged #69240 readiness gate (#67498):
- _start_polling_once now returns its (generation, progress_event) pair
so the strict cold-start gate binds to exactly the generation it
started, instead of re-reading self._polling_progress_event which a
concurrent recovery task may have replaced with a newer generation's
event (the G1/G2 race flagged in the #69240 review).
- Strict cold start no longer schedules background polling recovery: a
polling error during the readiness wait is captured by a strict
callback and fails the connect attempt immediately with a loud
OSError, so GatewayRunner disposes the partial adapter and retries
with a fresh one — no more waiting out the full readiness deadline on
a generation that already errored, and no G2-on-partial-app healing.
- After readiness is proven the strict callback delegates every later
polling error to the real background-recovery callback, preserving
the existing degraded/reconnect semantics for the polling lifetime.
- The readiness-timeout error message now states the deadline and that
the gateway will retry with a fresh adapter (loud failure, not a
silent wait).
- Regression tests: current-generation progress connects; a polling
error during strict cold start fails fast without scheduling
background recovery (the #67498 idle-threads shape); stale-generation
progress is rejected.
Progresses #67498
Give Telegram a 180s default outer connect budget so cold polling can prove getUpdates readiness. Preserve the 30s default for other platforms and all explicit config/env overrides.\n\nRefs #67498
Use wall deadlines for deleteWebhook and start_polling, then fail cold startup unless getUpdates proves progress. This lets the gateway discard partial PTB state and retry with a fresh adapter.\n\nRefs #67498
Follow-up to the cherry-picked #69063: egilewski's review found that the
_dir_has_any_entry(parent) guard treats ANY entry in the mount point's
parent as proof the volume is attached — but unmounting exposes the
UNDERLAY directory's own files (e.g. a .keep placeholder), so a populated
underlying mount-point dir still classified the project as an orphan and
deleted its ref/index/metadata. Reproduced on both main and the PR head.
Attachment evidence is now positive instead of circumstantial:
* _volume_evidence() records the parent directory's (st_dev, st_ino)
identity in the project's metadata while the workdir is observably
live (at _register_project/_touch_project time). A mount point
resolves to the mounted filesystem's root while attached and to the
underlay directory after detach — same path, different directory,
different identity.
* _workdir_is_observably_gone() now requires the parent visible at
prune time to match that recorded identity before the populated-parent
check can classify an orphan. A mismatch means a different directory
(the underlay) is showing through — a detached volume, not an
observed deletion.
* Metadata without a recorded identity (written by older versions) is
never orphan-classified — unsure never deletes; the retention/stale
rule still reclaims genuinely abandoned projects off last_touch.
* The frozen pre-v2 layout has no metadata channel for the identity, so
it keeps the structural checks only (require_parent_identity=False).
* A failed evidence probe on re-registration preserves the previously
recorded identity — stale evidence can only make pruning MORE
conservative.
Windows: st_dev/st_ino of 0 (filesystems without file IDs, some network
shares) is treated as "no evidence recorded", which falls into the
conservative never-orphan path. os.path.ismount and Path.stat are
cross-platform; no POSIX-only calls added.
tests/tools/test_checkpoint_manager.py: adds egilewski's exact
regression (checkpoint history for mnt/volume/project, detach exposes
mnt/volume/.keep, prune with orphan deletion enabled → NOT deleted;
fails on the bare cherry-pick, passes with this fix), plus
no-recorded-identity conservatism and probe-failure identity
preservation. His absent-parent/empty-parent/retention/genuine-deletion/
live-project controls all still pass.
Reported-by: egilewski (review on #69063)
Addresses @egilewski's review: the parent-directory check still deleted
checkpoint history for the most common unmount layout.
Detaching storage removes the parent outright in some layouts
(`/Volumes/Ext/proj` on macOS, `/media/<user>/<label>/proj`), which the first
commit handles. But in the classic static layout — `/mnt/volume/proj`, an
fstab entry, a container bind-mount — unmounting removes the contents and
leaves the mount point behind as an empty directory. `parent.is_dir()` is then
true, the project is absent, and the startup sweep deletes its ref, index and
metadata: exactly the case this PR set out to protect.
Reproduced against the real predicate before this commit:
mount root vanished (macOS) -> False ok
empty surviving mount point -> True <-- history deleted
really deleted (siblings) -> True ok
An empty parent carries no information: it looks identical whether the volume
was detached or the project was deleted. So require the parent to actually say
something — it holds some other entry (we observed a populated directory that
does not contain the project), or it is itself a live mount point (the volume
is attached right now and demonstrably does not hold the project).
The cost is that a project deleted out of an otherwise-empty parent is no
longer reclaimed by the orphan rule. It is not leaked: the retention rule
reads `last_touch` rather than probing the filesystem and still collects it,
so reclamation is deferred, not lost. That is the right direction for a
predicate whose false positive destroys a user's restore points unattended.
`_dir_has_any_entry` stops at the first entry via `os.scandir` instead of
materializing a listing, since a project root can hold a large tree.
tests/tools/test_checkpoint_manager.py: `test_surviving_empty_mountpoint_
keeps_its_checkpoints` pins the reviewed case, and `test_empty_parent_project_
is_still_reclaimed_by_retention` pins the deferral above so the safety valve
cannot silently regress into a leak. Both fail on the previous commit. The
real-orphan control now seeds a sibling so it exercises a populated parent
rather than the ambiguous empty one. 80 passed in the checkpoint suite; the 2
remaining failures (`TestGitEnvIsolation`, `TestClearFunctions`) fail
identically on clean main.
Orphan pruning decides a project is gone from a single probe:
if delete_orphans and (not workdir or not Path(workdir).exists()):
reason = "orphan"
then deletes its ref, index, and metadata — the project's entire checkpoint
history. `Path.exists()` is False for a deleted directory, but it is equally
False for one whose storage is not attached right now: an unplugged external
drive, a share behind a downed VPN, a bind-mount absent from this container,
an offline Windows mapped drive. The project is fine; only our view of it is.
This is not an opt-in maintenance command. `maybe_auto_prune_checkpoints`
runs unattended at startup from both `cli.py` and `gateway/run.py`, with
`delete_orphans=True` by default. So starting Hermes once while the drive is
unplugged silently destroys the restore points for every project on it — the
one thing checkpoints exist to provide, and there is nothing to restore from
afterwards.
Reproduced against the real store: a project registered under an unmounted
path and one on local disk, then a startup prune —
prune: {'scanned': 2, 'deleted_orphan': 1}
unreachable project index still on disk: False
The legacy pre-v2 branch has the same flaw plus a second one: a
`HERMES_WORKDIR` marker that exists but cannot be read leaves `workdir = None`,
which the same condition treats as an orphan. Failing to read a file is not
evidence that a project was deleted.
Require corroboration before deleting: the workdir's parent must be present,
so its absence is something we actually observed. A missing parent means the
volume is not there and we know nothing, so the entry is left alone — and an
unreadable marker never deletes at all. Genuinely abandoned projects are still
reclaimed, both by the unchanged orphan path (parent present, project gone)
and by the retention/stale rule, which runs off `last_touch` rather than a
filesystem probe.
tests/tools/test_checkpoint_manager.py: a project whose whole mount disappears
keeps its history; controls prove a genuinely deleted project is still pruned
and a live project is untouched. The data-loss test fails on main; both
controls pass there. 81 passed across the checkpoint suites (2 failures in
test_checkpoint_manager.py are pre-existing and fail identically on clean
main).
export_session() reads through get_messages(), so before the read fix an
already-serialized string went straight back into _insert_message_rows() and
got re-dumped — an export/import round trip permanently corrupted the row.
Guard the three write paths the same way tool_calls already is: parse a
string argument before storing it, and drop metadata that isn't an object
rather than persisting something no reader can use.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: aml1973 <aml1973@users.noreply.github.com>
get_messages(), get_messages_around() and get_anchored_view() returned the
raw display_metadata column instead of the dict every caller expects. The
desktop paints a resumed transcript from the REST prefetch, which reads
through get_messages(), so any session holding an async_delegation_complete
event failed resume with "Cannot use 'in' operator to search for
'task_count'" — on every such session, not just corrupted ones.
Route all four read paths through one shared codec that also unwraps rows
carrying a second JSON layer, so sessions already broken on disk recover on
read rather than needing a migration.
Co-authored-by: Studio729 <Studio729@users.noreply.github.com>
Co-authored-by: aml1973 <aml1973@users.noreply.github.com>
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
/api/status is the only public liveness route, and its handler loads the
gateway config, probes gateway health, and counts sessions before it can
answer. That work is wrong for a readiness probe: a caller that only needs
to know the process is up pays for a cold plugin import tree.
Add /api/health, which returns process liveness, version, and the auth-gate
shape and touches nothing else.
The bundled office skills (#68595) read user documents and agent-authored
payloads with the locale-default codec:
- docx/powerpoint validators/base.py opened OOXML part XML in text mode
before handing it to lxml. On Windows (cp1251/GBK) the bytes decode to
mojibake that lxml then parses, so validation runs against silently
corrupted document text; on locales where the UTF-8 bytes don't decode
the validator crashes with UnicodeDecodeError instead of validating.
Opening as bytes lets lxml honor the encoding declared in the XML prolog.
- The pdf form scripts (fill_fillable_fields, fill_pdf_form_with_annotations,
create_validation_image, check_bounding_boxes) read the fields JSON the
agent authors — UTF-8 by construction — with the locale codec, so
non-ASCII form values (any Cyrillic/CJK/accented input) either crash or
get written into the user's PDF as mojibake. The json.dump writers use
ensure_ascii=True and were already safe; only the readers needed pinning.
Adds a contract test asserting every document/payload reader is
locale-independent, plus a live regression test that runs
check_bounding_boxes.py on a non-ASCII fields.json under a forced
non-UTF-8 locale — it fails without the fix on both POSIX (C locale)
and Windows (cp1251 chokes on the 0x98 byte of U+2018).
_setup_worktree read both files with the locale default encoding. On a
cp1251/GBK Windows machine a UTF-8 include list either decodes to
mojibake paths (non-ASCII entries silently not copied) or raises
UnicodeDecodeError, which the enclosing handler logs at DEBUG and
swallows — no include is copied at all, so the worktree starts without
.env/keys and the agent breaks invisibly. A Notepad BOM likewise glues
to the first include entry on every platform, and to the first
.gitignore line, defeating the '.worktrees/' membership check and
appending a duplicate entry on each run.
Read both files with utf-8-sig + errors=replace, matching the canonical
.env readers in hermes_cli/config.py (utf-8-sig because Notepad adds a
BOM) and the UTF-8 append this same block already performs on
.gitignore.
Regression tests exercise the real cli._setup_worktree: the two BOM
tests fail without the fix on any platform, the non-ASCII include test
additionally reproduces the Windows locale failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_read_hub_installed_names() reads ~/.hermes/skills/.hub/lock.json with a
strict utf-8 decode. Hub skill descriptions can carry Windows-1252
typographic bytes (em-dash 0x97, smart quotes, bullets) as single high
bytes; read_text(encoding="utf-8") then raises UnicodeDecodeError, which
is a ValueError sibling not caught by the function's
except (OSError, json.JSONDecodeError). It escapes and 500s the whole
/api/skills endpoint, blanking the desktop Skills panel.
Decode with errors="replace" so the offending byte degrades to U+FFFD
and the structurally valid JSON — and every other skill — stays readable.
Fixes#68053
`_render_distribution_plan` reads the target profile's `.env` to decide
whether a required env var is already set (so it doesn't nag the user),
using `Path.read_text()` with no encoding. Two bugs:
1. `Path.read_text()` defaults to the system locale (cp1251/GBK on Windows),
which raises `UnicodeDecodeError` on any non-ASCII byte. The surrounding
`except OSError` does NOT catch that — `UnicodeDecodeError` is a
`ValueError` — so a mis-encoded `.env` aborts the entire install preview.
2. Even on a UTF-8 locale, a Notepad-added BOM prefixes the first key
(`KEY`), so the very first required env var is mis-reported as
"needs setting" when it is actually present.
`.env` is written as UTF-8 everywhere in the codebase. Read it as
`utf-8-sig` (tolerates the BOM) and also catch `UnicodeDecodeError` so a
genuinely un-decodable file skips the pre-check instead of crashing.
Regression tests: a BOM-prefixed `.env` whose first key must still read as
"set", and an invalid-UTF-8 `.env` that must not abort the preview.
Follow-up to review feedback:
- mem0 _prompt_api_key read .env with the locale default, so a Notepad
BOM hid the first key from the masked current-value lookup; read it
with utf-8-sig + errors=replace like the canonical readers in
hermes_cli/config.py.
- hindsight _load_simple_env used plain utf-8; it also parses the Hermes
.env during post_setup, where a BOM stuck to the first key. Switch to
utf-8-sig + errors=replace.
- Add hindsight regressions: BOM key matching in _load_simple_env and in
the cloud post_setup writer, plus non-ASCII round-trip preservation,
and a mem0 regression for the BOM'd masked-key lookup. The BOM tests
fail without the fix on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mem0 and hindsight memory-provider setup routines round-trip the
user's ~/.hermes/.env: they read existing lines, update the keys they
manage, and rewrite the whole file preserving every other line verbatim.
Both used env_path.read_text() / write_text() with no encoding.
read_text()/write_text() with no encoding fall back to the system locale
(cp1252/GBK on Windows), so on a non-UTF-8 host the preserved lines get
mangled or the call crashes on any non-ASCII value, and — because the
reader never strips a BOM — a Notepad-edited .env makes the first key
fail the in-place match and get duplicated instead of updated.
Match the canonical .env readers in hermes_cli/config.py: read with
encoding='utf-8-sig' (BOM-tolerant) and write with encoding='utf-8'.
mem0/_setup.py already pins utf-8 for mem0.json, so this just aligns the
.env path in the same file. Fixes both memory plugins in one class fix.
Adds regression tests: a BOM'd .env updates the first key in place
(locale-independent, fails without the fix) and non-ASCII existing lines
survive the round-trip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses hermes-sweeper review on PR #54866: the installer runs
tools/skills_sync.py as a child python.exe whose PYTHONIOENCODING /
PYTHONUTF8 the scoped install.ps1 block sets, but there was no
regression test for this child-Python UTF-8 path. The existing
test_child_process_inherits_utf8_mode covers a different (bootstrap
entry-point) flow.
Add TestSkillsSyncUtf8Guard: three subprocess tests that import
skills_sync (triggering its import-time stdout/stderr reconfigure)
and assert the checkmark/up-arrow glyphs the script prints at
tools/skills_sync.py:596,675 emit valid UTF-8 and exit 0 even when
the child env is left unset or explicitly hostile (gbk). A third
test proves the guard is load-bearing by reproducing the crash
without it.
Also keep the new install.ps1 comment ASCII-only (the checkmark
spelled out as U+2713) per the file's PS 5.1 parser-compatibility
contract at scripts/install.ps1:79-80; the literal glyph in the
comment violated that contract.
Several file-I/O call sites still use open() / Path.read_text() /
Path.write_text() without an explicit encoding, so they fall back to
the platform default. On Windows CN/JP/KR locales (GBK/CP932/CP949)
any non-ASCII byte in a config/state/user-content file raises
UnicodeDecodeError or UnicodeEncodeError and crashes the caller.
to the remaining hot paths:
- agent/copilot_acp_client.py: fs/read_text_file and fs/write_text_file
(Copilot's read_file / write_file tools,
directly reported in #18637 bug 2)
- agent/model_metadata.py: context-length YAML cache load + two
save sites (context probing is on the
call path of every model invocation)
- agent/nous_rate_guard.py: cross-session rate-limit JSON state
(read + atomic write via os.fdopen)
- cron/scheduler.py: user config.yaml read in run_job
- gateway/delivery.py: cron output writes for AI-generated
content, very likely non-ASCII
yaml.dump call sites also gain allow_unicode=True so the emitted
YAML preserves non-ASCII chars as-is instead of emitting \u escape
sequences.
Adds regression tests that monkeypatch builtins.open / Path.read_text
/ Path.write_text to simulate a GBK locale: each test raises
UnicodeDecodeError / UnicodeEncodeError unless the caller explicitly
passes encoding='utf-8'. Verified that the tests fail on main and
pass with this change, on Linux as well as on Windows.
Refs #18637
The three refresh tests asserted the replaced shared client gets
close()d — the exact cross-thread close#70773 removes. They now pin
the new contract: close() is NOT called from the refresh path; the
old client is retired (sockets shutdown, FD release deferred to GC).
Widen the #70773 fix beyond the three in-request cleanup sites removed in
the cherry-picked commit: every remaining path that swaps out the shared
OpenAI client could still hard-close its pool from a thread that doesn't
own the in-flight sockets (credential rotation/refresh on the turn thread,
dead-connection cleanup, gateway cache eviction, transport recovery) —
the same FD-recycle corruption vector, just rarer.
Add AIAgent._retire_shared_openai_client(): shutdown(SHUT_RDWR) all pooled
sockets (FD-safe from any thread, unblocks in-flight readers) but never
call client.close() — FD release is deferred to GC, which cannot run until
every borrowing thread has unwound its SSL BIO. Refcounting is the
ownership handshake; with no borrowers the FDs are released immediately.
Wired into:
- _replace_primary_openai_client (rotation/refresh/dead-conn cleanup)
- try_recover_primary_transport (primary_recovery)
- release_clients (gateway cache_evict)
agent.close() keeps the hard close: full teardown is a real session
boundary where no request may be in flight.
Tests: new tests/run_agent/test_70773_shared_client_fd_corruption.py
covers the three watchdog/retry sites plus retire semantics; existing
close-assertions updated to pin retire-not-close.
Follow-up to the salvaged #69164 commits: policy forbids introducing new
HERMES_* environment variables, so the four watchdog env knobs
(HERMES_GATEWAY_LOOP_WATCHDOG / _INTERVAL / _TIMEOUT / _STRIKES) are
replaced with a single config.yaml boolean:
gateway:
loop_watchdog: true # default; false disables both guards
- gateway/config.py: new GatewayConfig.loop_watchdog field (default True),
parsed from top-level or nested gateway: form, round-trips via
to_dict/from_dict.
- gateway/run.py: _start_loop_liveness_guards() checks config.loop_watchdog
before arming the floor timer + watchdog (getattr-guarded for bare
object.__new__ runners).
- gateway/shutdown_watchdog.py: start_loop_liveness_watchdog() no longer
reads the environment; probe interval/timeout/strikes are module
constants (30s/10s/3 — ~90s to restart, matching the systemd watchdog
layer's posture).
- hermes_cli/config.py: documented gateway.loop_watchdog default so
'hermes config set gateway.loop_watchdog false' validates.
- tests: env-knob tests replaced with config-gate + round-trip tests;
the final-strike boundary test injects its probe via max_strikes
directly instead of patching the removed env helper.
- A stop() landing while the final diagnostics (critical log,
traceback dump) are executing could still reach os._exit(75) after
the pre-diagnostic check. Add a third stop_event recheck immediately
before the hard exit: diagnostics may complete, but a disarmed
watchdog never exits.
- Deterministic regressions for both windows (stop triggered from
inside logger.critical and from inside faulthandler.dump_traceback);
mutation-verified (removing the check turns both red). Frozen-loop
semantics unchanged.
Addresses the second round of the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Re-check stop_event after a missed probe (before the strike
increment) and again on entering the final-strike branch (before the
critical log, dump, and hard exit), so a normal stop() landing
between the last timeout check and the exit path can no longer be
misclassified as a freeze and trigger a supervisor restart.
- Deterministic boundary tests pin both re-checks independently
(mutation-verified: removing either check turns its own test red);
frozen-loop semantics are unchanged.
Addresses the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A self-rescheduling 5s call_later floor timer, armed before any
adapter connects, guarantees the selector always has a finite
timeout, so the existing async defenses (polling heartbeat, timeout
guards) regain a chance to run after a zero-pending-timer stall.
- A resident daemon-thread liveness watchdog probes the loop via
call_soon_threadsafe every 30s; after 3 consecutive 10s-timeout
misses (~120s of total unresponsiveness) it dumps all thread
tracebacks and exits with the established
GATEWAY_SERVICE_RESTART_EXIT_CODE (75) so a supervisor restarts the
gateway - async-level recovery cannot run on a frozen loop.
- stop() disarms both guards before any teardown await so a busy
shutdown is never misjudged as a freeze.
HERMES_GATEWAY_LOOP_WATCHDOG=0 disables; _INTERVAL/_TIMEOUT/_STRIKES
tune the thresholds.
Fixes#69089
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When connected_count == 0 and at least one platform failed with a
non-retryable error, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE
(78) even if OTHER platforms failed for merely transient reasons.
Real-world shape (NS-609, hosted instance): WhatsApp enabled but never
paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during
polling startup (retryable) => exit 78 => the gateway either goes
permanently down (supervisors honoring the exit-78 contract via
RestartPreventExitStatus / the s6 finish->125 translation from #51228) or
crash-loops (anything else). Either way Telegram never gets its retry and
the dashboard drops with every exit, so a single unpaired platform plus
one network blip disconnected every channel on the instance.
Now exit 78 is reserved for the case where ALL startup failures are
non-retryable (true config error, nothing to wait for). With mixed
failures the gateway stays alive in degraded state: the reconnect watcher
recovers the retryable platforms and the misconfigured ones stay
fatal-parked and visible in runtime status.
Three-part fix for the gateway going silently deaf after a retryable
fatal adapter error (e.g. httpx.ConnectError on Telegram):
1. **Detach-on-timeout in _connect_adapter_with_timeout** — Replaced
plain asyncio.wait_for with the task-detach pattern used by
_await_adapter_cleanup_with_timeout. asyncio.wait_for cancels the
overdue task but then waits for it to exit, so a connect() that
catches CancelledError can block recovery forever. The detach
pattern releases the runner at the deadline via
consume_detached_task_result.
2. **Ensure reconnect watcher always runs after escalation** — Added
_ensure_reconnect_watcher_running(), called after queueing a
retryable fatal error. If the reconnect watcher task has died
(exhausted restart budget, terminal exception), it is respawned
so queued platforms are never permanently stranded.
3. **Faulthandler at gateway startup** — Enabled faulthandler +
SIGUSR2 dump to a rotating file under HERMES_HOME/logs/ for
post-mortem diagnosis of future event-loop freezes.
Tests added for _ensure_reconnect_watcher_running (alive, dead,
not-started, not-running), fatal-error integration (retryable calls
ensure, non-retryable does not), and _connect_adapter_with_timeout
(timeout raises, success returns).
/api/status?profile= now passes pid_path=/path=/expected_home= to the
PID and runtime-status readers; the profile-unification fakes had
zero-arg signatures and raised TypeError. Plain /api/status call shapes
are unchanged (pinned by the existing zero-arg tests in
test_web_server.py).
Follow-up for the salvaged #70498 fix: replace the original PR's
mock-signature churn (28 lambda **kw edits, needed only because it changed
the no-profile call shape) with two targeted regression tests:
- ?profile=<name> must pass the profile's gateway.pid / gateway_state.json
paths and expected_home to the gateway status readers (HOME-anchored
per-profile state under ~/.hermes/profiles/<name>/)
- ?profile=<unknown> must 404 via _resolve_profile_dir
The production change keeps plain /api/status on the exact zero-arg calls,
so every pre-existing test passes unmodified.
The .env sanitizer inferred missing newlines from known KEY= substrings
inside existing values. Plain secrets containing those bytes could therefore
be split into synthetic assignments and rewritten to disk.
Treat each physical line as the only assignment boundary and keep bytes after
the first equals sign opaque for boundary discovery. Preserve safe formatting,
null-byte removal, BOM handling, and normal one-assignment-per-line parsing.
Cover direct loading, dotenv loading, sanitization, writers, and migration
with behavioral regressions.
Fixes#29155
Follow-up to the cherry-picked #68258 base: the cross-session-stable
prefix (_cached_system_prompt_static) was only recorded on fresh
builds, so two paths silently degraded to the legacy single-breakpoint
layout (flagged in review of #68258/#69341/#69704):
- Session restore: gateway surfaces build a fresh AIAgent per turn and
restore the persisted prompt verbatim from the session DB; the static
prefix stayed None from turn 2 onward, flip-flopping the wire layout.
- Post-compression cached-prompt reuse: _invalidate_system_prompt()
clears the static prefix, and the keep-cached-prompt branch never
restored it.
Both sites now reconstruct the stable tier and adopt it ONLY when the
authoritative prompt string literally startswith() it — stable-tier
drift (skills edited, identity changed) falls back to the legacy layout
with the stored bytes untouched. Fail-open on any builder error. The
restore-path rebuild is gated on _use_prompt_caching so non-Anthropic
routes skip it entirely.
Refs #68191
Co-authored-by: JonthanaHanh <92574114+JonthanaHanh@users.noreply.github.com>
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Co-authored-by: webtecnica <webtecnica@users.noreply.github.com>
The conversation loop normalizes message text right before the API call so
the request prefix is byte-identical across turns -- the stated reason is
KV cache reuse on local inference servers and better cache hit rates on
cloud providers. Cache breakpoints were injected *before* that pass, which
defeats it.
`_apply_cache_marker` rewrites a plain-string `content` into a
`[{"type": "text", ...}]` block. The normalization pass is guarded on
`isinstance(content, str)`, so every message that just got marked is
silently skipped by it and keeps its raw leading/trailing whitespace. A
message is only marked while it sits in the last-3 window, so:
turn N in the window -> marked, content "file1\nfile2\n"
turn N+1 rolled out -> plain, content "file1\nfile2"
The same logical message is sent with different bytes on consecutive
turns. The prefix stops matching at that position -- which is inside the
span the breakpoints were placed to protect -- so the reusable prefix
collapses back toward the system breakpoint on every turn. Tool results
carry a trailing newline almost by default (any shell command output), so
this is the common case, not an edge case.
Move the injection below every message mutation. Besides fixing the
whitespace divergence this stops breakpoints from being spent on messages
that the orphan sweep or the thinking-only drop is about to remove or
merge away -- a marker on a dropped message is a wasted breakpoint out of
the four available.
Nothing between the old and new call sites reads `cache_control`, and the
mutators now see the plain-string shapes they were written against.
Follow-up for salvaged PR #69141, addressing the last open review point:
cmd_prune() only set orphan_allowlist inside 'if orphans or pre_v2_orphans',
so a zero-orphan preview passed the unrestricted None sentinel down to
prune_checkpoints(), authorizing deletion of any project that became
orphaned between the preview and the rescan — with zero confirmation
calls. The allowlist is now bound unconditionally for every non-force
run (empty preview => empty allowlist); --force keeps None. Adds the
zero-orphan-preview timing regression plus allowlist-identity tests.
