DESIGN.md already said app-wide surfaces must not compete through ad-hoc
z-index literals, and the code disagreed: three overlapping numbering
schemes, and comments narrating the fight ("defaults to z-130, renders
UNDER the onboarding overlay (z-1300) ... bump it above with z-[1310]").
Picking a number meant reading someone else's near miss.
Name the rungs — modal, over-modal, switcher, and the boot chain — and
point the call sites at them. Every rung keeps the exact value it had, so
nothing moves; what changes is that the next overlay has a name to reach
for instead of a number to guess. Local stacking within a component stays
on plain z-10/z-20.
Cmd-Q went straight through to teardown, killing the backend mid-tool-call
— the turn is gone and whatever the agent was part-way through writing
stays part-way written, with nothing on screen to warn about it.
Renderers now report which chats are mid-turn; before-quit merges the
reports and asks, naming them, defaulting to Keep Running. Update, swap,
and uninstall relaunches skip the prompt: those are the app replacing
itself, and a modal there would strand the detached script waiting on a
PID that never exits.
The tail "Hermes is thinking" indicator resets on every flush, but its
timer never did: with no timer key, useElapsedSeconds anchors to mount,
and the indicator mounts with the assistant message. A stall two minutes
into a turn therefore claimed two minutes of silence instead of the two
seconds that had actually passed.
Give the hook an explicit epoch and hand it the timestamp of the activity
the quiet spell followed. Compaction still counts from the turn's start,
which is the span it owns.
A socket opening replays state that already existed — a session parked on
an approval re-emits its request so the UI can draw the prompt. Those
arrive as ordinary events, so launching Hermes, switching profiles, or
riding out a reconnect fired an OS notification for a prompt the user had
known about for an hour.
Hold native notifications for a beat after any gateway opens. The sidebar
row and the inline approval bar still appear immediately; only the OS
notification waits for something that actually just happened.
Diff add/remove lines were hardcoded to Tailwind's emerald/rose while the
overview ruler beside them — and the rest of the app — used --ui-green /
--ui-red, so every diff sat slightly off-brand and stayed put when the
semantic palette moved. Derive the tint, gutter, and text from those two
colors instead. One renderer feeds the tool card, the file preview, and
the review pane, so all three follow.
The call-block decoration reads agent._use_prompt_caching / _cache_ttl /
_use_native_cache_layout directly; the redecoration helper wrapped each in
getattr with divergent defaults (e.g. or-'5m' vs verbatim _cache_ttl).
The flags are unconditionally initialized on AIAgent, so the defaults
served only test fixtures and would mask a real init bug as silent
cache-off. Align with the house style.
_peel_moa_guidance hand-implemented the inverse of moa_loop's
_attach_reference_guidance from a different module — a drifting separator
or shape would make the peel silently no-op and put the last cache
breakpoint on the turn-varying guidance block (the #72626 bug class).
Move the inverse into moa_loop.peel_reference_guidance directly adjacent
to the attach, keep a thin wrapper in conversation_loop, and pin the
contract with a round-trip test over all three attach shapes.
Also fix the empty-list residue: peeling a guidance-only content part now
drops the whole message (mirroring the appended-user-message shape)
instead of leaving an empty-content user turn behind.
guidance=None is a real prepared shape (all references failed / silent
degraded policy builds prepared_request without attaching guidance), and
the MoA facade sends prepared['messages'] — not api_kwargs['messages'].
Gating the rebase on 'and guidance' left the stale decoration in the
prepared object for the no-guidance MoA sub-path, so #72626 persisted
there. rebase_prepared_request already handles falsy guidance (copies
messages, skips the attach).
The static-prefix reconstruction pattern (build_system_prompt_parts ->
['stable'] -> startswith gate -> fail-open) existed in three copies:
session restore (conversation_loop), compression keep-prompt path
(conversation_compression), and the new failover redecoration helper.
Hoist it into agent/system_prompt.reconstruct_static_prefix and call it
from all three sites.
Also memoize failed rebuilds per stored prompt (_static_rebuild_failed_for):
the redecoration chokepoint runs at the top of every retry attempt, and a
persistent stable-tier mismatch (restored session whose SOUL.md/skills
changed since save) would otherwise re-run the full prompt build — SOUL.md,
context files, memory I/O — on every attempt of every API call for the
life of the session. A legitimately changed stored prompt retries once.
strip_anthropic_cache_control flattened ANY pure-text multi-part content
list with a separator-less join. Decoration only ever produces a single
text part or the 2-part [static, volatile] system split; organic
multi-part text (merged user turns, imported transcripts) got word-jammed
and parts carrying extra keys (citations) were silently dropped — on the
common no-failover path, since redecoration runs on every attempt.
Restrict the flatten to the exact decoration-produced shapes and make
marker removal copy-on-write on part dicts (the per-call message copy is
shallow, so parts alias the persistent history).
Add strip_anthropic_cache_control coverage and policy-change cases
(cache-off→on, on→off, native→envelope, MoA guidance outside marker)
that TestSyncFailoverPreservesCacheDecoration did not exercise.
try_activate_fallback refreshes the cache policy flags for the new
provider, but the retry loop reused the primary's decorated api_messages.
Cache-off→cache-on shipped zero breakpoints; cache-on→cache-off left
stale markers. Strip and re-render at each retry attempt (same chokepoint
as reasoning-echo reapply), peel/rebase MoA guidance so the last marker
stays off the turn-varying block, and rebuild the static system prefix
when caching becomes active mid-turn (#72626).
compress_context now runs on a daemon pool worker thread (via
run_compress_context_with_progress_timeout). The session id rotation
updates hermes_logging._session_context (a threading.local) on the
WORKER thread, not the caller thread. After the wrapper returns,
propagate self.session_id back to the caller's logging context so
subsequent log lines carry the rotated id (#34089).
Fixes CI failure in test_compression_logging_session_context.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
PR #72425 added getattr(agent, '_incremental_persistence_failed', False)
checks at the top of execute_tool_calls_{sequential,concurrent,segmented}.
A bare MagicMock auto-creates a truthy value for any attribute access,
so the interrupt-skip test's MagicMock agent short-circuited before
appending cancelled-tool messages — assert len(messages)==3 got 0.
Production is unaffected: run_conversation resets the flag to False
explicitly at turn start (conversation_loop.py:~1028).
- codex app-server sibling path: surface a WARNING (was silent debug) when
the projected-message flush fails — same bug class as the main fix, but
codex output has already streamed so fail-closed and agent_persisted=False
are both wrong here (#860/#42039 duplicate-write hazard); loud durability
gap logging instead.
- map session_persistence_failed in _format_turn_completion_explanation so
the user sees an actionable reason instead of 'The request failed:
unknown error' + explainer test.
- contributors/emails mapping for elco@thedaoist.gg (attribution CI).
Three helpers each re-derived part of the same decision: which backend
serves profile P, and does its REST path need a `?profile=` scope.
profileUsesPrimaryBackend answered the first half, pathWithGlobalRemoteProfile
answered the second, and ensureBackend re-checked globalRemoteActive() around
both. Splitting one table across three predicates is how the global-remote
case ended up registering reapable pool entries for a backend it never owned.
resolveProfileBackendRoute() states the four routes in one place and returns
the backend, the descriptor scope, and whether the path needs a query
parameter. The call sites read the answer instead of recomputing it.
One behavior change falls out: `hermes:api` now passes the primary profile
through, so the primary no longer sends itself a redundant `?profile=<self>`
on a global remote that already serves it.
Electron pre-installs its own uncaughtException listener and only warns on
unhandled rejections, so a main-process fault usually leaves the app running
with the reason on stderr — which nothing captures when the app is launched
from Finder or the Start menu. The fault never reaches desktop.log, so it is
absent from `hermes debug share` and the user can only describe symptoms.
Record both to desktop.log and flush synchronously, since a fault that does
prove fatal leaves no chance for the batched async flush. Five loadURL calls
were also unhandled, each able to leave a blank window with no explanation
anywhere the user can send us; they now name the surface that failed.
Co-authored-by: Rodrigo Fernandez <rod@nxtlevel.dev>
A pooled backend entry pointing at a remote host has no child process, so
the 'exit' handler that clears a dead local backend never fires. The
renderer's 60s keepalive touch also spares it from the idle reaper. Nothing
was left to retire the descriptor, so once the host went away the pool kept
serving it and every profile bound to that host stayed broken until restart.
Pooled remote descriptors now share the primary's liveness policy: probed on
the same revalidate tick, keyed per base URL, and dropped only after the
same consecutive-failure limit, so the next ensureBackend() rebuilds.
Co-authored-by: Rodrigo Fernandez <rod@nxtlevel.dev>
Keep non-primary profiles that inherit the app-global remote on the primary connection descriptor instead of creating processless pool entries that the idle reaper repeatedly removes.
Preserve per-profile remote overrides and local pooled backends, and cover the routing policy with behavioral tests.
Co-authored-by: Rodrigo Fernandez <rodrigo@nxtlevelsaas.com>
The change-detector test asserted 'hasattr(signal, "SIGPIPE")' in source.
PR #72677 replaced inline hasattr+signal.signal with _install_signal()
which does the same guard via getattr internally. Update the test to
accept either form.
Importing tui_gateway.entry from a worker thread raised
'ValueError: signal only works in main thread of the main interpreter'
because signal.signal() was called unconditionally at import time.
On the Desktop/WebSocket path, server._build() runs in a daemon thread and
does 'from tui_gateway.entry import ensure_mcp_discovery_started' as the
first import of entry (entry.main() is never run there), which crashed and
aborted MCP discovery startup — every session then lost its MCP servers
(e.g. Dart-mcp) with ClosedResourceError.
signal handlers are process-global, so installing them only when the module
is first imported in the main thread is sufficient; importing from a worker
thread becomes a safe no-op.
Fixes#72667
The stale-.pyc bug class (#6207, #60242, live WhatsApp report: gateway
ImportError 'cannot import name parse_model_flags_detailed' after /update)
has one shared shape: the checkout's .py files change while __pycache__
retains bytecode from the previous revision, and a later process trusts
the stale .pyc.
Update-time clears can never fully close this class: 'hermes update'
always executes the PRE-pull updater code, so hardening added to it takes
effect one update late — and a manual 'git pull' never runs the updater
at all.
Class fix:
- Launch-time guard in main(): compare the checkout fingerprint (cheap
file reads via _read_git_revision_fingerprint, no git subprocess)
against a .bytecode-fingerprint stamp; sweep __pycache__ once when they
diverge. Covers manual pulls, old updaters, ZIP restores — every entry
point (CLI, gateway service, desktop backend) passes through main().
- Record the stamp at all three update-time clear sites (git path pre-
install, git path post-install, ZIP path) so a normal update never
triggers a redundant launch sweep.
- Sibling site: 'hermes plugins update' + dashboard plugin update now
clear __pycache__ under the plugin dir after git pull (plugin trees
live outside the repo guard).
E2E validated: reproduced the stale-pyc shadowing (same-size same-mtime
source swap → old symbol wins in a fresh process), confirmed the launch
sweep restores the new symbol, no-ops when the checkout is unchanged,
and re-sweeps on the next pull. Sabotage-tested: reverting the sweep
fails 4 of the new tests.
Refresh update-sensitive modules before lazy backend refresh so an in-place git update does not keep using pre-pull module objects when newly pulled code imports fresh helpers.
Constraint: Issue #60242 reports post-update lazy backend refresh importing stale hermes_constants after a large Windows update.
Rejected: Only clearing __pycache__ earlier | the update process can still hold old modules in sys.modules.
Confidence: high
Scope-risk: narrow
Directive: Keep update-time lazy refresh guarded against in-process code skew after git pull.
Tested: ./.venv/bin/python -m pytest tests/hermes_cli/test_update_autostash.py::test_cmd_update_reloads_runtime_modules_before_lazy_refresh tests/hermes_cli/test_update_autostash.py::test_reload_updated_runtime_modules_restores_new_hermes_constants_symbol -q
Tested: ./.venv/bin/python -m pytest tests/hermes_cli/test_update_autostash.py -q
Tested: ./.venv/bin/python -m ruff check hermes_cli/main.py tests/hermes_cli/test_update_autostash.py
Tested: ./.venv/bin/python -m py_compile hermes_cli/main.py tests/hermes_cli/test_update_autostash.py
Tested: git diff --check
Not-tested: Windows v0.14.0 to v0.18.0 end-to-end update.
* fix(desktop): front workspace pane when navigating sidebar routes
Capabilities/Messaging/Artifacts (and other full-page workspace routes) rendered their content correctly on navigate(), but the workspace pane itself stayed behind an active session tile in the pane tree if one was focused. Clicking the same sidebar item again after switching to a session tile appeared to do nothing.
Session switches already call revealTreePane('workspace') + noteActiveTreeGroup(null) to front the pane (store/session-states.ts), but sidebar/keybind/command-palette navigation to full-page routes did not have an equivalent.
Add navigateToWorkspacePage() helper in routes.ts that wraps navigate() and fronts the workspace pane for non-overlay, non-chat views. Apply it at all 5 call sites that navigate to full-page workspace routes: selectSidebarItem, use-keybinds (nav.skills/messaging/artifacts), command palette 'go', Command Center's onNavigateRoute, and cold-start route restore.
Fixes#72602
* test(desktop): cover workspace pane reveal in selectSidebarItem
Adds a regression test for #72602: navigating to a sidebar route now calls navigate() and fronts the workspace pane (noteActiveTreeGroup(null) + revealTreePane('workspace')).
Mocks @/components/pane-shell/tree/store to assert the calls without depending on real pane-tree DOM state.
* fix(desktop): classify router targets by pathname, not the raw target
Every route classifier reasoned about the full navigation target, so a
query put them on the wrong branch: `/skills?tab=mcp` failed the reserved
path check and fell through to the session-id parser as the session
`skills?tab=mcp`, which made `appViewForPath` report Capabilities as a
chat. The command palette reaches Capabilities exclusively through those
targets, and Settings redirects old `/settings?tab=mcp` deep links there.
Strip the query and hash once, up front. Session ids are percent-encoded
by `sessionRoute`, so `?`/`#` can only ever start a query or a hash.
* fix(desktop): front the workspace pane from the router location
Capabilities, Messaging and Artifacts render inside the `workspace` pane,
so navigating to one has to bring that pane to the front of its group.
Nothing did. With the main zone parked on a session tile, the route and
the page content changed behind the tile and the click looked dead until
the app restarted. Session switches already front the pane in
`store/session-states.ts`; pages had no equivalent.
Front it from the router location, in the effect that already mirrors
`$workspaceIsPage`. One place decides, so every entry point inherits it:
sidebar, keybinds, palette, Command Center, contributed statusbar and
titlebar `to` targets, back/forward, and cold-start restore — which no
longer needs its own call.
`navigateToWorkspacePage` stays for the one case the location can't see:
hitting Capabilities while already on `/skills` leaves the location
untouched, so no effect fires and only an imperative reveal brings the
page back.
* test(desktop): cover the workspace-page reveal bug class
Both layers, and the classification underneath them: a page route fronts
the pane whether it carries a query or not, moving between two pages
fronts it again even though `$workspaceIsPage` never changes, contributed
routes count, and chat and overlay targets leave the tab alone.
---------
Co-authored-by: alelpoan <alelpoan@proton.me>
claude-fable-5 is a Mythos-class reasoning model (1M context, 128K output,
adaptive thinking per anthropic_adapter.py) but was missing from the
_REASONING_STALE_TIMEOUT_FLOORS table. Without a floor entry it got the
default 180s stale timeout (300s with context scaling), which is too short
for fable-5's thinking phase on large contexts.
Each stale kill bumped the cross-turn circuit breaker streak; after 5
consecutive kills _check_stale_giveup() fired immediately (elapsed: 0.00s),
aborting all calls with "Provider has been unresponsive for 5 consecutive
stale attempts." Users with 191K-token contexts hit this reliably.
Add ("claude-fable", 600) — deep-reasoning tier alongside o1/deepseek-r1/
nemotron-3-ultra. The claude-fable slug matches claude-fable-5 and future
variants via the existing right-anchor regex.
Both layers, and the classification underneath them: a page route fronts
the pane whether it carries a query or not, moving between two pages
fronts it again even though `$workspaceIsPage` never changes, contributed
routes count, and chat and overlay targets leave the tab alone.
Capabilities, Messaging and Artifacts render inside the `workspace` pane,
so navigating to one has to bring that pane to the front of its group.
Nothing did. With the main zone parked on a session tile, the route and
the page content changed behind the tile and the click looked dead until
the app restarted. Session switches already front the pane in
`store/session-states.ts`; pages had no equivalent.
Front it from the router location, in the effect that already mirrors
`$workspaceIsPage`. One place decides, so every entry point inherits it:
sidebar, keybinds, palette, Command Center, contributed statusbar and
titlebar `to` targets, back/forward, and cold-start restore — which no
longer needs its own call.
`navigateToWorkspacePage` stays for the one case the location can't see:
hitting Capabilities while already on `/skills` leaves the location
untouched, so no effect fires and only an imperative reveal brings the
page back.
Every route classifier reasoned about the full navigation target, so a
query put them on the wrong branch: `/skills?tab=mcp` failed the reserved
path check and fell through to the session-id parser as the session
`skills?tab=mcp`, which made `appViewForPath` report Capabilities as a
chat. The command palette reaches Capabilities exclusively through those
targets, and Settings redirects old `/settings?tab=mcp` deep links there.
Strip the query and hash once, up front. Session ids are percent-encoded
by `sessionRoute`, so `?`/`#` can only ever start a query or a hash.
Adds a regression test for #72602: navigating to a sidebar route now calls navigate() and fronts the workspace pane (noteActiveTreeGroup(null) + revealTreePane('workspace')).
Mocks @/components/pane-shell/tree/store to assert the calls without depending on real pane-tree DOM state.
Capabilities/Messaging/Artifacts (and other full-page workspace routes) rendered their content correctly on navigate(), but the workspace pane itself stayed behind an active session tile in the pane tree if one was focused. Clicking the same sidebar item again after switching to a session tile appeared to do nothing.
Session switches already call revealTreePane('workspace') + noteActiveTreeGroup(null) to front the pane (store/session-states.ts), but sidebar/keybind/command-palette navigation to full-page routes did not have an equivalent.
Add navigateToWorkspacePage() helper in routes.ts that wraps navigate() and fronts the workspace pane for non-overlay, non-chat views. Apply it at all 5 call sites that navigate to full-page workspace routes: selectSidebarItem, use-keybinds (nav.skills/messaging/artifacts), command palette 'go', Command Center's onNavigateRoute, and cold-start route restore.
Fixes#72602
Reshapes the salvaged fix so it recovers the `tsc: not found` build without
mis-diagnosing healthy trees.
The npm config forcing is dropped. It rested on the premise that an inherited
`npm_config_omit=dev` can beat the `--include=dev` CLI flag; npm resolves
command-line flags above environment config and filters `omit` by `include`,
so the flag already wins. Verified on npm 10 and 11.5.1: with
`npm_config_omit=dev` set and `--include=dev` passed, devDependencies install.
`npm_config_production` is worse than redundant — npm 9 removed it, so setting
it prints `npm warn config production Use --omit=dev instead.` on every
install.
The readiness probe now reads every root `npm run build` searches, not just
the workspace root. npm links a package's bin shims under the package itself
when it owns its lockfile (#42973), so a root-only check called a working tree
broken: it forced a redundant install, skipped the build entirely, and made
`hermes web` exit 1 on a layout that builds fine today.
The pre-build probe is gone with it. A build that works is never second-guessed
and no filesystem introspection gates it; recovery is driven by the failure
instead. When the build cannot resolve tsc or vite, we reinstall (visibly) and
retry before the generic delayed retry, which otherwise just reruns the same
command and leaves the stale dist in place forever. The lockfile-hash skip
still invalidates on an incomplete toolchain so the next update repairs itself.
Also drops the branches that changed behavior based on whether a test mock was
installed, and replaces the mock-call-count tests with real temp trees covering
both hoisting layouts, the Windows shim extensions, and each shell's wording of
an unresolvable binary.
Co-authored-by: Gerardo Camorlinga Jr. <gercamjr.dev@gmail.com>
hermes update could report a successful workspace npm ci while
devDependencies were omitted (production/omit-dev env leakage), leaving
tsc/vite unlinked. The subsequent web UI build then failed with
`tsc: not found` and fell back to a stale dashboard dist.
Force npm_config_include=dev on install, verify toolchain shims after
workspace install (and refuse to hash-cache a half tree), repair/reinstall
when node_modules exists without tsc/vite, prepend workspace .bin dirs to
PATH, and retry the build after a tsc/vite-not-found failure.
