The static-prefix reconstruction pattern (build_system_prompt_parts ->
['stable'] -> startswith gate -> fail-open) existed in three copies:
session restore (conversation_loop), compression keep-prompt path
(conversation_compression), and the new failover redecoration helper.
Hoist it into agent/system_prompt.reconstruct_static_prefix and call it
from all three sites.
Also memoize failed rebuilds per stored prompt (_static_rebuild_failed_for):
the redecoration chokepoint runs at the top of every retry attempt, and a
persistent stable-tier mismatch (restored session whose SOUL.md/skills
changed since save) would otherwise re-run the full prompt build — SOUL.md,
context files, memory I/O — on every attempt of every API call for the
life of the session. A legitimately changed stored prompt retries once.
strip_anthropic_cache_control flattened ANY pure-text multi-part content
list with a separator-less join. Decoration only ever produces a single
text part or the 2-part [static, volatile] system split; organic
multi-part text (merged user turns, imported transcripts) got word-jammed
and parts carrying extra keys (citations) were silently dropped — on the
common no-failover path, since redecoration runs on every attempt.
Restrict the flatten to the exact decoration-produced shapes and make
marker removal copy-on-write on part dicts (the per-call message copy is
shallow, so parts alias the persistent history).
Add strip_anthropic_cache_control coverage and policy-change cases
(cache-off→on, on→off, native→envelope, MoA guidance outside marker)
that TestSyncFailoverPreservesCacheDecoration did not exercise.
try_activate_fallback refreshes the cache policy flags for the new
provider, but the retry loop reused the primary's decorated api_messages.
Cache-off→cache-on shipped zero breakpoints; cache-on→cache-off left
stale markers. Strip and re-render at each retry attempt (same chokepoint
as reasoning-echo reapply), peel/rebase MoA guidance so the last marker
stays off the turn-varying block, and rebuild the static system prefix
when caching becomes active mid-turn (#72626).
compress_context now runs on a daemon pool worker thread (via
run_compress_context_with_progress_timeout). The session id rotation
updates hermes_logging._session_context (a threading.local) on the
WORKER thread, not the caller thread. After the wrapper returns,
propagate self.session_id back to the caller's logging context so
subsequent log lines carry the rotated id (#34089).
Fixes CI failure in test_compression_logging_session_context.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
PR #72425 added getattr(agent, '_incremental_persistence_failed', False)
checks at the top of execute_tool_calls_{sequential,concurrent,segmented}.
A bare MagicMock auto-creates a truthy value for any attribute access,
so the interrupt-skip test's MagicMock agent short-circuited before
appending cancelled-tool messages — assert len(messages)==3 got 0.
Production is unaffected: run_conversation resets the flag to False
explicitly at turn start (conversation_loop.py:~1028).
- codex app-server sibling path: surface a WARNING (was silent debug) when
the projected-message flush fails — same bug class as the main fix, but
codex output has already streamed so fail-closed and agent_persisted=False
are both wrong here (#860/#42039 duplicate-write hazard); loud durability
gap logging instead.
- map session_persistence_failed in _format_turn_completion_explanation so
the user sees an actionable reason instead of 'The request failed:
unknown error' + explainer test.
- contributors/emails mapping for elco@thedaoist.gg (attribution CI).
Three helpers each re-derived part of the same decision: which backend
serves profile P, and does its REST path need a `?profile=` scope.
profileUsesPrimaryBackend answered the first half, pathWithGlobalRemoteProfile
answered the second, and ensureBackend re-checked globalRemoteActive() around
both. Splitting one table across three predicates is how the global-remote
case ended up registering reapable pool entries for a backend it never owned.
resolveProfileBackendRoute() states the four routes in one place and returns
the backend, the descriptor scope, and whether the path needs a query
parameter. The call sites read the answer instead of recomputing it.
One behavior change falls out: `hermes:api` now passes the primary profile
through, so the primary no longer sends itself a redundant `?profile=<self>`
on a global remote that already serves it.
Electron pre-installs its own uncaughtException listener and only warns on
unhandled rejections, so a main-process fault usually leaves the app running
with the reason on stderr — which nothing captures when the app is launched
from Finder or the Start menu. The fault never reaches desktop.log, so it is
absent from `hermes debug share` and the user can only describe symptoms.
Record both to desktop.log and flush synchronously, since a fault that does
prove fatal leaves no chance for the batched async flush. Five loadURL calls
were also unhandled, each able to leave a blank window with no explanation
anywhere the user can send us; they now name the surface that failed.
Co-authored-by: Rodrigo Fernandez <rod@nxtlevel.dev>
A pooled backend entry pointing at a remote host has no child process, so
the 'exit' handler that clears a dead local backend never fires. The
renderer's 60s keepalive touch also spares it from the idle reaper. Nothing
was left to retire the descriptor, so once the host went away the pool kept
serving it and every profile bound to that host stayed broken until restart.
Pooled remote descriptors now share the primary's liveness policy: probed on
the same revalidate tick, keyed per base URL, and dropped only after the
same consecutive-failure limit, so the next ensureBackend() rebuilds.
Co-authored-by: Rodrigo Fernandez <rod@nxtlevel.dev>
Keep non-primary profiles that inherit the app-global remote on the primary connection descriptor instead of creating processless pool entries that the idle reaper repeatedly removes.
Preserve per-profile remote overrides and local pooled backends, and cover the routing policy with behavioral tests.
Co-authored-by: Rodrigo Fernandez <rodrigo@nxtlevelsaas.com>
The change-detector test asserted 'hasattr(signal, "SIGPIPE")' in source.
PR #72677 replaced inline hasattr+signal.signal with _install_signal()
which does the same guard via getattr internally. Update the test to
accept either form.
Importing tui_gateway.entry from a worker thread raised
'ValueError: signal only works in main thread of the main interpreter'
because signal.signal() was called unconditionally at import time.
On the Desktop/WebSocket path, server._build() runs in a daemon thread and
does 'from tui_gateway.entry import ensure_mcp_discovery_started' as the
first import of entry (entry.main() is never run there), which crashed and
aborted MCP discovery startup — every session then lost its MCP servers
(e.g. Dart-mcp) with ClosedResourceError.
signal handlers are process-global, so installing them only when the module
is first imported in the main thread is sufficient; importing from a worker
thread becomes a safe no-op.
Fixes#72667
The stale-.pyc bug class (#6207, #60242, live WhatsApp report: gateway
ImportError 'cannot import name parse_model_flags_detailed' after /update)
has one shared shape: the checkout's .py files change while __pycache__
retains bytecode from the previous revision, and a later process trusts
the stale .pyc.
Update-time clears can never fully close this class: 'hermes update'
always executes the PRE-pull updater code, so hardening added to it takes
effect one update late — and a manual 'git pull' never runs the updater
at all.
Class fix:
- Launch-time guard in main(): compare the checkout fingerprint (cheap
file reads via _read_git_revision_fingerprint, no git subprocess)
against a .bytecode-fingerprint stamp; sweep __pycache__ once when they
diverge. Covers manual pulls, old updaters, ZIP restores — every entry
point (CLI, gateway service, desktop backend) passes through main().
- Record the stamp at all three update-time clear sites (git path pre-
install, git path post-install, ZIP path) so a normal update never
triggers a redundant launch sweep.
- Sibling site: 'hermes plugins update' + dashboard plugin update now
clear __pycache__ under the plugin dir after git pull (plugin trees
live outside the repo guard).
E2E validated: reproduced the stale-pyc shadowing (same-size same-mtime
source swap → old symbol wins in a fresh process), confirmed the launch
sweep restores the new symbol, no-ops when the checkout is unchanged,
and re-sweeps on the next pull. Sabotage-tested: reverting the sweep
fails 4 of the new tests.
Refresh update-sensitive modules before lazy backend refresh so an in-place git update does not keep using pre-pull module objects when newly pulled code imports fresh helpers.
Constraint: Issue #60242 reports post-update lazy backend refresh importing stale hermes_constants after a large Windows update.
Rejected: Only clearing __pycache__ earlier | the update process can still hold old modules in sys.modules.
Confidence: high
Scope-risk: narrow
Directive: Keep update-time lazy refresh guarded against in-process code skew after git pull.
Tested: ./.venv/bin/python -m pytest tests/hermes_cli/test_update_autostash.py::test_cmd_update_reloads_runtime_modules_before_lazy_refresh tests/hermes_cli/test_update_autostash.py::test_reload_updated_runtime_modules_restores_new_hermes_constants_symbol -q
Tested: ./.venv/bin/python -m pytest tests/hermes_cli/test_update_autostash.py -q
Tested: ./.venv/bin/python -m ruff check hermes_cli/main.py tests/hermes_cli/test_update_autostash.py
Tested: ./.venv/bin/python -m py_compile hermes_cli/main.py tests/hermes_cli/test_update_autostash.py
Tested: git diff --check
Not-tested: Windows v0.14.0 to v0.18.0 end-to-end update.
* fix(desktop): front workspace pane when navigating sidebar routes
Capabilities/Messaging/Artifacts (and other full-page workspace routes) rendered their content correctly on navigate(), but the workspace pane itself stayed behind an active session tile in the pane tree if one was focused. Clicking the same sidebar item again after switching to a session tile appeared to do nothing.
Session switches already call revealTreePane('workspace') + noteActiveTreeGroup(null) to front the pane (store/session-states.ts), but sidebar/keybind/command-palette navigation to full-page routes did not have an equivalent.
Add navigateToWorkspacePage() helper in routes.ts that wraps navigate() and fronts the workspace pane for non-overlay, non-chat views. Apply it at all 5 call sites that navigate to full-page workspace routes: selectSidebarItem, use-keybinds (nav.skills/messaging/artifacts), command palette 'go', Command Center's onNavigateRoute, and cold-start route restore.
Fixes#72602
* test(desktop): cover workspace pane reveal in selectSidebarItem
Adds a regression test for #72602: navigating to a sidebar route now calls navigate() and fronts the workspace pane (noteActiveTreeGroup(null) + revealTreePane('workspace')).
Mocks @/components/pane-shell/tree/store to assert the calls without depending on real pane-tree DOM state.
* fix(desktop): classify router targets by pathname, not the raw target
Every route classifier reasoned about the full navigation target, so a
query put them on the wrong branch: `/skills?tab=mcp` failed the reserved
path check and fell through to the session-id parser as the session
`skills?tab=mcp`, which made `appViewForPath` report Capabilities as a
chat. The command palette reaches Capabilities exclusively through those
targets, and Settings redirects old `/settings?tab=mcp` deep links there.
Strip the query and hash once, up front. Session ids are percent-encoded
by `sessionRoute`, so `?`/`#` can only ever start a query or a hash.
* fix(desktop): front the workspace pane from the router location
Capabilities, Messaging and Artifacts render inside the `workspace` pane,
so navigating to one has to bring that pane to the front of its group.
Nothing did. With the main zone parked on a session tile, the route and
the page content changed behind the tile and the click looked dead until
the app restarted. Session switches already front the pane in
`store/session-states.ts`; pages had no equivalent.
Front it from the router location, in the effect that already mirrors
`$workspaceIsPage`. One place decides, so every entry point inherits it:
sidebar, keybinds, palette, Command Center, contributed statusbar and
titlebar `to` targets, back/forward, and cold-start restore — which no
longer needs its own call.
`navigateToWorkspacePage` stays for the one case the location can't see:
hitting Capabilities while already on `/skills` leaves the location
untouched, so no effect fires and only an imperative reveal brings the
page back.
* test(desktop): cover the workspace-page reveal bug class
Both layers, and the classification underneath them: a page route fronts
the pane whether it carries a query or not, moving between two pages
fronts it again even though `$workspaceIsPage` never changes, contributed
routes count, and chat and overlay targets leave the tab alone.
---------
Co-authored-by: alelpoan <alelpoan@proton.me>
claude-fable-5 is a Mythos-class reasoning model (1M context, 128K output,
adaptive thinking per anthropic_adapter.py) but was missing from the
_REASONING_STALE_TIMEOUT_FLOORS table. Without a floor entry it got the
default 180s stale timeout (300s with context scaling), which is too short
for fable-5's thinking phase on large contexts.
Each stale kill bumped the cross-turn circuit breaker streak; after 5
consecutive kills _check_stale_giveup() fired immediately (elapsed: 0.00s),
aborting all calls with "Provider has been unresponsive for 5 consecutive
stale attempts." Users with 191K-token contexts hit this reliably.
Add ("claude-fable", 600) — deep-reasoning tier alongside o1/deepseek-r1/
nemotron-3-ultra. The claude-fable slug matches claude-fable-5 and future
variants via the existing right-anchor regex.
Both layers, and the classification underneath them: a page route fronts
the pane whether it carries a query or not, moving between two pages
fronts it again even though `$workspaceIsPage` never changes, contributed
routes count, and chat and overlay targets leave the tab alone.
Capabilities, Messaging and Artifacts render inside the `workspace` pane,
so navigating to one has to bring that pane to the front of its group.
Nothing did. With the main zone parked on a session tile, the route and
the page content changed behind the tile and the click looked dead until
the app restarted. Session switches already front the pane in
`store/session-states.ts`; pages had no equivalent.
Front it from the router location, in the effect that already mirrors
`$workspaceIsPage`. One place decides, so every entry point inherits it:
sidebar, keybinds, palette, Command Center, contributed statusbar and
titlebar `to` targets, back/forward, and cold-start restore — which no
longer needs its own call.
`navigateToWorkspacePage` stays for the one case the location can't see:
hitting Capabilities while already on `/skills` leaves the location
untouched, so no effect fires and only an imperative reveal brings the
page back.
Every route classifier reasoned about the full navigation target, so a
query put them on the wrong branch: `/skills?tab=mcp` failed the reserved
path check and fell through to the session-id parser as the session
`skills?tab=mcp`, which made `appViewForPath` report Capabilities as a
chat. The command palette reaches Capabilities exclusively through those
targets, and Settings redirects old `/settings?tab=mcp` deep links there.
Strip the query and hash once, up front. Session ids are percent-encoded
by `sessionRoute`, so `?`/`#` can only ever start a query or a hash.
Adds a regression test for #72602: navigating to a sidebar route now calls navigate() and fronts the workspace pane (noteActiveTreeGroup(null) + revealTreePane('workspace')).
Mocks @/components/pane-shell/tree/store to assert the calls without depending on real pane-tree DOM state.
Capabilities/Messaging/Artifacts (and other full-page workspace routes) rendered their content correctly on navigate(), but the workspace pane itself stayed behind an active session tile in the pane tree if one was focused. Clicking the same sidebar item again after switching to a session tile appeared to do nothing.
Session switches already call revealTreePane('workspace') + noteActiveTreeGroup(null) to front the pane (store/session-states.ts), but sidebar/keybind/command-palette navigation to full-page routes did not have an equivalent.
Add navigateToWorkspacePage() helper in routes.ts that wraps navigate() and fronts the workspace pane for non-overlay, non-chat views. Apply it at all 5 call sites that navigate to full-page workspace routes: selectSidebarItem, use-keybinds (nav.skills/messaging/artifacts), command palette 'go', Command Center's onNavigateRoute, and cold-start route restore.
Fixes#72602
Reshapes the salvaged fix so it recovers the `tsc: not found` build without
mis-diagnosing healthy trees.
The npm config forcing is dropped. It rested on the premise that an inherited
`npm_config_omit=dev` can beat the `--include=dev` CLI flag; npm resolves
command-line flags above environment config and filters `omit` by `include`,
so the flag already wins. Verified on npm 10 and 11.5.1: with
`npm_config_omit=dev` set and `--include=dev` passed, devDependencies install.
`npm_config_production` is worse than redundant — npm 9 removed it, so setting
it prints `npm warn config production Use --omit=dev instead.` on every
install.
The readiness probe now reads every root `npm run build` searches, not just
the workspace root. npm links a package's bin shims under the package itself
when it owns its lockfile (#42973), so a root-only check called a working tree
broken: it forced a redundant install, skipped the build entirely, and made
`hermes web` exit 1 on a layout that builds fine today.
The pre-build probe is gone with it. A build that works is never second-guessed
and no filesystem introspection gates it; recovery is driven by the failure
instead. When the build cannot resolve tsc or vite, we reinstall (visibly) and
retry before the generic delayed retry, which otherwise just reruns the same
command and leaves the stale dist in place forever. The lockfile-hash skip
still invalidates on an incomplete toolchain so the next update repairs itself.
Also drops the branches that changed behavior based on whether a test mock was
installed, and replaces the mock-call-count tests with real temp trees covering
both hoisting layouts, the Windows shim extensions, and each shell's wording of
an unresolvable binary.
Co-authored-by: Gerardo Camorlinga Jr. <gercamjr.dev@gmail.com>
hermes update could report a successful workspace npm ci while
devDependencies were omitted (production/omit-dev env leakage), leaving
tsc/vite unlinked. The subsequent web UI build then failed with
`tsc: not found` and fell back to a stale dashboard dist.
Force npm_config_include=dev on install, verify toolchain shims after
workspace install (and refuse to hash-cache a half tree), repair/reinstall
when node_modules exists without tsc/vite, prepend workspace .bin dirs to
PATH, and retry the build after a tsc/vite-not-found failure.
* fix: Branch in new chat loses the question and the branched session (#issue)
- session.branch on the backend now accepts a count param to truncate
the parent history to the clicked message, instead of always forking
the entire transcript. Also returns stored_session_id/messages/info
so the frontend has parity with session.create's response shape.
- branchCurrentSession (open live chat) now slices history from 0
instead of from the clicked message index, so the question preceding
an assistant reply is no longer dropped when branching.
- forkBranch now calls session.branch (not session.create) when
branching an open live chat, since session.create only persists a DB
row lazily on first prompt - a branched chat that nobody types into
never got saved, and vanished as 'session not found' on the next
app restart. branchStoredSession (branching from the sidebar, no
live runtime) keeps using session.create as before.
* test: cover session.branch count truncation and open-chat branching
- backend: assert session.branch with a count param only persists the
first N messages of the live history to the new session.
- frontend: BranchHarness now exposes branchCurrentSession; assert
branching an open chat from a middle message calls session.branch
with the parent session id and the correct trimmed count, instead of
session.create.
* fix: Branch button is a dead no-op inside a branched chat tile
session-tile.tsx wired onBranchInNewChat to () => undefined for
tiled/branched sessions (nested branching isn't supported there), but
the button in AssistantMessage's action bar rendered unconditionally
regardless of whether a real handler was supplied. The button looked
clickable but silently did nothing, with no visual feedback.
- AssistantMessage now only renders the Branch button when
onBranchInNewChat is actually provided, matching the existing
pattern used for onDismissError/onRestoreToMessage.
- session-tile.tsx no longer passes a no-op handler; the prop is
simply omitted so the button doesn't render in tiles.
- onBranchInNewChat is now optional on ChatViewProps, and the
latestChatActions passthrough wrapper uses the existing
latestOptional helper instead of an unconditional call.
* test: assert Branch button visibility matches handler presence
Adds coverage for the bug #2 fix: renders Thread with and without an
onBranchInNewChat handler and asserts the Branch in new chat button
is shown only when a real handler is supplied, hidden otherwise -
covering both the normal open-chat case and the session-tile
(branched chat) case that used to leave a dead, clickable button.
Update _pre_msg_count after adopting the durable transcript so the
post-compression log reflects the correct pre-adoption message count.
Also use the existing _live_child_id() helper in the updated test
instead of hand-rolled child-id extraction.
Follow-up to #72631.
is_truthy_value(..., default=False) and getattr(..., False) disagreed with
compression.in_place: true from #38763, so partial/failed config loads fell
back into rotation mode and re-armed the pre-lease drift path. Also report
compression.in_place in hermes dump overrides so stale false values are visible.
Busy sessions (memory review / shared session writers) kept outrunning the
in-memory snapshot, so rotation-mode compress aborted every attempt with
"changed before lease acquisition" and surfaced as a fake "No changes from
compression". Adopt the durable transcript and continue compressing instead
of returning the stale snapshot unchanged.
Per hermes-sweeper review on #56671: the fallback exclusion matched any
canonical string starting with "custom" (e.g. "customproxy"), not just
the durable named-custom-provider syntax ("custom" bucket or
"custom:<name>" slugs). Narrow it to an exact/prefix match on that
syntax so unrelated vendor names aren't accidentally exempted from the
openrouter fallback.
Also clarifies the test suite: a properly configured custom:<name>
provider now resolves via resolve_custom_provider before this fallback
is ever reached (added upstream in 9a15fad0d6), so the existing test
was mislabeled as exercising that primary path when it was actually
exercising the fallback-safety-net case (missing/unresolved config
entry). Split into explicit primary-path and fallback-safety-net tests,
plus a regression test for the "customproxy"-style false positive.
_normalize_main_model_assignment() (POST /api/model/set, the endpoint
Desktop's Settings -> Model page uses to persist the main model slot) has
a fallback for a specific analytics bug: an older session row with no
billing_provider sends the model's bare vendor prefix as "provider"
(e.g. "anthropic" from "anthropic/claude-opus-4.6"), so the code detects
an unrecognized provider paired with a slash-bearing model and treats it
as that stray-vendor-prefix case.
Named custom providers are represented as "custom:<name>" slugs
everywhere else in the codebase (runtime_provider.py, model_switch.py),
but _KNOWN_PROVIDER_NAMES only lists the bare "custom" bucket. So picking
a named custom provider (e.g. "custom:litellm", a LiteLLM proxy fronting
Ollama) together with a slash-bearing model ("ollama/glm-5.2") looked
identical to the stray-vendor-prefix case and got silently rewritten to
provider: openrouter in config.yaml on save -- reassigning the provider
entirely, not just mangling the model id.
Exclude anything starting with "custom" from the fallback, matching the
guard the same function already applies later for the actual
normalize_model_for_provider call.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_MATCHING_PREFIX_STRIP_PROVIDERS includes "custom" so that manually typed
config values like "zai/glm-5.1" repair themselves for their matching
native provider. But "custom" is a generic bucket for arbitrary
user-defined endpoints, not a vendor identity -- unlike zai/gemini/xai,
where a matching alias really does mean "this prefix names the same
backend as the target provider."
_PROVIDER_ALIASES maps "ollama" -> "custom", so a model configured as
"ollama/glm-5.2" against a named custom provider (e.g. a LiteLLM proxy
fronting Ollama, which registers its routes as "ollama/<model>") had its
prefix stripped to bare "glm-5.2" -- a name the proxy doesn't recognize.
_strip_matching_provider_prefix now only strips a literal "custom/"
prefix when the target resolves to "custom"; an alias that merely
resolves to custom (ollama) no longer qualifies, since custom has no
vendor identity for it to redundantly repeat.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
