Salvage of #35508 (@dchenk), rebased onto current main. Resolved the
tests/tools/test_stage2_hook_puid_pgid.py conflict (kept both the
envdir-creation regression test on main and the new config-migration
tests).
Docker image upgrades replace code under $INSTALL_DIR but preserve
$HERMES_HOME on the mounted volume, so the persisted config.yaml never
received the schema migrations that non-Docker `hermes update` runs
(#35406). This adds scripts/docker_config_migrate.py, invoked from
stage2-hook after first-boot seeding and before gateway services start:
it backs up config.yaml + .env, runs migrate_config(interactive=False),
and honors HERMES_SKIP_CONFIG_MIGRATION=1 for manual control.
Also fixes a latent bug in check_config_version(): it called load_config()
which deep-merges DEFAULT_CONFIG, so a legacy config with no raw
_config_version falsely reported as already-current. It now reads the raw
on-disk file so legacy configs are correctly detected for migration.
Differs from #35508 as submitted (Option B cleanup): dropped the
`_config_version` line added to cli-config.yaml.example and removed the
accompanying test_cli_config_example_declares_latest_version change-detector
test. The example is a copy-template and has no business asserting a schema
version; check_config_version() reads the user's real config.yaml, not the
example. This removes a second sync point that drifts on every version bump.
Closes#35508. Fixes#35406.
Co-authored-by: Dmitriy Cherchenko <17372886+dchenk@users.noreply.github.com>
`docker run --user $(id -u):$(id -g)` was a tini-era trick to make
container-written files match the host user. Under s6-overlay it no longer
works: the bootstrap (UID remap, volume + build-tree chown, config seeding)
needs root, and the baked image dirs (/opt/data, /opt/hermes/.venv, ui-tui,
node_modules) are owned by the hermes build UID (10000). A pinned arbitrary
UID can't write them, so the runtime fails with EACCES on a bind mount or
hard-crashes on a named volume (Docker inits the volume from the image as
10000; the non-root start can't even `cd /opt/data`, and the profile
reconciler dies with PermissionError on gateway_state.json).
Detect that start early in both the cont-init hook (stage2-hook.sh) and the
CMD wrapper (main-wrapper.sh) and fail fast with actionable guidance pointing
at the supported path: root start + HERMES_UID/HERMES_GID (or the PUID/PGID
aliases), which remaps the hermes user and chowns the volume — the same
host-UID-matching outcome --user was used for, without breaking s6.
The guard fires only when the current UID is neither root NOR the hermes UID.
This preserves the supported non-root start from #34648/#34837 (running with
`--user 10000:10000`, i.e. pinned to the hermes UID itself), which is
unaffected — only the arbitrary-UID variant that #34837 never actually made
writable is rejected.
Verified live across five scenarios (built image, bind + named volume):
arbitrary --user on bind -> rejected with guidance, hermes does not run;
arbitrary --user on named volume -> guidance shown, no raw 'can't cd' crash;
--user 10000:10000 -> boots; root + HERMES_UID=4242 remap -> boots, guard not
tripped; default root start -> boots. Pre-fix control reproduces the raw
PermissionError + 'can't cd' crash with no guidance.
`hermes mcp add --auth header` built `Authorization: Bearer ${MCP_X_API_KEY}`
and passed it straight to the discovery probe without interpolation, so the
probe sent the literal placeholder and auth-requiring servers (e.g. n8n)
returned 401. Runtime tool loading worked because `_load_mcp_config()`
interpolates, but the four CLI probe call sites (add/test/login/configure)
all used unresolved config.
Resolve `${ENV}` inside `_probe_single_server` via a new
`_resolve_mcp_server_config()` (load_hermes_dotenv + _interpolate_env_vars),
mirroring runtime loading. This covers all four call sites, not just add.
Also strip a leading `Bearer ` from pasted tokens before saving to
`MCP_*_API_KEY`, so a token pasted with the prefix doesn't produce
`Bearer Bearer <jwt>` (also a 401).
Reported with a precise root-cause analysis in #37792.
Co-authored-by: ThyFriendlyFox <116314616+ThyFriendlyFox@users.noreply.github.com>
The runtime resolver reads model.base_url from config and ignores the
OPENAI_BASE_URL env var, so a self-hosted endpoint could not be configured
from the GUI. Two changes enable it:
- POST /api/model/set accepts an optional base_url and persists it as
model.base_url when provider=custom (still clearing stale base_url for
hosted providers).
- POST /api/providers/validate now returns the model ids a custom endpoint
advertises at /v1/models, so the GUI can auto-pick a default without
asking the user to type a model name.
Refs desktop onboarding "Local / custom endpoint" bug.
The stage2 hook gates the recursive chown of the build trees under
$INSTALL_DIR (.venv, ui-tui, node_modules) so a HERMES_UID/PUID remap
leaves them writable by the new runtime UID — needed for lazy_deps
'uv pip install' of platform extras (#15012, #21100) and the TUI esbuild
rebuild into ui-tui/dist (#28851).
#35027 folded that chown under the $HERMES_HOME ownership check
('stat $HERMES_HOME != hermes_uid'). But 'usermod -u <new> hermes'
re-chowns the hermes home dir ($HERMES_HOME == /opt/data) to the new UID
as a side effect, so after any remap that stat is already satisfied and
needs_chown is false — silently skipping the build-tree chown on the
common PUID/NAS path. The venv stays owned by the build-time UID (10000),
so lazy installs and TUI rebuilds fail with EACCES.
Probe the build trees directly instead: chown only when /opt/hermes/.venv
is not already owned by the runtime hermes UID. Independent of
$HERMES_HOME ownership, idempotent across restarts.
Verified live: built the image, booted with HERMES_UID/HERMES_GID on a
fresh named volume, confirmed .venv/ui-tui/node_modules end up owned by
the remapped UID and 'uv pip install' into the venv succeeds; confirmed
the recursive chown fires once and is skipped on restart.
Dashboard plugins (kanban, hermes-achievements) read
window.__HERMES_SESSION_TOKEN__ directly and hand-assembled WebSocket
URLs with ?token=. That works in loopback/--insecure mode but is
rejected on OAuth-gated deployments, where the session token is absent
and _ws_auth_ok only accepts single-use ?ticket= auth. The result was
401s on plugin REST calls and 1008/403 on the kanban live-events WS
whenever the dashboard ran behind OAuth (e.g. hosted Fly agents).
Make the plugin SDK the single sanctioned auth surface:
- web/src/lib/api.ts: add authedFetch() (raw Response for FormData
uploads / blob downloads, token-or-cookie auth, no throw / no 401
redirect) and buildWsUrl() (assembles a ws(s):// URL with the correct
auth param for the active mode — fresh single-use ticket in gated
mode, token in loopback).
- web/src/plugins/registry.ts: expose authedFetch, buildWsUrl,
buildWsAuthParam, and sdkVersion on window.__HERMES_PLUGIN_SDK__;
add SDK_CONTRACT_VERSION.
- web/src/plugins/sdk.d.ts: hand-authored typed contract for the
plugin SDK + registry globals (single source of truth for the
Window declarations).
- plugins/kanban + hermes-achievements dist bundles: stop reading the
session token directly; route uploads/downloads through
SDK.authedFetch and the live-events WS through SDK.buildWsUrl.
- plugins/kanban plugin_api.py: _ws_upgrade_authorized() delegates the
/events WS upgrade to the canonical web_server._ws_auth_ok gate, so
it transparently accepts loopback token / gated ticket / internal
credential and can never drift from core auth again.
- tests: guard test asserting no plugin dist reads
__HERMES_SESSION_TOKEN__ directly; kanban gated-ticket WS test.
Verified live on a gated staging Fly agent: kanban /events upgrades
101 with a minted ticket (ticket_len=43, ws_auth_ok=True) where the
old code got 403.
Fireworks/Mistral reject HTTP 400 'Extra inputs are not permitted, field:
messages[N].tool_calls[M].extra_content' on any session whose history
contains prior Gemini tool calls. Gemini 3 thinking models attach
extra_content (thought_signature) to tool_calls; it survived to the wire
because the sanitize paths only stripped call_id/response_item_id.
Strip extra_content from the outgoing wire copy in both sanitize paths
(ChatCompletionsTransport.convert_messages + _sanitize_tool_calls_for_strict_api),
but gate it on the target model: keep extra_content for Gemini-family
targets (the thought_signature MUST be replayed or Gemini 400s), strip it
for everyone else — including non-Gemini models that inherit a stale Gemini
signature earlier in a mixed-provider session. Native Gemini is unaffected
(GeminiNativeClient bypasses these paths).
Original stored history is never mutated (only the per-call copy).
Fixes#17986.
The TUI hardcoded --max-old-space-size=8192. V8 is not cgroup-aware, so in a
Docker/k8s container capped below ~9-10GB the heap grows past the container
limit and the cgroup OOM-killer SIGKILLs the Node parent BEFORE V8's own heap
monitor fires. SIGKILL runs no JS handler, writes no [tui-parent] breadcrumb,
and closes the gateway child's stdin — the user sees only a bare gateway
'stdin EOF'. Complements #38224 (trail-text cap), which reduced pressure but
left the 8GB-vs-container mismatch in place.
- _read_cgroup_memory_limit(): read cgroup v2 (memory.max) then v1
(memory.limit_in_bytes); handle 'max', the v1 unlimited sentinel, blank/zero,
and >=1PB as unconstrained.
- _resolve_tui_heap_mb(): unconstrained -> 8192; constrained -> 75% of the
cgroup limit (headroom for non-heap RSS + the Python child sharing the
cgroup), floored at 1536MB, never above 8192.
- NODE_OPTIONS block uses the sized value; still respects a user-supplied
--max-old-space-size.
Net: V8 now GCs/exits gracefully (onCritical breadcrumb fires) instead of being
reaped silently. Display/transport only — no agent context or behavior change.
Tests: tests/hermes_cli/test_tui_heap_sizing.py (20 tests).
The stash/restore cycle in the update path was observed to clobber
freshly-pulled source files (apps/desktop/ deletion -> Vite
'[UNRESOLVED_ENTRY] Cannot resolve entry module index.html'). On a
managed clone the user never edits the source tree, so any 'dirty' state
is pure git artifact (CRLF renormalization, npm lockfile churn, files
left behind when a directory was deleted upstream such as
apps/bootstrap-installer/). Stashing that and re-applying it after a pull
is fragile and unnecessary.
- hermes update (hermes_cli/main.py): on a non-fork (managed) clone,
discard working-tree dirt via reset --hard HEAD + clean -fd instead of
stash/apply. Forks keep the stash machinery so intentional edits
survive. Also pin core.autocrlf=false on Windows so the dirt is never
created (mirrors install.ps1 #38239).
- install.sh: replace the update-path stash/restore dance with a hard
reset to origin/<branch>; the installer is a managed-only entry point.
- install.sh + install.ps1 desktop stage: prefer 'npm ci' (wipes and
reinstalls node_modules from the lockfile) over bare 'npm install',
which can report 'up to date' against a stale marker while node_modules
is empty -- leaving tsc unresolved so 'npm run pack' fails.
Tests: managed clone cleans instead of stashing; fork still stashes;
existing stash tests force the stash path explicitly.
The native Hindsight memory provider lazy-installs hindsight-client into
/opt/hermes/.venv at first use (tools/lazy_deps.py: memory.hindsight).
That venv lives inside the immutable image layer, not the mounted
/opt/data volume, so the dependency is wiped on every container recreate
/ image update. After an update, profile config still points at Hindsight
and the Hindsight server is healthy, but recall/retain fails with:
ModuleNotFoundError: No module named 'hindsight_client'
The manual workaround (uv pip install hindsight-client inside the running
container) doesn't survive the next recreate, and pip-install-into-.venv
is not an officially supported durable Docker workflow.
Fix: add --extra hindsight to the image's uv sync line, same pattern as
the --extra anthropic/bedrock/azure-identity providers (#30504) and
--extra messaging (#24698) — bake the optional dependency into the build
layer so it survives container recreate. The pyproject [hindsight] pin
(hindsight-client==0.6.1) already matches tools/lazy_deps.py and uv.lock,
so this is a pure additive --extra with no lockfile churn.
Verified: 'uv sync --frozen --no-install-project --extra hindsight'
against the committed uv.lock installs hindsight-client 0.6.1 and the
module imports cleanly.
Adds a regression test (mirrors test_dockerfile_preinstalls_gateway_
messaging_dependencies) so a future Dockerfile cleanup can't silently
drop the extra.
Self-review of #38465 surfaced three real items:
1. SystemExit escape (defense): `_login_nous` raises SystemExit(130)/(1) on
cancel/failure. The logged-out login path inside `_model_flow_nous` catches
it, but the expired-session re-login path (main.py) only catches Exception,
so a Ctrl-C during re-auth could propagate past `_run_portal_one_shot` and
kill the CLI. Add SystemExit to the portal handler so all cancel/abort cases
end with the graceful 'Setup cancelled / retry later' message.
2. Doc sweep: the model-pick step was only added to the bare-`hermes portal`
prose. Propagate it to the surfaces describing `hermes setup --portal`
behavior that still omitted model selection:
- `--portal` argparse help (main.py)
- nous-portal.md intro + the numbered 'what it does' step list (EN + zh-Hans)
- run-hermes-with-nous-portal.md 'default model after setup --portal' line,
which was now contradictory (there's a picker, not a forced default) (EN + zh)
3. Test coverage: add parametrized regression test asserting the portal handler
swallows KeyboardInterrupt / EOFError / SystemExit (returns None, no escape).
Note on 'Skip (keep current)': delegating to _model_flow_nous means picking
Skip preserves the prior provider instead of force-switching to nous — this is
intentional and matches quick setup exactly; docs now say 'sets Nous as your
provider (when you pick a model)' rather than unconditionally.
`hermes portal` / `hermes setup --portal` previously logged in and set
provider=nous but left the model UNSELECTED (blank -> runtime default) and
never showed a picker — unlike the first-time quick setup, which runs the
model picker.
Route `_run_portal_one_shot` through `_model_flow_nous` — the exact same
routine quick setup (`_run_first_time_quick_setup`) and `hermes model` -> Nous
use. It handles both the logged-out path (device-code OAuth, which picks a
model internally) and the logged-in path (curated Nous model picker), then
offers the Tool Gateway opt-in and sets provider=nous. Net effect: `hermes
portal` now offers a model picker every time and is a true single-command
collapse of quick setup's Nous step.
Removes the hand-rolled auth_add_command + manual provider write + separate
Tool Gateway prompt (now a single source of truth). Re-syncs the in-memory
config from disk afterward so a caller's later save_config can't clobber the
model/provider written by the login flow.
Docs (CLI help, portal_cli docstrings, nous-portal EN + zh-Hans) updated to
mention model selection. New regression test asserts `_run_portal_one_shot`
delegates to `_model_flow_nous`.
Verified live: `hermes portal` now shows the 27-model curated picker, 'Skip
(keep current)' preserves prior provider/model.
`hermes portal` (no subcommand) now runs the one-shot Nous Portal onboarding
— OAuth login, switch provider to Nous, offer Tool Gateway — identical to
`hermes setup --portal` and the human-readable alias for
`hermes auth add nous --type oauth` (which still works).
The prior status default moves to `hermes portal info`; `status` is kept as a
hidden back-compat alias. `open`/`tools` subcommands are unchanged.
User-facing hints and docs (status.py, conversation_loop 401 guidance,
SystemPage, README, website docs + zh-Hans) now point at `hermes portal` /
`hermes portal info`. `--manual-paste` references keep the explicit auth
command since `hermes portal` does not expose that flag.
* fix(packaging): ship locales/ i18n catalogs in wheel, sdist, and Nix
locales/ is a bare data dir (no __init__.py), invisible to packages.find
and package-data. Sealed installs (pip wheel, Nix store venv) dropped it,
so gateway/CLI commands rendered raw i18n keys like
gateway.reset.header_default.
- pyproject: [tool.setuptools.data-files] locales = ["locales/*.yaml"] (wheel)
- MANIFEST.in: graft locales (sdist)
- agent/i18n._locales_dir: env override -> source -> sysconfig data scheme
- nix/hermes-agent.nix: copy locales into the store + set HERMES_BUNDLED_LOCALES
as defense-in-depth. The wheel's data-files already materialize into the
uv2nix venv, so resolution works with no env var; the override pins the
store path against a future uv2nix change that could drop data-files.
- tests: metadata regression, wheel + sdist build-install smoke tests, and a
bundled-locales flake check that verifies BOTH the wrapper override and the
env-var-less data-files path. Smoke test wired into CI.
Closes#23943, #27632, #35374.
Supersedes #23966, #27716, #30261, #33841, #35429, #35494, #35735, #36697.
* test: cap locale e2e timeout, tighten catalog count guard
The two wheel/sdist e2e tests inherit the global --timeout=30 from
addopts; a cold-CI run (isolated build env + venv create + network pip
install) can plausibly exceed it. Add @pytest.mark.timeout(300) so they
don't ride the unit-test budget and flake intermittently.
Also assert the shipped catalog count equals len(SUPPORTED_LANGUAGES)
instead of a hardcoded >=16 floor, so the guard self-updates and trips
on a single dropped catalog (not just a fully-empty graft).
Avoid stale WebSocket events from an old reconnect attempt flipping the gateway state after a newer socket opens. Also limit session-search dedupe to compression edges so branch-specific hits still open the branch instead of collapsing to the parent.
Four related desktop session-management bugs:
- Pins lost until refresh: pinned sessions are joined against the
paginated in-memory session list, so a pinned chat that aged off the
most-recent page got evicted on the next refresh (every message.complete
triggers one) and the Pinned section went empty. mergeWorkingSessions ->
mergeSessionPage now also preserves pinned rows (matched by live id or
lineage root). Pin id checks in the chat header, command center, and
delete/archive are normalized to the durable sessionPinId so pins survive
auto-compression.
- Stuck on "Starting Hermes" after sleep: macOS sleep drops the renderer
WebSocket; nothing reconnected on wake so the composer stayed disabled.
The gateway boot hook now auto-reconnects with backoff on close/error and
on wake signals (powerMonitor resume/unlock-screen IPC, window online,
visibilitychange). connect() gains an open timeout so a hung reconnect
can't deadlock in 'connecting'. Composer placeholder distinguishes
"Reconnecting to Hermes" from a cold start.
- Loses chats from itself: the same hard-replace that dropped pins also
dropped loaded sessions; mergeSessionPage keeps them.
- Multiple copies/branches in search: /api/sessions/search deduped only by
raw session_id, so compression segments and branches surfaced as separate
hits. It now dedupes by lineage root and returns the live compression tip,
matching the session_search tool's behavior.
Assert _exec_schtasks passes an explicit encoding and errors="replace" to
subprocess.run, and that _schtasks_encoding falls back to utf-8 when the
locale lookup is empty or raises (#38172).
* fix(dashboard): clamp PTY resize dimensions for WSL2 winsize garbage
WSL2 reports columns=131072, rows=1 from a broken winsize probe. The
dashboard /chat tab forwards xterm.js dimensions through PtyBridge.resize(),
which packs them as unsigned short via struct.pack. 131072 > 65535 raised
struct.error — uncaught (only OSError was handled) — breaking the resize
path and leaving the TUI laid out for a one-row, absurdly-wide screen, which
surfaces as blank/disappearing text.
Clamp cols/rows to a sane [1, 2000]x[1, 1000] range before packing.
Non-finite/non-integer probes fall back to the minimum so nothing can reach
struct.pack and raise.
* test(dashboard): de-flake pub/events broadcast test
test_pub_broadcasts_to_events_subscribers round-tripped a frame through
two nested Starlette TestClient WebSocket portals within a 10s wall-clock
budget. Under heavy parallel CI load a starved ASGI thread occasionally
blew that budget even though the server logic is correct, producing
intermittent 'broadcast not received within 10s' failures.
Drive _broadcast_event directly under asyncio with fake subscribers
instead. Same fan-out contract (verbatim delivery to every subscriber on
the channel, nothing to other channels), zero scheduling surface. Runs in
~0.3s, deterministic across 10 consecutive runs.
* fix(tui): save TUI /save snapshots under Hermes home with system prompt
The TUI gateway's session.save RPC wrote hermes_conversation_<ts>.json to
the workspace/project CWD via os.path.abspath(...) and only exported model
and messages. This diverged from the classic CLI /save (which writes under
the Hermes profile home) and from the dashboard save (which includes the
system prompt).
Write the snapshot under get_hermes_home()/sessions/saved/ and include
system_prompt, session_id, and session_start so the TUI export matches the
CLI and dashboard behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tui): prefer agent.session_start for /save export; assert it in test
Address review feedback: derive session_start from the agent's session_start
datetime (matching the classic CLI export) and fall back to the gateway
session's created_at only when unavailable. Assert session_start in the
regression test.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(desktop): enrich profiles dashboard and de-dupe channel env vars
Add active-profile switching, role descriptions (manual + auto-generate
via the auxiliary LLM), per-profile model selection, and gateway-running
/ distribution badges to the GUI Profiles page. New profile creation
gains clone-all, optional description and model assignment.
Hide messaging-platform credentials (channel_managed) from the Keys/Env
page since the Channels page is the canonical surface for them, and
relabel the trimmed "messaging" category as "Gateway".
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(desktop): address review feedback on profiles/env changes
- ProfilesPage: scope the action-menu outside-click handler to the menu's
own container via a ref so opening one card's menu no longer leaves
others open.
- EnvPage: route the "Gateway" label and hint through i18n
(t.common.gateway / gatewayHint) instead of hard-coded English, with an
English fallback for untranslated locales.
- web_server: only report description_auto=true when auto-generation
actually succeeded.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(desktop): address second-round review on profiles
- ProfilesPage: treat describe-auto success by null-checking the
description and trust the response's description_auto flag instead of
assuming true; disable the model-editor Save button unless the selected
choice resolves to a real /api/model/options entry (avoids silent
no-op saves).
- tests: cover the new profile endpoints (active get/set + 404,
description round-trip + 404, model round-trip + 400 validation, and
describe-auto success/failure contracts).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(desktop): more profiles review fixes (toggles, races, tests)
- ProfilesPage: use the canonical `active` returned by setActiveProfile;
make the SOUL/description/model action-menu items toggle their editor
closed when already open; guard description save/auto-describe against
stale responses via an activeDescRequest ref so a late reply can't
clobber a different open editor.
- tests: assert /api/env channel_managed classification matches
_channel_managed_env_keys().
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(doctor): detect + repair stale HERMES_MAX_ITERATIONS .env ghost shadowing config.yaml
hermes doctor now flags when ~/.hermes/.env carries a HERMES_MAX_ITERATIONS
value that disagrees with agent.max_turns in config.yaml, and 'hermes doctor
--fix' removes the stale .env line so config.yaml is authoritative. 'hermes
config show' surfaces the same drift inline under Max turns.
The setup wizard stopped dual-writing this value, but users who edited only
config.yaml from a pre-fix install keep a .env ghost. The gateway bridge
normally overrides it at startup, but if the bridge bails on any earlier
config-parse error the ghost silently wins — config says 400 while the
gateway activity line reads N/90.
The detector reads the .env FILE directly (load_env), not get_env_value/
os.environ, since the startup bridge may already have overwritten os.environ
with the config value.
Closes#17534.
* fix(config): stop offering HERMES_MAX_ITERATIONS as an editable env var
Removes HERMES_MAX_ITERATIONS from OPTIONAL_ENV_VARS so the dashboard env
editor (PUT /api/env) and any env-var prompt no longer let a user write it
to .env — which would recreate the stale ghost that shadows config.yaml's
agent.max_turns (issue #17534). The iteration budget is configured only via
config.yaml; the env var stays a read-only backward-compat fallback in the
gateway/CLI, never a promoted write target.
Regression test asserts it is absent from OPTIONAL_ENV_VARS.
CI slice 3 caught that tests/test_transform_tool_result_hook.py monkeypatches
invoke_hook but not has_hook, so the new has_hook("transform_tool_result")
gate skipped the emit and the transform never ran. Stub has_hook=True in the
shared _run_handle_function_call helper whenever a custom invoke_hook is
supplied (the test intends hooks to fire). The no-hook-registered test keeps
the real has_hook=False path — that's the gate's intended behavior.
The salvaged observer contract gated the API-request hot path on has_hook()
but left the per-tool emit ungated: every tool call ran result-field
derivation + payload dict build + invoke_hook dispatch even with zero
plugins registered.
- _emit_post_tool_call_hook now short-circuits on has_hook("post_tool_call")
and derives status/error fields lazily (after the gate, only when a
listener will consume them). status defaults to None -> derived; explicit
blocked/cancelled callers still pass status through.
- transform_tool_result emit (pre-existing hook) likewise gated on
has_hook(); skips _tool_result_observer_fields when no listener.
- Removed the now-redundant _tool_result_observer_fields pre-computation at
the three ok-path call sites (model_tools, agent_runtime_helpers,
tool_executor) — the helper derives them, so the no-listener path costs
one dict lookup and the call sites shrink.
- Tests: stub has_hook=True where payload correctness is asserted; add a
no-listener regression proving post_tool_call/transform_tool_result emit
is skipped when nothing is registered.
The salvaged PR incidentally stripped a trailing blank line from two
unrelated test files (test_file_tools_cwd_resolution.py,
test_tool_search.py). Restore them to keep the salvage diff scoped to
the observability feature.
Adds backend-neutral observer hooks for plugins: session, turn, API
request, tool, approval, and subagent lifecycle events with stable
correlation IDs (session_id, task_id, turn_id, api_request_id,
tool_call_id, parent/child subagent ids). Extends VALID_HOOKS with
api_request_error and subagent_start.
Hot path is zero-cost when no plugin subscribes: has_hook()/presence
checks gate all payload construction, request payloads are returned
by reference when no middleware rewrites, and the sanitized response
payload no longer embeds raw response objects.
Bundles the optional NeMo-Relay observability plugin
(plugins/observability/nemo_relay) as an in-repo consumer of the new
hooks, peer to the existing langfuse plugin. Fails open when the
optional nemo-relay package is not installed.
Authored-by: Bryan Bednarski <bbednarski@nvidia.com>
Salvaged from #29722 onto current main.
A kanban worker that exhausted its retries purely on a provider rate
limit / quota wall (e.g. opencode-go's 5-hour window) exited with code 1.
The dispatcher counted that as a crash, and with DEFAULT_FAILURE_LIMIT=2
two quota-wall hits permanently blocked the card. Fanning out many
workers against one shared quota made this routine.
Now a rate-limited worker exits with EX_TEMPFAIL (75); the dispatcher
classifies that as a 'rate_limited' exit, releases the task back to
'ready' WITHOUT incrementing consecutive_failures (the breaker can't trip
on a transient throttle), and the respawn guard defers the next attempt
on a cooldown (default 5min, HERMES_KANBAN_RATE_LIMIT_COOLDOWN_SECONDS)
until the quota window clears. Genuine crashes still count and trip the
breaker as before. The 120s Retry-After cap is unchanged — no worker
parks for hours holding a slot.
- conversation_loop.py: surface failure_reason in the exhaustion return
- cli.py: kanban worker picks exit 75 on rate_limit/billing failure
- kanban_db.py: rate_limited exit kind, no-count requeue, cooldown guard
A heavy --tui session (browser snapshots, large tool outputs) silently
OOM-killed the Node parent within minutes — closing the gateway child's
stdin, which the user saw only as a bare "gateway exited" / stdin EOF.
CLI was immune. Root cause: each completed tool's verbose trail line
embedded up to 16KB of result_text, persisted in transcript Msg.tools[]
for the whole session and rendered EXPANDED by default, so an Ink
render-node tree was built for every one of up to 800 messages at once.
That tree blew past Node's heap at a few hundred MB — far below the 2.5GB
memory-monitor exit threshold, so the death was never even attributed.
- text.ts: persisted verbose tool-trail blocks now cap to a small preview
(VERBOSE_TRAIL_MAX_CHARS=800/12 lines), not the 16KB live-render budget.
Retained trail strings drop ~17x (12.2MB -> 0.7MB at 800 msgs); the live
streaming tail still uses the larger LIVE_RENDER budget.
- tui_gateway/server.py: lower the gateway-side verbose text cap to match
(1KB/16 lines) so we stop shipping output the TUI no longer renders.
- memoryMonitor.ts: derive critical/high thresholds from the real V8 heap
ceiling (~88%/70%) instead of the hardcoded 2.5GB that killed the process
at 31% of an 8GB ceiling; add a one-shot onWarn early-warning on fast
sub-threshold heap growth so the next such death is diagnosable, not silent.
- entry.tsx: wire onWarn to a crash-log breadcrumb + stderr line.
Full tool output is unchanged in the agent context and SQLite session — this
is display/transport only, no behavior or context change.
Fixes#34095. Related #27282.
Tests: ui-tui text + new memoryMonitor suites (33 pass), python verbose-cap
guard (5 pass); full ui-tui suite shows no new failures vs pristine main.
E2E repro confirms the retention drop.
The dashboard's update button ran 'hermes update' immediately with no
preview. Now the System page shows whether an update is available and
asks the user to confirm before applying it.
- New GET /api/hermes/update/check: reports install method, current
version, and commits-behind (via banner.check_for_updates, 6h-cached;
?force=1 busts the cache). Soft-fails to behind=null on network error;
marks docker/nix/homebrew as can_apply=false with the out-of-band cmd.
- System page: update-status badge on the Hermes version row (latest /
N behind), a Check-for-updates button, and an Update-now button that
opens a ConfirmDialog showing the commit count before POST /api/hermes/
update fires. Cached status loads with the rest of the page.
- Docs + 5 endpoint tests (git/up-to-date/docker/soft-failure + auth gate).
The Electron desktop app writes boot failures, backend spawn output, and
Python tracebacks to HERMES_HOME/logs/desktop.log, but debug-share only
captured agent/errors/gateway — so desktop boot issues never made it into
shared debug reports.
- logs.py: register desktop -> desktop.log (enables 'hermes logs desktop')
- debug.py: capture desktop snapshot, add to summary report, upload full
desktop.log in 'share', update privacy notice
- gateway /debug inherits the desktop tail via collect_debug_report()
- main.py + docs: help text and log-name table (also adds missing gui row)
- tests: desktop seed in fixture, new report test, three_pastes -> four_pastes
get_mcp_status() treated every non-connected server as a failure, so a
server configured with enabled: false rendered as red '— failed' in the
startup banner even though it was intentionally off. Add a 'disabled'
field derived from the enabled flag and render disabled servers dim as
'— disabled' instead.
Follow-up to the salvaged contributor commit:
- Underscore→hyphen tolerance now emits a resolvable token. Previously
the detect set accepted the hyphenated variant but emit returned the
raw token, so '!set_home' produced '/set_home' which the dispatcher
could not resolve. Now emits '/set-home'. Aliases are left as-is — the
gateway dispatcher canonicalizes them itself.
- Fix dead skill-command branch: skill command keys are stored
slash-prefixed (e.g. '/arxiv') in get_skill_commands(), but the check
compared the bare token, so '!arxiv' never normalized. Now compares
the '/candidate' form, making skill aliases (e.g. !gif-search) work.
- Re-run bang normalization after Matrix reply-fallback stripping so a
quoted reply whose content is a bang command reaches command parity
with the slash form.
- Replace silent 'except Exception: pass' with logger.debug(exc_info=True).
- Add AUTHOR_MAP entry for @nepenth.
Tests: +5 (underscore-alias, skill-command branch, quoted-reply bang +
slash parity). 162 Matrix tests pass.
Follow-up to Ben's PR #37892. Adds a TestInternalCredential block to
test_dashboard_auth_ws_tickets.py exercising the mint-once stability,
multi-use, unminted-rejection, empty-value, wrong-value, reset-and-remint,
and ticket-store-independence branches directly (previously only covered
indirectly via _ws_auth_ok, which left the unminted and empty-value
branches unexercised).
Also corrects the consume_internal_credential docstring: the returned
identity dict is discarded by the current _ws_auth_ok caller (which only
needs the boolean outcome), so the prior 'carry it into its session log'
wording over-promised.
The embedded-TUI PTY child attaches to two server-internal WebSockets:
/api/ws (its primary JSON-RPC gateway backend) and /api/pub (the event
sidecar). Both URLs are built server-side in web_server.py and handed to
the child via its environment.
In OAuth-gated mode (auth_required=true, every hosted Fly agent), _ws_auth_ok
unconditionally rejects the legacy ?token=<_SESSION_TOKEN> path — a leaked
session token must not grant WS access once the gate is engaged. But
_build_gateway_ws_url() still only emitted ?token=, with no gated-mode
branch (its sibling _build_sidecar_url had been given a ticket branch; the
gateway-url builder was missed). So the TUI child's /api/ws upgrade was
rejected 4401 -> 'gateway websocket connection failed' -> 'gateway startup
timeout', leaving the embedded chat unusable on every gated deployment.
A single-use 30s browser ticket is the wrong shape for this link: the child
reads its attach URL once at startup and reuses it on every reconnect, and
on a slow cold boot it may not dial within the TTL. (_build_sidecar_url's
own docstring already flagged this fragility.)
Fix: add a process-lifetime, multi-use internal credential to
dashboard_auth.ws_tickets (internal_ws_credential / consume_internal_credential),
minted once per process and NEVER injected into the SPA — it only leaves the
process via a spawned child's env, so browser-side XSS can't read it, and a
leak grants no more than a ticket already does. _ws_auth_ok accepts it via
?internal= in gated mode only. Both _build_gateway_ws_url and
_build_sidecar_url now use it, so the child can reconnect both sockets.
Loopback / --insecure behavior is unchanged (still ?token=).
Needs review: touches _ws_auth_ok + dashboard_auth (core auth surface).
The embedded dashboard Chat tab dies on hosted images with a 502 /
"[session ended]": the PTY child's `hermes --tui` spawn runs a runtime
`npm install` that fails.
Root cause: the root package-lock.json describes the WHOLE npm monorepo
workspace set (root + web + ui-tui + apps/*), but the image only installs
root/web/ui-tui — apps/* (the desktop app) is never `npm install`ed here, and
its deps hoist into the shared root node_modules. So the actualized
node_modules permanently disagrees with the canonical lock,
`_tui_need_npm_install()` returns True on every launch, and the runtime
`npm install` it triggers (a) can never converge against the partial monorepo
and (b) races itself across concurrent /api/pty connections -> ENOTEMPTY ->
the launcher `sys.exit(1)`s, the slow install blows past Fly's WS-upgrade
window -> 502 -> the browser shows "[session ended]".
Fix: set `ENV HERMES_TUI_DIR=/opt/hermes/ui-tui` so `_make_tui_argv` takes the
prebuilt-bundle fast path (`node --expose-gc /opt/hermes/ui-tui/dist/entry.js`)
and never reaches the install check — exactly the nix/packaged-release path
the launcher was designed for. The bundle is already built at Layer 8
(`ui-tui && npm run build`); this just tells the launcher to use it.
Verified on a freshly-built image: HERMES_TUI_DIR is set, the prebuilt
dist/entry.js is present, `_make_tui_argv` resolves to the prebuilt node
invocation (no npm), and `docker run ... --tui` no longer prints
"npm install failed". New regression guard: tests/docker/test_tui_prebuilt_bundle.py.
A separate launcher hardening (make _tui_need_npm_install tolerant of
partial-monorepo installs) is tracked independently; this Docker-side fix
resolves the hosted-chat symptom on its own.
Area: docker (Dockerfile + tests/docker).
On a fresh volume there is no gateway_state.json, so the boot reconciler
(cont-init.d/02-reconcile-profiles) registers the gateway-default s6 slot
but leaves it down — it only auto-starts when the last recorded state was
"running". A freshly-provisioned container therefore comes up with the
gateway down until something starts it (e.g. the dashboard's start button).
Add a generic, first-boot-only env-seed in stage2-hook.sh (which runs
before 02-reconcile-profiles): when HERMES_GATEWAY_BOOTSTRAP_STATE=running
and no gateway_state.json exists yet, seed {"gateway_state":"running"} so
the reconciler brings the supervised slot up on the very first boot.
This mirrors the existing HERMES_AUTH_JSON_BOOTSTRAP pattern: it seeds the
same state file the reconciler already consults, guarded by [ ! -f ] so
persisted runtime state always wins on later boots (a deliberately-stopped
gateway stays stopped across restarts). Only the literal "running" is
honoured (the sole value in the reconciler's _AUTOSTART_STATES).
Generic container contract — no host-specific code. Useful to any
orchestrator that provisions a blank volume and wants the gateway up from
first boot (the supervised gateway/dashboard already work on such hosts;
only the first-boot autostart was missing because the CLI lifecycle
commands can't drive the s6 layer when container self-detection misses).
Adds a shell-level contract test and documents the env var.
Generalises #37747. The WS Origin guard (_ws_host_origin_is_allowed) only
trusted the packaged Electron app's non-web origin (file:// / null / app://)
when the bind was NOT OAuth-gated. The packaged Hermes Desktop renderer loads
over file://, so when it drives a remote OAuth-gated gateway its /api/ws
upgrade was rejected with HTTP 403 even though _ws_auth_ok had already
validated the single-use ?ticket= one line earlier.
This guard runs only AFTER _ws_auth_ok has accepted the WS credential, which
is the real auth boundary in every mode:
* loopback bind -> legacy dashboard session token
* non-loopback --insecure -> legacy session token (Tailscale / LAN, #37747)
* OAuth-gated public bind -> single-use, 30s-TTL, identity-bound ?ticket=
A non-web origin can only come from a native client; a DNS-rebinding attack
always arrives from an http(s) origin and is still match-checked against the
bound host. So once the upstream credential check has passed, the Origin guard
adds nothing for a non-web origin. Collapsed the loopback/non-gated special
cases to 'return True' for non-web origins.
http(s) origins keep the strict same-host check, so browser DNS-rebinding
defence is unchanged.
Tests: gated file:///null/app:// now asserted ALLOWED; cross-site http(s)
still rejected on gated and loopback binds; #37747's loopback and
non-loopback-insecure cases retained. 37/37 test_dashboard_auth_ws_auth +
test_web_server_host_header pass.
The Browser Automation and Text-to-Speech provider pickers listed the paid
"Nous Subscription" gateway row first, so on a fresh install the menu cursor
defaulted to index 0 (Nous). Pressing Enter selected it and ran the inline
Nous Portal device-code login — walking users into a paid offering they
never chose.
Reorder both provider lists so the free, no-key local backend is index 0
(Local Browser / Microsoft Edge TTS). Users who already configured Nous are
unaffected: _detect_active_provider_index still resolves their active row
first, so the cursor lands on Nous (now index 1) for them.
Reported by Javier via Kujila.
Add `display.interface` config key so users can make the modern TUI the
default for bare `hermes` / `hermes chat` without exporting HERMES_TUI=1 in
every shell. Default stays "cli" to preserve current behavior.
Add a `--cli` flag (mirrors `--tui`) so an explicit invocation can force the
classic prompt_toolkit REPL even when `display.interface: tui` is configured.
Precedence (highest first): `--cli` > `--tui`/`HERMES_TUI=1` > config
`display.interface` > classic REPL. Two resolvers enforce it:
* `_resolve_use_tui(args)` — the args-aware resolver used by `cmd_chat`
and the Termux fast-TUI path (uses full load_config()).
* `_wants_tui_early(argv)` — a dependency-free early resolver used by
mouse-residue suppression and the Termux fast paths, which run before
argparse / hermes_cli.config are importable (minimal cached YAML read).
Both `--cli` and `--tui` are registered via `_inherited_flag`, so they are
carried across self-relaunch automatically.
- config: add display.interface ("cli" default), bump _config_version 25->26.
The generic missing-field migration + load_config() deep-merge seed the key
for existing configs; no bespoke migration block needed.
- docs: document --cli flag and display.interface in cli-commands.md and
the TUI user guide.
- tests: new test_default_interface_resolution.py covering resolver
precedence at every layer, early resolver edge cases (missing/garbage
config), parser flags, and relaunch inheritance.
Pin #37718: the inherit plist must grant audio-input, every device.*
entitlement on the main app must also be inherited by the Helper/Setup
processes, and both entitlement files must stay valid plists.
Electron's chrome-sandbox helper must be root:root 4755 on Linux or the
sandboxed renderer aborts before the desktop app starts. The existing
installer only searched for macOS .app bundles, so a successful Linux
build was reported as missing.
Changes:
- Add _desktop_linux_sandbox_fixup() to hermes_cli/main.py, called
before launching a packaged desktop app on Linux.
- Use lstat() + S_ISREG check to reject symlinks — chown/chmod on a
symlink target would set SUID on an arbitrary path.
- Update install.sh to recognize Linux unpacked artifacts and configure
chrome-sandbox with proper error handling (the original PR silently
ignored chown/chmod failures).
- Add regression tests: normal fixup flow, symlink rejection, and
already-configured skip path.
Closes#37529 (rebased, merge conflicts resolved, copilot review
feedback addressed).
Production code now uses ensure_uv()/update_managed_uv() from
managed_uv.py instead of shutil.which("uv") directly. Tests that
patched shutil.which to control uv availability no longer controlled
the actual code path, causing CI failures.
Add an autouse _patch_managed_uv fixture to test_update_autostash.py
and test_uv_tool_update.py (matching the existing fixture in
test_cmd_update.py). The fixture makes managed_uv functions delegate
to shutil.which so existing test patches flow through naturally.
