_collect()'s active_platforms pre-filter was derived solely from
self.adapters (the default profile), so a subscription owned by a
secondary profile on a platform the default profile never connected
(e.g. beta owns discord, default has no discord adapter at all) was
skipped before claim_unseen_events_for_sub ever ran. Unlike the
disconnected-adapter path, an unclaimed event is never rewound, so this
was a permanent, silent notification/wake loss — directly contradicting
the point of routing notifications via the owning profile
(c69643026/b225b30d0). Same cross-profile-adapter-lookup bug class the
delivery-side _authorization_adapter chokepoint already guards against,
one gate earlier. The precise per-profile check still runs unchanged at
delivery time, with its existing rewind-on-None safety net.
The dashboard/desktop profile switch was partial — switching to profile X
ran the launch profile's resources in two ways:
1. MCP discovery was gated on the launch profile's config having
mcp_servers. If the launch profile had none, the background thread
never started and zero MCP servers existed for every profile. Fix:
always start discovery and let discover_mcp_tools() handle the
empty-config case.
2. The profile secret scope (.env credentials) was never installed on the
tui_gateway path. get_secret() fell through to os.environ, resolving
secrets from the launch profile instead of the selected one. Fix:
install set_secret_scope(build_profile_secret_scope(...)) alongside
every set_hermes_home_override() call site:
- compute_host.py:_ensure_server_session (build-time)
- server.py:_build (lazy resume)
- server.py:_handle_resume_session (_make_agent scope)
- server.py:_handle_resume_session (_init_session scope)
- server.py:_handle_submit_or_edit (per-turn handler)
Include last_activity_ts in the progress token sampled from each child.
_touch_activity ticks on every streamed chunk ('receiving stream
response'), every tool transition, and API-call start/completion — so a
child mid-stream on a long response is alive even though api_call_count
only advances when the call completes. Same liveness signal as the
compaction inactivity budget (PR #71508): if tokens are flowing it never
dies; staleness is measured from the last streamed token / tool activity
/ API call.
Replace the wall-clock timeout watchdog (from #60234) with progress-based
staleness detection, on by default with zero config:
- The async registry now accepts a progress_fn per dispatch; delegate_task
wires a sampler over the batch's child agents (api_call_count +
current_tool from get_activity_summary()).
- A single monitor thread sweeps running delegations: a child whose
progress token keeps advancing is never touched, no matter how long it
runs. A frozen token past the stale threshold (450s idle / 1200s
in-tool, mirroring the sync-path heartbeat monitor) marks the record
'stalling' and interrupts the child.
- A stalling child that unwinds within the grace window (120s) finalizes
through the NORMAL path, preserving its partial results. One that never
returns is force-finalized with a terminal 'stalled' completion event so
the owning session hears an outcome and the async slot frees.
- Late runner returns after force-finalization are deduped by the
begin/push/finish finalization split (kept from #60234).
Why not a timeout: delegation.child_timeout_seconds defaults to 0 by
deliberate design (DEFAULT_CHILD_TIMEOUT rationale) — a timeout-based
watchdog never arms for default configs, leaving the reported silent-
profile symptom (#60203) unfixed, and when armed it kills legitimately
slow heavy subagents mid-task. Progress detection distinguishes 'wedged
at first API call' from 'grinding through a 2h review'.
Builds on izumi0uu's finalization-atomicity work from #60234.
Async background delegation can leave gateway sessions holding only a dispatched handle when the detached runner wedges before it can return and enqueue a completion. Enforce the configured child timeout in the async registry so the parent observes a terminal timeout event and the async slot is released.
Constraint: Issue #60203 reports long-lived gateway processes with background child delegates that never produce completion events despite child_timeout_seconds being configured.
Rejected: Relying only on _run_single_child timeout handling | it cannot finalize the async registry when the outer runner thread itself never reaches normal completion.
Confidence: high
Scope-risk: narrow
Directive: Keep background delegation completion owned by the async registry whenever detached workers can outlive the caller's immediate control.
Tested: .venv/bin/python -m pytest tests/tools/test_async_delegation.py tests/tools/test_delegate_subagent_timeout_diagnostic.py tests/tools/test_delegate.py -q
Tested: .venv/bin/python -m ruff check tools/async_delegation.py tools/delegate_tool.py tests/tools/test_async_delegation.py
Tested: git diff --check
Not-tested: Multi-day real gateway degradation; covered with deterministic stuck-runner registry tests.
Fixes the boot-storm half of issue #29905: kanban_notify_subs.last_event_id
defaulted to 0, so a subscription created on an already-active task replayed
the task's ENTIRE terminal-event backlog on the next notifier tick. With
many stale subs (27 observed in the report) a gateway boot after downtime
burst 100+ notifications in one go.
add_notify_sub now snaps the cursor to COALESCE(MAX(task_events.id), 0) for
the task inside the same INSERT, so new subscriptions start caught up and
only receive events that occur AFTER subscribing. The gateway slash-command
and kanban-tool auto-subscribe paths run at task creation, where the
snapshot is just the 'created' event — behavior there is unchanged.
Stale fixtures that asserted the literal 0 creation cursor now assert
'cursor unchanged/unclaimed' instead, which is what they actually meant.
Follow-ups from review of salvaged PRs #59278 and #62712:
* test_kanban_notifier_isolates_per_subscription_failure previously
created the good subscription first; list_notify_subs() has no
ORDER BY, so the good delivery happened before the bad claim raised
and the test passed even without the isolation fix. The bad task is
now created first AND a deterministic-order shim forces the failing
subscription to be iterated first, so the test fails on the old
whole-tick-abort behavior.
* New test_notifier_delivers_block_loop_detected_triage_ping: drives a
block_loop_detected event through one notifier tick end-to-end,
asserting the triage ping reaches the adapter and the cursor advances
(the sweeper review of #62712 flagged that only DB-level emission was
tested).
Salvaged from PR #63001 (reduced scope): probe each board with the new
read-only kanban_db.count_notify_subs() before the writable connect(),
so boards with zero subscriptions are never opened writable on the 5s
notifier tick (no schema migration, no WAL/-shm sidecar churn, no
checkpoints).
The PR's machine-global .notifier.lock singleton gate was deliberately
NOT salvaged: a lock-winning default-profile gateway cannot deliver a
secondary profile's subscriptions in standalone-profile deployments
(profile routing fails closed in _authorization_adapter), so the lock
could suppress delivery entirely. The probe captures the per-tick cost
win without that regression.
- honor SendResult(success=False) instead of discarding it, so an adapter
that REPORTS (not raises) a soft send failure — e.g. the Telegram adapter's
"Not connected" mid-reconnect — no longer advances the cursor past an
undelivered event and silently loses the notification. Addresses the
notifier half of #31901.
- add block_loop_detected to the notifier's TERMINAL_KINDS so a task routed to
triage for a human decision (re-blocked past the recurrence limit) actually
pings its subscribers instead of stalling silently.
- raise MAX_SEND_FAILURES 3 -> 12 (~60s at the 5s tick) so a transient
Telegram/API outage does not permanently unsubscribe a live channel now that
reported soft-failures also reach this counter.
- route active-profile-stamped subscriptions via the primary adapter on a
single-profile gateway (self.adapters[platform] when the stamped
notifier_profile equals the active profile). Related to #56802.
Adds test_kanban_notifier_rewinds_claim_on_reported_send_failure asserting a
reported send failure leaves the event unseen (rewound) rather than consumed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The kanban notifier _collect() loop iterates subscriptions without
per-subscription error handling. When claim_unseen_events_for_sub raises
for one subscription (e.g. DB corruption, lock contention), the entire
tick aborts — silently blocking delivery for ALL other subscriptions.
Wrap the per-subscription logic in try/except so one bad subscription
logs a warning and continues to the next, instead of jamming the
entire notifier.
Closes#59269
Both the wake chat-scope salvage (#72191, merged) and the DM-topic
metadata salvage added HERMES_SESSION_CHAT_TYPE plumbing; the rebase
auto-merge kept both copies. Dedupe the ContextVar declaration, _VAR_MAP
entry, set_session_vars parameter/token, and the run.py call-site kwarg,
and prefer the persisted chat_type column with delivery_metadata as the
legacy fallback in the notifier wake path.
Companion fixes from a full dashboard QA pass (every page dogfooded
live), on top of the cherry-picked #31863 header-slot fix:
- ChatPage: harden the header-slot effect further — useLayoutEffect and
never write the slot while inactive, so the handoff commentary and
ownership rule live next to the code.
- LogsPage: level classification used raw substring matching, so INFO
lines carrying 'parse_errors=0' (or paths like errors.log) rendered
red. New unit-tested classifier (web/src/lib/log-classify.ts) anchors
on the hermes_logging level token with a word-boundary fallback.
- Channels API: plugin platforms (irc, ntfy, photon, teams, …) rendered
as nameless title-cased cards ('Irc', 'Ntfy') with empty descriptions.
Two root causes: (1) plugin discovery never ran in the dashboard
server process, so plugin_entries() was empty; (2) Platform enum
pseudo-members claimed plugin ids before the registry could attach
labels. The catalog now discovers plugins explicitly and resolves
plugin metadata first; added descriptions + docs links for bundled
plugin platforms and the msgraph_webhook / whatsapp_cloud / relay
enum members. Regression test sabotage-verified against the old
enum-first ordering.
- Config schema: updates.refresh_cua_driver declared type 'bool'
(schema vocabulary is 'boolean'), so the switch rendered as a text
input holding 'true'.
- Page titles: '/mcp' rendered as 'Mcp' via the naive capitalize
fallback; literal-label table now covers MCP/Files/Channels/Webhooks/
Pairing/System (unit-tested).
- AuthWidget: skip the guaranteed-401 /api/auth/me probe in loopback
mode — every dashboard load logged a console error for nothing.
- Model picker: with no filter, providers that actually have models
float above the wall of '0 models' rows.
- Cron: empty state now carries an actionable Create button.
When embedded chat is enabled, ChatPage renders persistently outside
<Routes> but is initially hidden during the plugin-loading window
(~2-4s). Once plugins finish loading, ChatPage mounts for the first time.
Its header-slot effect had early-return branches for !isActive and !narrow
that actively called setEnd(null). Because the user was on /cron, /models,
/sessions, or any non-chat page, this wiped the action buttons that the
current page had already placed in the header.
Affected pages include:
- Cron page — CREATE button disappears
- Models page — 7D/30D/90D filter buttons disappear
- Sessions page — search box disappears
The fix collapses the two early-return branches into one and removes the
setEnd(null) calls. Now ChatPage only sets end when it actually owns the
slot (isActive && narrow), and lets the normal cleanup handle unmounting.
PageHeaderProvider already clears all slots on pathname change via
useLayoutEffect, so ChatPage's active clearing was redundant and harmful.
Fixes#31862
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy gateway notification subscriptions from parent tasks to child tasks created by create_task(..., parents=...), link_tasks(), and decompose_triage_task().
Inherited subscriptions start at the child's current event cursor, so linking an existing child does not replay pre-link task events.
Repair signalled "reinstall me" by deleting the bootstrap marker. That was
already destructive -- repair is reachable from a transient backend error on a
fully working install -- and it stranded users in first-run setup with no way
back short of hand-writing the marker file.
Carry the intent in an explicit flag instead. Repair forces the next resolve
through the installer and clears itself once the reinstall starts, so a forced
reinstall still works without destroying provenance about how the install was
created.
Closes#72166
Follow-up to @LionGateOS's #72135 salvage:
- Route ensure_mcp_discovery_started through hermes_cli.mcp_startup's
shared owner instead of a hand-rolled bare thread, keeping the start
lock, retry-after-zero-connected allowance, and interactive-OAuth
suppression. The shared owner now captures the caller's context-local
HERMES_HOME override and re-installs it inside the discovery thread,
so discovery reads the selected profile's mcp_servers (#67605).
- Restore stdio TUI startup discovery in main() and the
_mcp_discovery_enabled retry gate in wait_for_mcp_discovery, both
dropped by the original branch.
- Restore the 3 WSTransport regression tests (send serialization,
cross-batch ordering, drained-token ordering) deleted by the PR.
- Harden the profile-scoped discovery test against sibling-state leaks.
Marker presence was the launch gate, but the marker is provenance about who
ran the install -- not proof the runtime works. A CLI-installed repo+venv, or
a healthy install whose marker a repair deleted, both read as "never
installed" and dropped the user into first-run bootstrap on every launch.
Split the two questions: classifyActiveRuntime() reports marker validity and
runtime usability separately, and the resolver launches whenever the runtime
is usable, logging when it proceeds without a marker. An unusable runtime
still falls through to bootstrap even with a valid marker, so an interrupted
install can't spawn a dead backend.
Drops isBootstrapComplete(), which had no callers left once the gate moved.
Co-authored-by: iveywest <iveywest@users.noreply.github.com>
Co-authored-by: lihengming <lihengming@users.noreply.github.com>
- hermes dashboard --status now verifies each matched PID is alive AND
bound to a listening socket before reporting it, so stale PIDs and the
desktop app's IPC-only 'serve --port 0' backends no longer masquerade
as running dashboards (#58578).
- The git and Windows ZIP update paths share one
_finish_dashboard_update_cleanup(), so the ZIP fallback gets the same
stop/restart reporting.
- _kill_stale_dashboard_processes returns a structured
{matched, killed, failed, unrecovered} result; the explicit was-stopped
notice fires only for processes that could NOT be auto-restarted,
meshing with the auto-respawn from #72192.
install.ps1 wrote the marker on Windows and the Rust installer now writes it,
but install.sh -- the path every Mac and Linux CLI install takes -- never did.
A machine set up with install.sh therefore looked uninstalled to the desktop
app, which re-ran first-run bootstrap on every launch.
Stamp the same schema-v1 payload install.ps1 writes, from both the staged
`complete` stage and monolithic main(). An unresolvable HEAD skips the marker
rather than writing one the desktop validator rejects: absent reads as a clean
"bootstrap needed", malformed reads as a confusing half-state.
The expanded terminal row printed the same string as the title, as the
`$` transcript, and again as detail. shellCommand preferred the
backend's display preview over the real `command` arg, so the transcript
showed a summary of what ran rather than what ran; and a terminal call
with no output fell through to the generic fallback, which echoes
args.context under a transcript already showing it.
_tool_ctx switched to build_tool_label in #55166, so every tool.start
carried an already-phrased string ("Running sleep 70 + 2 commands").
Both clients then apply their own verb on top: the TUI renders
Terminal("Running sleep 70 + 2 commands") and the desktop row reads
"Ran Running sleep 70 + 2 commands". The friendly labels stay where they
belong — the CLI spinner and the gateway progress line, which compose
verb + preview at their own call sites.
The macOS launcher fast path gates on hermes_is_installed(), which needs
.hermes-bootstrap-complete next to a built desktop app. Nothing in the Rust
bootstrap pipeline ever wrote that marker -- only install.ps1 did -- so every
reopen of /Applications/Hermes.app re-ran setup instead of launching.
Publish the marker atomically (temp sibling + fsync + rename) because
hermes_is_installed() only checks existence: a torn direct write would arm
the fast path against a half-installed tree. A marker write failure emits
BootstrapEvent::Failed so the installer UI leaves the progress state.
Co-authored-by: giggling-ginger <giggling-ginger@users.noreply.github.com>
Two problems, both found by distrusting the harness's own numbers.
1. The scenario slept a fixed 1s after mounting tabs, then recorded. Boot
and session hydration are not reliably done by then, so a variable
amount of unrelated work landed inside the measurement window. Three
back-to-back runs on identical code spread 2.2x on total_renders and
3.8x on wasted_renders — wide enough that a single-run before/after
delta could be mostly noise. Replaced with a quiesce gate that waits
for commits to hold still before recording, and reports 'quiet:N' or
'timeout:...' so a contaminated run is visible instead of silent.
2. The counter attributed a context-driven re-render as 'wasted', which
pointed at memo() as the fix when memo cannot block context at all.
Adds contextChanged via the fiber's context dependency list, and
excludes it from wasted.
The gate also turned up a finding worth more than the fix: with five busy
tiles and NO driver running, the renderer still commits ~18x/sec. The
report now names the cascade roots (own state changed, props did not)
rather than leaving them to be guessed at — Streamdown re-renders itself
105 times while idle, which is what drives Block/Ct.
'Refreshing cua-driver (Computer Use)...' could hang for minutes on
Windows: when the driver's native check-update verb returned an
indeterminate result (old driver without the verb, offline, GitHub
rate-limited, or the probe timing out), install_cua_driver(upgrade=True)
fell through to the full upstream installer — a silent, output-captured
run with a 660s ceiling, plus install.ps1's 600s concurrency-lock wait
on Windows on top. Every 'hermes update' paid that cost.
Two changes:
- install_cua_driver() grows require_confirmed_update: with it set, an
indeterminate check keeps the installed version and returns fast,
printing the force path (hermes computer-use install --upgrade).
'hermes update' passes it; the explicit --upgrade CLI keeps the old
fall-through so a force refresh still works when the check can't
answer.
- cua_driver_update_check() default timeout is now 25s on Windows
(8s unchanged on POSIX): first-spawn of the exe under Defender /
SmartScreen routinely exceeds 8s, and a false timeout is exactly the
indeterminate result that used to trigger the multi-minute reinstall.
Capture each manually-started dashboard/serve process's argv before the
stale-process kill (/proc/<pid>/cmdline on Linux, ps -o command= on macOS),
then respawn it detached after the update — headless (--no-open) with output
to logs/dashboard-restart.log under the active profile's HERMES_HOME.
Supervised PIDs keep their systemd-unit restart; --stop stays a plain stop.
Salvaged from PR #41508 with scope fixes: serve matching preserved, profile-
aware log path, restart only on the update path (restart_managed=True).
Strengthen the salvaged regression test to prove the end-to-end claim in
#56580/#68874: a DM-created task's terminal wake must build the creator's
':dm:<chat_id>' session key via build_session_key(), not a group-scoped
key that forks a fresh session. Sabotage-verified: reverting the watcher
to the hardcoded chat_type='group' fails this test.
Follow-up to the main fix in this PR. rodriguez46p-ui's review on the
equivalent #56632 (closed stale) flagged that only the auto-subscribe
path in tools/kanban_tools.py was covered; the same gap existed in two
more call sites:
- gateway/slash_commands.py: the `/kanban create` slash command auto-
subscribes the calling session but didn't pass chat_type. Read it
from source.chat_type (already available on SessionSource).
- hermes_cli/kanban.py: the `kanban notify-subscribe` CLI command now
accepts --chat-type and threads it through.
The dashboard plugin API (plugins/kanban/dashboard/plugin_api.py) still
has the gap because the home_channel config schema doesn't carry
chat_type — that's a follow-up that needs a config schema change.
Verified: 258 tests pass on the kanban + session_context suites.
DEFAULT_CONFIG declared "kanban" twice. Python keeps only the last
literal for a duplicate key, so the first kanban block was silently
dropped and its "auto_subscribe_on_create": True default never made it
into DEFAULT_CONFIG. The consumer in tools/kanban_tools.py masks the
miss with cfg_get(..., default=True), but the documented default was
absent from DEFAULT_CONFIG (so config templates / 'hermes config show'
omit it), and the duplicate key is a standing hazard: any future key
added to the first block would also vanish.
Merge auto_subscribe_on_create into the single canonical kanban block.
Adds a regression test asserting both default sets survive and a guard
against any duplicate top-level DEFAULT_CONFIG key.
Follow-up on top of @waroffchange's alignment fix (#55673): the real
default changed from 90 to 500 in #72176, so bring the example value
and comment up to the current default.
