Two fixes for desktop hands-free voice:
- The live speech session bound to the first assistant bubble with text, so
a tool-calling turn spoke only the opening narration and silently dropped
every later interim AND the final answer. The conversation selector now
aggregates all unspoken assistant bubbles in order (turn-scoped speech);
auto-speak keeps its latest-reply-only behavior.
- The speak-stream WS producer blocked forever on the text queue, so a
narration line with no trailing whitespace ("Let me check.") sat in the
sentence chunker until end-of-turn — spoken long after the tool finished,
with the UI stuck on "Preparing audio…". Mirror the CLI speaker's idle
flush: sentence-terminated buffers flush after 0.5s of producer silence,
anything else after ~2s; open <think> blocks are never flushed.
* fix(desktop): render agent credit notices as toasts (#69808)
The desktop renderer had no handler for the `notification.show` /
`notification.clear` WS events, so every credit-usage notice the backend
sends (`agent/credits_tracker.py` → `tui_gateway/server.py`) was silently
dropped. Credit warnings like "• Credits 50% used · $220.00 cap" never
appeared, even though the Ink TUI renders them in its status bar.
Add the two missing branches to the gateway-event dispatcher, delegating
to a small, pure-testable module:
- `store/agent-notices.ts` — `noticeToToast()` maps a notice to a toast
(level → toast kind, sticky → durationMs 0, ttl → ttl_ms), and uses the
notice `key` as the toast id. Re-emitting the same key REPLACES the
toast, so the credits 50→75→90 line escalates in place instead of
stacking, and a key-matched `notification.clear` maps straight to
`dismissNotification(key)`.
- The notice `text` already carries its own glyph (• ⚠ ✕ ✓), so no toast
icon is added.
- Notices are account-wide, so the toast shows regardless of which
session is focused.
The Ink TUI (`ui-tui/src/app/turnController.ts`) is the reference for the
latest-wins / sticky-vs-ttl / key-matched-clear behavior.
Export `NotificationInput` so the mapping's return type can be named.
* feat(desktop): native OS credit alerts + billing-page nudge (#69808)
Round out the credit-notice handling from the previous commit with the two
optional pieces from the issue:
- Native OS notification for the urgent pair. `credits.depleted` /
`credits.restored` also fire an Electron notification when Hermes is
backgrounded, via a new `credits` NativeNotificationKind (the existing
five didn't fit) with its own toggle in Settings → Notifications (the
panel is data-driven off NATIVE_NOTIFICATION_KINDS, so the toggle and
i18n are the only additions). The escalating usage line and grant-spent
notice stay in-app toasts only. Dispatch is `global` (account-wide, not
session-bound) and gated by the user's prefs + backgrounded check.
- Billing-page nudge. A `credits.*` crossing invalidates the
`['billing','state']` query so Settings → Billing reflects the change
immediately instead of waiting up to 30s for its poll.
`nativeNoticeInput()` is a pure mapping (urgent-key gate → native input),
unit-tested directly; the gateway-event branch does the localized-title
lookup and gated dispatch. i18n added for all four locales.
* feat(credits): report $used of $cap instead of % in the usage notice
The usage gauge is Nous subscription-cap-only (used_fraction requires a cap;
non-Nous providers emit no headers, so no notice fires). A bare percentage
implied a universal unit that doesn't exist, so report the absolute dollars
used of the cap instead: used = cap - remaining, from micros (money-safe),
clamped to [0, cap]. Still a snapshot at band-crossing (re-emits on band
change, not every turn) to keep the single escalating line and stay quiet on
append-only surfaces (messaging pushes one message per crossing).
* fix(desktop): de-dupe credit toast icon, band-color the figure, split detail
Three fixes to how agent credit notices render as toasts:
- Strip the leading severity glyph (the toast already draws a kind icon, so the
raw text doubled it). Native OS notifications keep the glyph (no icon there).
- Icon top-margin is now 0.42ch (font-relative) instead of a fixed rem.
- Band-color the $used figure (semibold) by $used/$cap: muted <75%,
--ui-orange >=75%, --ui-red >=90% (depleted red, restored green), reusing the
existing --ui-* usage palette. Icon shares the accent.
- Split a trailing '. detail' into a muted secondary line (title+description
convention) instead of an inline middot.
Generic 'accentColor' + 'meta' slots on the notification; degrades gracefully
when a notice has no figure.
* fix(desktop): billing page always fetches fresh state (team-account desync)
The x-nous-credits-* headers are best-effort and can drift out of sync,
notably in team/org accounts where another member's spend moves the shared
balance without touching this client's headers. The billing endpoint is the
source of truth, so the page no longer trusts a cache: staleTime 0 +
refetchOnMount 'always' force a fresh fetch on every open and focus (still
polling 30s while mounted). The credits.* invalidation nudge still pulls a
crossing in immediately.
* chore(desktop): dev-only credit-notice demo hotkey
Ctrl+Shift+C (and window.__creditsDemo()) steps the full credit-notice
lifecycle (usage 50->75->90, grant-spent, depleted/restored) through the real
gateway event fan-out via a new emitLocalGatewayEvent, so the toast/native/
billing-invalidation paths are testable without hitting real usage bands.
Installed only under import.meta.env.DEV, so it's tree-shaken from production.
Ctrl+Shift+C (and window.__creditsDemo()) steps the full credit-notice
lifecycle (usage 50->75->90, grant-spent, depleted/restored) through the real
gateway event fan-out via a new emitLocalGatewayEvent, so the toast/native/
billing-invalidation paths are testable without hitting real usage bands.
Installed only under import.meta.env.DEV, so it's tree-shaken from production.
The x-nous-credits-* headers are best-effort and can drift out of sync,
notably in team/org accounts where another member's spend moves the shared
balance without touching this client's headers. The billing endpoint is the
source of truth, so the page no longer trusts a cache: staleTime 0 +
refetchOnMount 'always' force a fresh fetch on every open and focus (still
polling 30s while mounted). The credits.* invalidation nudge still pulls a
crossing in immediately.
Three fixes to how agent credit notices render as toasts:
- Strip the leading severity glyph (the toast already draws a kind icon, so the
raw text doubled it). Native OS notifications keep the glyph (no icon there).
- Icon top-margin is now 0.42ch (font-relative) instead of a fixed rem.
- Band-color the $used figure (semibold) by $used/$cap: muted <75%,
--ui-orange >=75%, --ui-red >=90% (depleted red, restored green), reusing the
existing --ui-* usage palette. Icon shares the accent.
- Split a trailing '. detail' into a muted secondary line (title+description
convention) instead of an inline middot.
Generic 'accentColor' + 'meta' slots on the notification; degrades gracefully
when a notice has no figure.
The usage gauge is Nous subscription-cap-only (used_fraction requires a cap;
non-Nous providers emit no headers, so no notice fires). A bare percentage
implied a universal unit that doesn't exist, so report the absolute dollars
used of the cap instead: used = cap - remaining, from micros (money-safe),
clamped to [0, cap]. Still a snapshot at band-crossing (re-emits on band
change, not every turn) to keep the single escalating line and stay quiet on
append-only surfaces (messaging pushes one message per crossing).
Round out the credit-notice handling from the previous commit with the two
optional pieces from the issue:
- Native OS notification for the urgent pair. `credits.depleted` /
`credits.restored` also fire an Electron notification when Hermes is
backgrounded, via a new `credits` NativeNotificationKind (the existing
five didn't fit) with its own toggle in Settings → Notifications (the
panel is data-driven off NATIVE_NOTIFICATION_KINDS, so the toggle and
i18n are the only additions). The escalating usage line and grant-spent
notice stay in-app toasts only. Dispatch is `global` (account-wide, not
session-bound) and gated by the user's prefs + backgrounded check.
- Billing-page nudge. A `credits.*` crossing invalidates the
`['billing','state']` query so Settings → Billing reflects the change
immediately instead of waiting up to 30s for its poll.
`nativeNoticeInput()` is a pure mapping (urgent-key gate → native input),
unit-tested directly; the gateway-event branch does the localized-title
lookup and gated dispatch. i18n added for all four locales.
The desktop renderer had no handler for the `notification.show` /
`notification.clear` WS events, so every credit-usage notice the backend
sends (`agent/credits_tracker.py` → `tui_gateway/server.py`) was silently
dropped. Credit warnings like "• Credits 50% used · $220.00 cap" never
appeared, even though the Ink TUI renders them in its status bar.
Add the two missing branches to the gateway-event dispatcher, delegating
to a small, pure-testable module:
- `store/agent-notices.ts` — `noticeToToast()` maps a notice to a toast
(level → toast kind, sticky → durationMs 0, ttl → ttl_ms), and uses the
notice `key` as the toast id. Re-emitting the same key REPLACES the
toast, so the credits 50→75→90 line escalates in place instead of
stacking, and a key-matched `notification.clear` maps straight to
`dismissNotification(key)`.
- The notice `text` already carries its own glyph (• ⚠ ✕ ✓), so no toast
icon is added.
- Notices are account-wide, so the toast shows regardless of which
session is focused.
The Ink TUI (`ui-tui/src/app/turnController.ts`) is the reference for the
latest-wins / sticky-vs-ttl / key-matched-clear behavior.
Export `NotificationInput` so the mapping's return type can be named.
Guard the post-start set_agent_cursor_enabled on _session._started so
call_tool cannot re-enter session.start() (matches the start_session
lifecycle guard).
cua_driver_update_check now short-circuits to None when no driver
resolves; CI has none installed, so pin resolve_cua_driver_cmd in the
update-check and env-sanitization tests.
Cut steady-state Computer Use latency without changing default behavior
or waiting on cua-driver:
- Cap screenshots via set_config(max_image_dimension) on session start
(config: computer_use.max_image_dimension, default 1456)
- Cache aux-vision routing per (provider, model) so captures skip
repeated load_config()
- Add computer_use.capture_after_mode (default som) so users can opt
follow-ups down to ax (elements only) for speed
Auto-detect now disables the cursor overlay on darwin as well as
headless/WSL2 Linux. After start_session, also call
set_agent_cursor_enabled(false) when the policy is on so older drivers
without --no-overlay still tear the overlay down.
Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
Print gh-image stdout and stderr on attachment failures, then replace the
pending inline-evidence marker in the PR review comment with an escaped
failure notice before preserving the failing workflow result.
The hermes-sweeper review #4701565902 (2026-07-15) flagged two
consistency issues in `_cua_driver_supports_no_overlay` and one
additive-config concern:
1. `cua-backend.py:260` — the `cua-driver --help` support probe
inherited the full parent environment. cua-driver is a third-party
binary; every other spawn site in this file (manifest probe at
`:214`, MCP spawn at `:697`, install probe at `:997`) uses
`_sanitize_subprocess_env(cua_driver_child_env())`. The `--help`
probe should match. This was a low-impact leak (only help output
exits), but inconsistency is the wrong default for a third-party
subprocess.
2. `cua_backend.py:238` — when the manifest returned a `command`
different from the input `driver_cmd` parameter (e.g. a relocated
executable at `/opt/relocated/cua-driver` while the system binary
is at `/usr/bin/cua-driver`), the support probe ran against
`_CUA_DRIVER_CMD` (the default) instead of the manifest-discovered
`command`. Two failure modes:
- The wrapper binary supports `--no-overlay` but the system binary
doesn't → probe returns False → overlay kept despite capability.
- The system binary supports `--no-overlay` but the wrapper doesn't
→ probe returns True → MCP spawn crashes on the unknown flag.
3. The original commit bumped `_config_version` 31→32 for an additive
default (`computer_use.no_overlay: None`). AGENTS.md specifies that
additive defaults in existing sections are handled by deep merge
and should NOT trigger a version bump. After cherry-picking onto
current `origin/main` (which is already at 33), the bump is
effectively dropped — resolved to main's 33.
Changes:
- Add `env=_sanitize_subprocess_env(cua_driver_child_env())` to the
`--help` subprocess (with the same import + rationale comment as
the manifest probe).
- Pass `driver_cmd=command` (or `driver_cmd=driver_cmd` for the
fallback path) into `_mcp_args_with_overlay_flag`, so the support
probe runs against the binary that will actually be launched.
Tests (3 new):
- `test_help_probe_passes_sanitized_env` — verifies `subprocess.run`
is called with an `env=` kwarg.
- `test_manifest_command_drives_support_probe` — verifies the probe
runs against the manifest command when it differs from the input
driver_cmd.
- `test_fallback_uses_input_driver_cmd_for_support_probe` — verifies
the fallback path (no command in manifest) uses the input
driver_cmd.
- `test_probe_distinguishes_support_between_binaries` — sanity check
that the lru_cache key on `driver_cmd` prevents cross-binary
cache leakage.
File-revert negative test confirmed all three of the new
"manifest/probe" tests are load-bearing: with the pre-fix code, they
fail (probe runs against the default binary instead of the resolved
one); with the fix, they pass. 20/20 tests in
`tests/computer_use/test_cua_no_overlay.py` green.
`TestMcpInvocationResolution` (8/8) still green.
Refs: sweeper review #4701565902
Address review feedback from cross-vendor review (Flash + GPT-OSS):
1. Auto-detect now checks for headless Linux (no DISPLAY), WSL2
(/proc/version contains 'microsoft'), instead of all Linux.
Desktop Linux with a compositor keeps the overlay.
2. Add _cua_driver_supports_no_overlay() that probes cua-driver --help
to check if the flag is supported. Older drivers (< 0.6.x) reject
unknown flags, so passing --no-overlay would crash the MCP spawn.
3. Update tests to cover headless vs desktop Linux, WSL2 detection,
version probe, and the unsupported-driver fallback path.
- Add autouse fixture to TestMcpInvocationResolution to disable
--no-overlay flag so existing tests assert baseline args
- Make test_config_load_failure_fails_safe and test_missing_section_enables
platform-aware (Linux auto-detect returns True, macOS/Windows False)
cua-driver's cursor overlay rendering loop can consume CPU indefinitely
when idle (#28152, #47032). On Linux/WSL2, the overlay serves no visual
purpose and the rendering path is the primary source of idle CPU usage.
Add computer_use.no_overlay config option (default: auto-detect) that
passes --no-overlay to cua-driver when enabled. Auto-detection disables
the overlay on Linux (covers WSL2, headless, containers) where it has no
benefit, and keeps it enabled on macOS/Windows where it is visually
useful.
Refs: #28152, #47032
Keep thin-PATH resolution while asserting the absolute binary on the
already-installed version check. Map contributor emails for the salvage.
Co-authored-by: Tianqing Yun <yuntianqing@yahoo.com>
Co-authored-by: Trevor Gordon <trevorbgordon@gmail.com>
Co-authored-by: Adrian Soto Mora <adrian.soto6@gmail.com>
Supersedes #52948 and #67402. Closes#52945.
Standalone hermes dashboard/serve was trusting HERMES_WEB_DIST and
HERMES_SERVE_HEADLESS inherited from a Desktop Electron parent, which
could serve the packaged desktop renderer ("Desktop IPC bridge is
unavailable") or disable the SPA. Drop only Electron-packaged WEB_DIST
paths (app.asar*) when HERMES_DESKTOP!=1, and clear inherited headless
for non-serve launches, while preserving caller-managed custom dist
overrides and the desktop-spawned backend path.
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
Co-authored-by: Commander <commander@tianji.local>
* fix(desktop): improve fallback tool-call details
Show failed image-generation calls through the normal fallback row, remove duplicate normal-mode web-search JSON, and format Technical Mode payloads as readable JSON.
* feat(desktop): render terminal tool calls as transcripts
Show terminal commands with a prompt and exit status, then reveal ANSI-safe stdout and stderr in the expanded tool row.
* fix(desktop): reconcile tool calls by command
Match context-only tool starts with command-bearing completions when their IDs differ, preventing stale duplicate terminal rows. Show the web-search query above its result cards.
Adds the missing write path for the per-task model_override column (which
was previously only settable via manual SQL) and pairs it with a
provider_override so cross-provider switches resolve correctly:
- kanban_db: provider_override column (+migration), set_model_override()
with model_override_set event, create_task(model_override=,
provider_override=), dispatcher spawns worker with -m <model>
[--provider <name>]
- dashboard: Model row in the task drawer — dropdown fed by a new
/model-options endpoint (build_models_payload substrate, provider-grouped,
free-text fallback), PATCH + bulk model override support
- CLI: kanban create --model/--provider, new kanban set-model subcommand,
show prints the provider
- agent tools: kanban_create accepts model/provider; show/list expose
provider_override
Rate-limit recovery flow: override is settable on running tasks and takes
effect on the next dispatch, without touching the worker profile's config.
`resumeTile` — the cold-resume path for a session opened in a tile / split
pane — resumed with `{ session_id, cols }` and read messages with no profile,
so a tile opening a session from another profile let the gateway fall back to
the launch-profile DB and fork the conversation into the wrong profile: the
same cross-profile bleed the recovery resumes had (#67603), just a sibling
call path. Resolve the owning profile via the shared `resolveSessionProfile`
and carry it on both the transcript prefetch and the resume RPC.
Co-authored-by: oliviaaaa7788 <oliviaaaa7788@users.noreply.github.com>
Narrow the setup-error matcher, match the server's empty-key contract,
and route gateway-event plus create/resume/branch through one
credential-warning policy. Preserve configured=true only for
non-authoritative transport fallback.
Co-authored-by: Yingliang Zhang <zhangyingliang@outlook.com>
Co-authored-by: Brandon R <kingdomwarrior23@gmail.com>
`no-key-required` is a valid sentinel for local/self-hosted/custom
routers. Warning on it made Desktop treat a working setup as missing
API keys.
Co-authored-by: Yingliang Zhang <zhangyingliang@outlook.com>
Co-authored-by: Brandon R <kingdomwarrior23@gmail.com>
* fix(desktop): stop model-switch dup + route recovery resumes to the owning profile
Fixes two Desktop session-reconciliation symptoms from #67603.
Symptom 1 — duplicated user bubble after a model switch. The gateway
persists model-switch / personality notices as role=user `[System: …]`
rows (tui_gateway/server.py) so strict OpenAI-compatible providers don't
reject a non-leading system message (#48338). `preserveLocalPendingTurnMessages`
paired local optimistic rows with the stored transcript by user-role
ordinal, so a marker between two real user turns shifted every later
ordinal and the optimistic row was re-appended at the bottom. The single
trailing-marker case is already covered by the compression-era
`latestAuthoritativeUser` guard, but two switches around one turn (marker
before AND after the committed prompt) still duplicated it. Exclude
`[System:` bookkeeping markers from ordinal pairing on both sides.
Symptom 2 — a session appearing under two profiles. The main resume path
already resolves a session's owning profile via `resolveStoredSession`
(cache → active backend → cross-profile probe), but the recovery
`session.resume` calls (stale runtime id, session-not-found, wedged loop,
redirect) omitted `profile`, so the gateway fell back to the launch-profile
DB and forked the conversation into the wrong profile. Route every recovery
resume — and an uncached right-click branch — through the same resolver so
the profile is carried even for sessions outside the paginated sidebar
window (the cache-miss gap).
Tests: discriminating two-switch marker test (fails before, passes after);
cache-hit + cross-profile cache-miss coverage for the recovery resume and
for branching an uncached session.
Supersedes #68665 and #63590.
Closes#67603.
Co-authored-by: Dolverin <5910064+Dolverin@users.noreply.github.com>
Co-authored-by: oliviaaaa7788 <274182427+oliviaaaa7788@users.noreply.github.com>
* fix(desktop): scope the remembered session id per profile
A single global `hermes.desktop.lastSessionId` key remembered ONE session
across every profile, so a relaunch or cold start under profile B would try
to restore a session owned by profile A — reinforcing the impression that a
conversation had bled between profiles (#67603, second symptom).
Key the remembered id by the session's owning profile (resolved from the
session row's `profile`, falling back to the active gateway profile), read it
back for the active profile on restore, and clear an exhausted session under
its owner. The default profile keeps the original unsuffixed key so existing
installs' remembered session survives the upgrade.
Co-authored-by: oliviaaaa7788 <oliviaaaa7788@users.noreply.github.com>
* fix(gateway): hide [System:] bookkeeping markers from every transcript
Model-switch and personality notices are persisted as role=user `[System: …]`
rows so strict providers accept them mid-history, but they are model-facing
runtime metadata, not user turns. `_history_to_messages` — the single display
projection every client reads — passed them straight through, so on resume or
reload they rendered as a fake user bubble in the desktop, TUI, CLI, and web
transcripts.
Drop them in that projection. The raw marker stays in `session["history"]`
for the model, so nothing changes for inference; only the display loses a row
that never belonged to the user. This also removes the stored marker from the
payload the desktop reconciles against, killing the ordinal shift that
duplicated the optimistic prompt (#67603) at its source — the desktop-side
marker exclusion remains as a fallback for older backends.
Co-authored-by: Dolverin <Dolverin@users.noreply.github.com>
---------
Co-authored-by: Dolverin <5910064+Dolverin@users.noreply.github.com>
Co-authored-by: oliviaaaa7788 <274182427+oliviaaaa7788@users.noreply.github.com>
Co-authored-by: oliviaaaa7788 <oliviaaaa7788@users.noreply.github.com>
Co-authored-by: Dolverin <Dolverin@users.noreply.github.com>
* fix: serialize TUI gateway websocket sends
* fix(tui_gateway): preserve websocket batch order
* refactor(tui_gateway): drop unused _safe_send wrapper
The batch-serialization fix routes every send through _safe_send_many;
_safe_send became a dead single-line wrapper with no callers. Remove it.
---------
Co-authored-by: supplefrog <78985073+supplefrog@users.noreply.github.com>
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
Post-rebase consolidation over the merged C7/C16/C10 work:
- _ensure_dm_conversation now records workspace ownership via
_remember_channel_team and bounds _dm_conversation_cache (cap 5000)
- module-level _slack_dm_cache (C7 standalone path) bounded oldest-first
- _user_is_bot_cache (C10) bounded with _trim_oldest_dict_entries
- caption-mode contract tests updated for the C7+C8 merged media path
Widen the #19237 send_message fix to the live adapter: send, _upload_file,
send_multiple_images, send_image, send_video, send_document,
send_exec_approval, send_slash_confirm, and send_clarify now route bare
Slack user IDs (U.../W...) through a shared _ensure_dm_conversation helper
before calling chat.postMessage / files_upload_v2, which reject user IDs.
This closes the gap in #17261 where an attachment worked when replying in
a thread but failed when directed at a user DM, and extends the DM-open
fallback to clarify/approval Block Kit prompts so gated actions can reach
a user directly.
Resolution uses the workspace-scoped client (multi-workspace installs open
the DM with the right bot token), caches per (team, user), and records the
opened D... channel in the channel→team map. On failure the original
target passes through so the downstream API call surfaces the real Slack
error.
Fixes#17261
Refs #19236
The post-stream helper (_deliver_media_from_response) rescanned the
already-streamed response and promoted bare local filesystem paths into
real uploads via extract_local_files. Since the visible reply was already
streamed verbatim, any bare path there is either text the user has seen
or stale inspected/tool content — not an attachment request. On Slack this
uploaded images from stale inspected content after otherwise clean replies.
Post-stream delivery now honors only explicit MEDIA: directives. The
non-streaming path in gateway/platforms/base.py keeps its bare-path
auto-detect, because that path controls the visible text and strips the
path from the reply when it attaches — auto-attach is intentional there.
Regression tests: bare image/document paths in a streamed reply produce
no upload; explicit MEDIA: tags still deliver.
Fixes#20834
Add early auth check in _handle_slack_message() that runs BEFORE any
API calls (thread context fetch, user name resolution, file downloads)
or file processing. Unauthorized users could previously trigger Slack
API calls and file downloads before the runner's _is_user_authorized
gate rejected them.
Same pattern as Telegram fix#54164: build a SessionSource and check
the runner's _is_user_authorized at the adapter level before event
construction consumes resources.
Fixes the gap where Slack has no adapter-level auth gate while Discord
and Telegram have adapter-level allowlists.
