Guard the post-start set_agent_cursor_enabled on _session._started so
call_tool cannot re-enter session.start() (matches the start_session
lifecycle guard).
Cut steady-state Computer Use latency without changing default behavior
or waiting on cua-driver:
- Cap screenshots via set_config(max_image_dimension) on session start
(config: computer_use.max_image_dimension, default 1456)
- Cache aux-vision routing per (provider, model) so captures skip
repeated load_config()
- Add computer_use.capture_after_mode (default som) so users can opt
follow-ups down to ax (elements only) for speed
Auto-detect now disables the cursor overlay on darwin as well as
headless/WSL2 Linux. After start_session, also call
set_agent_cursor_enabled(false) when the policy is on so older drivers
without --no-overlay still tear the overlay down.
Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
The hermes-sweeper review #4701565902 (2026-07-15) flagged two
consistency issues in `_cua_driver_supports_no_overlay` and one
additive-config concern:
1. `cua-backend.py:260` — the `cua-driver --help` support probe
inherited the full parent environment. cua-driver is a third-party
binary; every other spawn site in this file (manifest probe at
`:214`, MCP spawn at `:697`, install probe at `:997`) uses
`_sanitize_subprocess_env(cua_driver_child_env())`. The `--help`
probe should match. This was a low-impact leak (only help output
exits), but inconsistency is the wrong default for a third-party
subprocess.
2. `cua_backend.py:238` — when the manifest returned a `command`
different from the input `driver_cmd` parameter (e.g. a relocated
executable at `/opt/relocated/cua-driver` while the system binary
is at `/usr/bin/cua-driver`), the support probe ran against
`_CUA_DRIVER_CMD` (the default) instead of the manifest-discovered
`command`. Two failure modes:
- The wrapper binary supports `--no-overlay` but the system binary
doesn't → probe returns False → overlay kept despite capability.
- The system binary supports `--no-overlay` but the wrapper doesn't
→ probe returns True → MCP spawn crashes on the unknown flag.
3. The original commit bumped `_config_version` 31→32 for an additive
default (`computer_use.no_overlay: None`). AGENTS.md specifies that
additive defaults in existing sections are handled by deep merge
and should NOT trigger a version bump. After cherry-picking onto
current `origin/main` (which is already at 33), the bump is
effectively dropped — resolved to main's 33.
Changes:
- Add `env=_sanitize_subprocess_env(cua_driver_child_env())` to the
`--help` subprocess (with the same import + rationale comment as
the manifest probe).
- Pass `driver_cmd=command` (or `driver_cmd=driver_cmd` for the
fallback path) into `_mcp_args_with_overlay_flag`, so the support
probe runs against the binary that will actually be launched.
Tests (3 new):
- `test_help_probe_passes_sanitized_env` — verifies `subprocess.run`
is called with an `env=` kwarg.
- `test_manifest_command_drives_support_probe` — verifies the probe
runs against the manifest command when it differs from the input
driver_cmd.
- `test_fallback_uses_input_driver_cmd_for_support_probe` — verifies
the fallback path (no command in manifest) uses the input
driver_cmd.
- `test_probe_distinguishes_support_between_binaries` — sanity check
that the lru_cache key on `driver_cmd` prevents cross-binary
cache leakage.
File-revert negative test confirmed all three of the new
"manifest/probe" tests are load-bearing: with the pre-fix code, they
fail (probe runs against the default binary instead of the resolved
one); with the fix, they pass. 20/20 tests in
`tests/computer_use/test_cua_no_overlay.py` green.
`TestMcpInvocationResolution` (8/8) still green.
Refs: sweeper review #4701565902
Address review feedback from cross-vendor review (Flash + GPT-OSS):
1. Auto-detect now checks for headless Linux (no DISPLAY), WSL2
(/proc/version contains 'microsoft'), instead of all Linux.
Desktop Linux with a compositor keeps the overlay.
2. Add _cua_driver_supports_no_overlay() that probes cua-driver --help
to check if the flag is supported. Older drivers (< 0.6.x) reject
unknown flags, so passing --no-overlay would crash the MCP spawn.
3. Update tests to cover headless vs desktop Linux, WSL2 detection,
version probe, and the unsupported-driver fallback path.
cua-driver's cursor overlay rendering loop can consume CPU indefinitely
when idle (#28152, #47032). On Linux/WSL2, the overlay serves no visual
purpose and the rendering path is the primary source of idle CPU usage.
Add computer_use.no_overlay config option (default: auto-detect) that
passes --no-overlay to cua-driver when enabled. Auto-detection disables
the overlay on Linux (covers WSL2, headless, containers) where it has no
benefit, and keeps it enabled on macOS/Windows where it is visually
useful.
Refs: #28152, #47032
Limit the _NET_ACTIVE_WINDOW xprop fallback to unqualified default
captures so exact pid/window_id targeting does not pay up to a 2s
subprocess probe on Linux/X11.
When Linux/X11 reports the same z_index for every on-screen window,
prefer _NET_ACTIVE_WINDOW via xprop instead of list order. Keep the
higher-z-index-is-frontmost contract when ordering is informative.
Bug 1 of #55048: when the MCP connection dropped (driver crash / restart),
_lifecycle_coro exited but left _started=True, so the next list_apps/capture
passed _require_started() and then operated on a None session — hanging
forever instead of reconnecting.
- _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead
session is re-enterable (idempotent no-op on the normal stop() path; atomic
bool write, safe from the bridge-loop thread without the lock stop() holds).
- call_tool() re-enters start() when the session isn't active, rebuilding it
before the call. The start_session/end_session handshake (driven by start()/
stop() themselves) is exempted so bootstrap doesn't recurse.
Tests: two cases in test_computer_use_delivery_ladder.py — finally resets
_started, and call_tool restarts a dead session exactly once. Full
computer_use suite green (233).
Refs #55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the
delivery_mode work in #67123.
Hermes' computer_use wrapper dropped cua-driver's structured action verdicts,
exposed no delivery_mode, and injected background-only guidance — so the agent
reported unverified no-ops as success and concluded cua-driver 'cannot drive'
Electron/Chromium surfaces (observed live on tldraw offline). Fixes#67052.
Phase A — preserve the result contract:
- ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode
- CuaDriverBackend._action() reads structuredContent (was data-only); a helper
normalizes it, additive and None-safe on old drivers
- _text_response surfaces the fields additively (ok stays transport-only)
Phase B — bounded, model-reachable foreground:
- delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher,
ABC, and all input methods
- foreground is capability-gated (input.delivery_mode); old drivers get a
structured foreground_unsupported refusal, never a silent background downgrade
- no automatic/hidden foreground retry — the model selects it from the signal
Phase C — guidance + isolation:
- system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from
background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven
by returned effect/escalation, not predicted from the app being Electron
- foreground approval scoped by (action, delivery_mode): a background approval
never silently authorizes foreground
- approval state keyed per session_id so concurrent gateway runs don't leak unlocks
Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/
unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating +
foreground_unsupported, and session-scoped foreground approval. Existing 265
computer_use tests still green.
Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click
returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground
request returned code='foreground_unsupported' — correct on a driver that predates
the input.delivery_mode capability.
* fix(computer-use): target Linux app windows reliably
Resolve app filters through the canonical cua-driver MCP app metadata and join running app PIDs back to windows. Preserve an exact selected window across capture_after, support direct capture by pid/window_id, and send the active window ID for coordinate pointer actions on Linux.
Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
* fix(computer-use): address review on PR #63725
Address three review comments from @f-trycua:
1. type_text, press_key, and hotkey now carry _active_window_id and
fail closed when it is missing. Previously they sent only the PID,
so CUA Driver fell back to the first window for that PID — input
could reach the wrong window in multi-window apps.
2. Coordinate scroll x/y are now capability-gated behind
input.scroll.coordinates. CUA Driver 0.7.1 Linux schema rejects
x/y on scroll; omitting them when the driver doesn't advertise
support avoids the schema rejection while still routing via
window_id.
3. Windows are sorted by z_index descending (higher = front, per CUA
Driver semantics) instead of ascending. Null z_index (Wayland) is
coerced to 0 in _ingest_windows so it doesn't crash the sort and
sorts to the back instead of being selected as the capture target.
---------
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
PR #58889 fixed the CLI-fallback transport; review of that fix found the
same leak class at four sibling spawn sites of the third-party cua-driver
binary:
- _resolve_mcp_invocation (cua-driver manifest): no env= at all — full
parent environment inherited
- cua_driver_update_check (check-update --json): telemetry env but no
secret sanitization
- doctor._drive_health_report (<binary> mcp Popen): telemetry env only
- permissions._run (every macOS/Linux permission probe): telemetry env only
All now route through _sanitize_subprocess_env(cua_driver_child_env()),
matching the sanctioned MCP spawn and the #53503/#55709/#58889 strip-by-
default policy for non-terminal spawns. Sanitization degrades gracefully
(falls back to the telemetry env) so doctor/permission probes never break
on an import error.
4 regression tests covering each site.
_CuaDriverSession._call_tool_via_cli() (the EAGAIN/silent-empty MCP
fallback transport) invokes `cua-driver call <tool> <json>` via
subprocess.run() with no env= argument, so the third-party cua-driver
binary inherits the full, unsanitized parent environment. The primary
MCP spawn site (_lifecycle_coro) already applies
_sanitize_subprocess_env(cua_driver_child_env()) before opening the
stdio client, per the same policy #53503/#55709 established for other
subprocess spawn points — this fallback path, added alongside the
EAGAIN/silent-empty-capture hardening, missed it.
Fix: apply the same env=_sanitize_subprocess_env(cua_driver_child_env())
to the subprocess.run() call in _call_tool_via_cli(), mirroring the
sanctioned spawn site exactly (telemetry policy applied first, then
Hermes-managed secrets filtered).
The SOM/AX element list dropped labels for two extremely common cua-driver
render forms, leaving the model unable to target elements by name:
- [79] AXButton (Dark) -> parenthesised label
- [4] AXStaticText = "Wi-Fi" -> = "value" form
- [92] AXPopUpButton = "Automatic" -> = "value" form
The old regex only matched quoted "label" and id=Label, so System Settings
buttons/text/popups all surfaced with empty labels. That's why selecting the
macOS Appearance 'Dark' button by element index required guessing — the
labels weren't available to aim with.
Fix: extend _ELEMENT_LINE_RE to capture all four label forms (= "value",
"quoted", (parenthesised), id=Label), skipping a pure-digit (N) order number
in favour of the id= label. Verified live against System Settings: the
Appearance buttons now surface as Auto/Light/Dark.
Adds a regression test covering all label forms. Full suite: 84 passed.
The first fix handled the EAGAIN McpError path. But the persistent MCP
session (long-running gateway/desktop worker) has a second failure mode:
list_windows or get_window_state 'succeed' over MCP yet return a
degenerate/empty payload (no windows, or no screenshot + blank tree)
WITHOUT raising — typically when the bridge reconnected mid-call and
dropped the heavy response. That surfaced to the model as a silent 0x0
capture with no error and no fallback firing (0.00s empty return).
Fix: detect empty results in capture() and re-fetch over the CLI
transport before giving up:
- empty list_windows -> CLI re-fetch the window list
- empty get_window_state (som/ax) -> CLI re-fetch the AX tree + screenshot
- empty screenshot (vision) -> CLI re-fetch get_window_state for the PNG
Adds 2 regression tests. Full suite: 83 passed.
The cua-driver MCP stdio bridge intermittently (and on some machines
persistently) fails to forward heavier calls like get_window_state to
the daemon with POSIX EAGAIN — 'daemon transport error forwarding
get_window_state: Resource temporarily unavailable (os error 35)'.
The wrapper surfaced this as an empty 0x0 capture, so computer_use
returned blank screenshots even though the display, permissions, and
the daemon were all healthy (the direct 'cua-driver call' CLI path
worked fine throughout).
Fix: when the MCP path raises the transient/transport error, fall back
to the 'cua-driver call' subprocess transport, which talks to the
daemon over a different socket. The CLI fallback routes get_window_state
screenshots to a temp file via screenshot_out_file (tiny JSON response
instead of a multi-MB base64 blob that congests the socket), reads the
PNG back, retries with backoff, and remaps the JSON into the same
{data, images, structuredContent, isError} shape the MCP path produces
so capture()/_action() are transport-agnostic.
Adds _is_transient_daemon_error() classifier and 3 regression tests.
Verified live: captures that returned 0x0 now return full
1567x905 screenshots with the AX element tree.
The 'never reached ready' error (issue #57025) was undiagnosable — doctor
and MCP test pass while the wrapper times out, with no hint where startup
stalled. Track a phase marker through _lifecycle_coro (binary-check →
manifest-discovery → mcp-initialize → capability-discovery → ready) and
include it in the timeout RuntimeError plus a pointer to doctor and the
agent.log phase timings.
Complements the 15s→30s bump + success-path phase timing log from #58760.
On Windows, the cua-driver MCP session initialization can exceed the 15s
timeout due to manifest subprocess discovery + MCP transport setup.
This makes the computer_use tool permanently unavailable even though
hermes computer-use doctor and hermes mcp test both pass.
- Increase _ready_event.wait timeout from 15s to 30s
- Add startup timing instrumentation (manifest + mcp_init durations)
- Log timing at INFO level for diagnosability
Fixes#57025
On X11 a window's PID comes from the optional _NET_WM_PID property, so
cua-driver's list_windows legitimately returns pid: null for windows that
don't set it (desktop root, panels, override-redirect popups). capture()
and focus_app() coerced every entry via int(w["pid"]) inside a list
comprehension, so a single null-pid window raised TypeError and aborted
the whole enumeration before any screenshot — capture was impossible on
any X11 desktop with even one such window.
Route both ingestion sites through a new _ingest_windows() helper that
skips entries lacking a usable pid/window_id (uncapturable anyway) and
coerces the rest.
Adds tests/tools/test_computer_use_null_pid_windows.py covering the
helper's filtering/coercion and an end-to-end capture() regression that
reproduced the crash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(windows): stop subprocess console-window popups + add CI guard
The single biggest source of Windows 'terminal popup' bug reports was bare
subprocess.run/Popen calls spawning a console window. The compat helpers
(windows_hide_flags / windows_detach_popen_kwargs) already existed but the
footgun checker had no rule to stop new bare calls from reintroducing the flash.
- scripts/check-windows-footguns.py: new AST-based rule flagging subprocess
calls that can create a new console — output-redirection-aware (capture/
redirect/check_output exempt) and POSIX-only-program-aware (launchctl/
systemctl/brew/etc. exempt). Comprehensive on real popups, no annotation
burden on calls that can't flash.
- Swept all genuine window-spawning sites through windows_hide_flags()/
windows_detach_popen_kwargs(); marked intentionally-visible launches
(editor/terminal/foreground re-exec) with '# windows-footgun: ok'.
- tests/scripts/test_windows_footgun_subprocess_rule.py: behavior-contract
tests + full-repo cleanliness invariant.
- CONTRIBUTING.md: documents the rule + the helper pattern.
* test: accept creationflags kwarg in psutil_android fake_subprocess_run
The Windows no-window sweep added creationflags=windows_hide_flags() to
install_psutil_android.py's subprocess.run call; the test's fake stub had a
fixed (cmd) signature and raised TypeError on the new kwarg.
The subprocess-stdin guard (TUI gateway fd-inheritance protection) flagged
the `permissions grant` call. None of the cua-driver probes/grant read
stdin, so DEVNULL is correct; apply it to the shared `_run` helper and the
grant call.
The card was macOS-only. cua-driver also runs on Windows and Linux, so
fold `cua-driver doctor` (cross-platform binary/health probes) into a
single OS-aware `ready` signal:
- macOS: ready == both TCC grants; keeps the permission rows + grant flow.
- Windows/Linux: no TCC toggles, so ready == driver health, with a
per-OS note (SmartScreen/UIAccess on Windows; X11/XWayland on Linux).
`computer_use_status()` replaces the macOS-only `permissions_status()` and
surfaces `platform`, `ready`, `can_grant`, and the doctor `checks` (non-ok
ones render as warnings). CLI `permissions status`, the REST endpoint, and
the desktop card all key off the one payload. Grant stays macOS-only (400
elsewhere — nothing to grant).
Vision mode called a `screenshot` MCP tool that cua-driver dropped in
0.5.x (full-window PNG capture was folded into `get_window_state`). The
driver replied "Unknown tool: screenshot", so `images` came back empty,
`png_b64` stayed None, and capture returned a 0x0 result with no image
on every call. `som`/`ax` were unaffected because they already use
`get_window_state`, which masked the regression.
Route vision by capability:
- driver advertises `screenshot` (older builds) -> use it (no AX walk)
- otherwise -> call `get_window_state` but discard the AX tree/elements,
returning only the PNG so vision stays free of element noise
- capabilities not yet discovered -> try `screenshot`, fall back to
`get_window_state` on an empty image, so the path self-heals
Add `_image_from_tool_result` to pull the PNG from either an MCP image
content-part or `structuredContent.screenshot_png_b64`, and use it on
the som path too so the image won't silently drop on driver builds that
deliver it via structuredContent instead of a content part.
Verified live (vision: 1568x954, 0 elements; som: image + 527 elements)
and with unit coverage of all four routing cases.
Computer Use already worked through the desktop backend (the cua-driver
toolset enables + installs via Settings -> Skills & Tools), but there was
no in-app way to see or grant the two macOS permissions it needs, so "give
a model my Mac" was tribal knowledge.
The grants attach to cua-driver's OWN TCC identity (com.trycua.driver /
the installed CuaDriver.app), not Hermes -- so no app entitlement is
involved. cua-driver 0.5+ exposes `permissions status/grant`, which we wrap:
- tools/computer_use/permissions.py: thin client over the two subcommands
- hermes computer-use permissions {status,grant}: CLI parity
- GET /api/tools/computer-use/status, POST .../permissions/grant: desktop REST
- ComputerUsePanel: live Accessibility + Screen Recording state with a
Grant button (dialog attributed to CuaDriver), shown in the expanded
Computer Use toolset row. Binary install stays in the existing provider
post-setup runner.
Follow-ups: i18n the card copy; a "Stop driver" control (cua-driver stop)
for the runaway-`serve` case.
capture(app='screen'|'desktop') now resolves to the OS shell/desktop
window (Windows Progman/WorkerW desktop or Shell_TrayWnd taskbar, macOS
Finder/Dock) so 'show me my screen' and 'click the taskbar' work.
Previously capture() only matched application windows, and the schema
advertised 'or the whole screen' without any code path delivering it.
cua-driver is window-oriented (no virtual-desktop or per-monitor MCP
tool), so a single image still cannot span multiple monitors — the
schema now states this and the no-desktop-window path returns a clear
message instead of silently grabbing the frontmost app.
cua-driver 0.6.x removed the standalone screenshot MCP tool, so
capture(mode='vision') hit 'Unknown tool: screenshot' and returned a
0x0 image with no PNG while som/ax (which use get_window_state) still
worked. Route vision through get_window_state(capture_mode='vision').
Salvaged from PR #50771; same fix submitted earlier as #39262 by
@Tranquil-Flow.
* feat(computer_use): disable cua-driver telemetry by default, add opt-in
cua-driver ships anonymous PostHog usage telemetry ENABLED by default
upstream (fires cua_driver_install / cua_driver_doctor events to
eu.i.posthog.com). Hermes now disables it for our users unless they
explicitly opt in.
- New config key `computer_use.cua_telemetry` (default false) in
DEFAULT_CONFIG.
- `cua_backend.cua_driver_child_env()` injects
`CUA_DRIVER_RS_TELEMETRY_ENABLED=0` into the child env when telemetry is
disabled (the default); leaves the var untouched on opt-in so the driver
uses its own default. Reads config fail-safe — any error defaults to
telemetry off.
- Routed every cua-driver spawn site through the policy: MCP backend
(StdioServerParameters env), `cua_driver_update_check`, doctor's
health_report Popen, the install.sh/install.ps1 runner, and the
`--version` / status probes.
- Docs: new Telemetry subsection in computer-use.md (EN).
- Tests: tests/computer_use/test_cua_telemetry.py — default disables,
explicit-false disables, opt-in leaves var untouched, config-failure
fails safe, inherited-enabled is overridden off.
Verified live on Linux against the real cua-driver-rs 0.6.0 binary: with
the var=0 the driver reports "telemetry: disabled via
CUA_DRIVER_RS_TELEMETRY_ENABLED" and sends no event; with it unset it logs
"sending event: cua_driver_doctor". 213 computer_use + install tests green.
* fix(dashboard): fold computer_use config category into agent tab
The new computer_use.cua_telemetry key created a single-field dashboard
config category, tripping test_no_single_field_categories (web_server's
invariant that categories with <2 fields must be merged to avoid tab
sprawl). Add computer_use -> agent to _CATEGORY_MERGE, matching the
existing onboarding/telegram single-field folds.
The runtime gate (check_computer_use_requirements) and the hermes tools
platform_gate both enable linux alongside darwin/win32, but several
docstrings/comments still described Linux as "alpha, gated off until it
flips upstream" — contradicting the code that ships it. Bring the prose in
line with the gate that's actually live:
- tool.py / cua_backend.py module docstrings: Linux is enabled (X11 today,
Wayland via XWayland), not gated off.
- toolsets.py description and hermes tools display name: (macOS/Windows) ->
(macOS/Windows/Linux).
No behavior change — the gate already allowed all three platforms.
Make the computer_use toolset platform-agnostic by driving cua-driver on
macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces
(capability discovery, structuredContent AX tree, opaque element_token,
click button enum, explicit mimeType, machine-readable manifest,
structured list_windows, structured health_report), each degrading
gracefully on older drivers.
Adds `hermes computer-use doctor` (drives cua-driver health_report with a
per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full
typed wrappers for the previously-uncovered cua-driver tools plus a generic
call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware
system-prompt guidance (host-deterministic, cache-safe), and honors
HERMES_CUA_DRIVER_CMD end-to-end.
Replaces the macOS-only skills/apple/macos-computer-use skill with a
cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans
docs.
Supersedes #44221 (Windows-enablement salvage of #30660).
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
- Use _sanitize_subprocess_env() to filter Hermes-managed credentials
from the cua-driver subprocess environment (issue #37878)
- Prevents credential exfiltration to the third-party cua-driver binary
- Aligns with existing pattern used by browser-tool and other tools
- Add regression test to verify environment sanitization
The cua-driver is a lower-trust MCP subprocess per SECURITY.md §2.3.
Its inherited environment is now scrubbed by default, removing provider
API keys, gateway tokens, and platform credentials that should not leak
to third-party binaries.
Fixes#37878
xAI's consent page renders the authorization code in-page instead of
redirecting to the loopback callback, so the listener just hangs and the
manual-paste flow demands a callback URL that never contains the token.
- auth.py: poll stdin non-blockingly while waiting for the xAI loopback
callback; accept a pasted bare Grok Build code and substitute the locally
generated state (PKCE code_verifier still binds the exchange). No need to
wait for timeout or re-run with --manual-paste.
- computer_use: parse PNG/JPEG dimensions from base64 and fall back to the
text/AX/SOM payload when the screenshot is below the provider minimum
(8x8), which xAI rejects with HTTP 400.
- model_setup_flows.py: xAI credential reuse prompt uses the standard radio
picker via a shared _prompt_auth_credentials_choice helper.
- main.py: thread a title through _prompt_provider_choice; re-home the helper
import (flows live in model_setup_flows.py post-decomposition).
Salvaged from #36781 onto current main (contributor's main.py edits re-homed
to model_setup_flows.py, where the flows were extracted since the PR opened).
Vulture + per-symbol verification (whole-repo grep incl. tests, string
literals, getattr, decorator/registry/argparse dispatch) confirmed each of
these has zero callers anywhere — not reachable via any dynamic-dispatch path,
not referenced by tests, not re-exported.
Removed:
- acp_adapter/tools.py: _build_patch_mode_content
- agent/anthropic_adapter.py: read_claude_managed_key (diagnostics-only, never called)
- agent/bedrock_adapter.py: get_bedrock_model_ids
- agent/browser_registry.py: get_active_browser_provider
- agent/chat_completion_helpers.py: _take_request_client (x2 nested closures, never invoked)
- gateway/platforms/weixin.py: _rewrite_headers_for_weixin, _rewrite_table_block_for_weixin
- hermes_cli/banner.py: _skin_branding
- hermes_cli/debug.py: _delete_hint
- hermes_cli/gateway.py: _setup_email, _setup_sms, _setup_yuanbao
(platform keys absent from the _builtin_setup_fn dispatch dict; handled by
the _setup_standard_platform fallback)
- hermes_cli/kanban_db.py: set_max_runtime, active_run
- hermes_cli/kanban_diagnostics.py: severity_of_highest, _latest_clean_event_ts
- hermes_cli/main.py: _build_provider_choices, cmd_portal
(portal subcommand is wired via portal_cli.add_parser, not this wrapper)
- hermes_cli/model_switch.py: CustomAutoResult (orphaned by the switch_model() extraction)
- hermes_cli/models.py: format_model_pricing_table, fetch_nous_account_tier
- hermes_cli/portal_cli.py: _nous_portal_base_url
- hermes_cli/proxy/server.py: handle_models_fallback (defined but never registered on the router)
- tools/computer_use/cua_backend.py: _parse_element, _is_arm_mac
- tools/file_operations.py: _get_safe_write_root (prod uses the imported
agent.file_safety.get_safe_write_root directly)
- tools/skills_tool.py: _load_category_description
Also dropped two imports left unused by the removals:
- tools/file_operations.py: get_safe_write_root alias
- tools/computer_use/cua_backend.py: import platform
Pure deletion: -551 LOC. No behavior change. Test files covering the edited
modules pass (640/640); the broader suite's pre-existing/env-dependent
failures reproduce unchanged on origin/main.
Remove unused imports (F401) and duplicate/shadowed import
redefinitions (F811) across the codebase using ruff's safe
autofixes. No behavioral changes -- imports only.
- ~1400 safe autofixes applied across 644 files (net -1072 lines)
- __init__.py re-exports preserved (excluded from F401 removal so
public re-export surfaces stay intact)
- Re-exports that are imported or monkeypatched by tests but look
unused in their defining module are kept with explicit # noqa:
F401 (gateway/run.py load_dotenv; run_agent re-exports from
agent.message_sanitization, agent.context_compressor,
agent.retry_utils, agent.prompt_builder, agent.process_bootstrap,
agent.codex_responses_adapter)
- Unsafe F841 (unused-variable) fixes deliberately skipped -- those
can change behavior when the RHS has side effects
- ruff lints remain disabled in pyproject.toml (only PLW1514 is
selected); this is a one-time cleanup, not a config change
Verification:
- python -m compileall: clean
- pytest --collect-only: all 27161 tests collect (zero import errors)
- core entry points import clean (run_agent, model_tools, cli,
toolsets, hermes_state, batch_runner, gateway)
- static scan: every name any test imports directly from an edited
module still resolves
_maybe_follow_capture() issued a follow-up screenshot unconditionally
when capture_after=True, even when res.ok=False. The model then received
a normal-looking screenshot alongside an error message, and in practice
it often ignored ok=False and proceeded as if the action had succeeded.
Fix: return _text_response(res) early when res.ok is False so the model
receives only the error and can decide how to recover.
Tests added:
- test_capture_after_skipped_when_action_failed: patches click to return
ok=False and asserts no capture call is issued.
- test_capture_after_fires_when_action_succeeds: ensures the happy path
still triggers the follow-up capture.
_dispatch() routes action="set_value" to backend.set_value(), but:
- ComputerUseBackend did not declare set_value as @abstractmethod, so
subclasses could silently omit it without a TypeError at class load time.
- _NoopBackend (the test/CI stub) had no set_value method at all, causing
AttributeError in any test that exercises the set_value action path.
Fix:
- Add set_value as @abstractmethod to ComputerUseBackend in backend.py.
- Add a recording stub in _NoopBackend in tool.py.
- Add two TestDispatch cases: one verifying the call reaches the backend,
one verifying the missing-value guard returns a clean error.
The cherry-pick of #22891 (max_elements cap) reshuffled _capture_response
so summary was assigned inside both the multimodal and AX branches,
but #30126's aux-vision routing call (_route_capture_through_aux_vision)
fires BEFORE either branch and references the not-yet-bound name.
Compute summary once up-front, keep the AX-branch rebuild for the
truncation note.
Four findings from Copilot's review on PR #22891, all in the AX
elements-array cap added by 22fa1ed:
1. The truncation note ("response truncated to N of M elements") was
appended unconditionally — including in the som/vision multimodal
path, whose response carries a screenshot rather than an `elements`
array. The note described a payload field that wasn't present.
Moved the note into the AX-text branch where the array actually
appears.
2. `_format_elements(cap.elements)` ran on the full untrimmed list with
its own `max_lines=40` cap, so a caller passing `max_elements=10`
would see summary lines referencing `#11..#40` even though the JSON
`elements` array only held #1..#10. Format on `visible_elements`
instead so the summary indices always exist in the response.
3. `_coerce_max_elements` enforced a lower bound but no upper bound,
so `max_elements=10_000_000` silently disabled the safeguard and
reintroduced the original context-blow-up. Added a hard cap
(`_MAX_ALLOWED_MAX_ELEMENTS = 1000`) that clamps oversized values.
4. The schema string said "Default 100" but the property carried no
`default` field, and claimed `max_elements` had no effect on som/
vision while the image-missing fallback path can still return an
elements array. Added `"default": 100`, `"maximum": 1000`, and
clarified the fallback-path wording.
Each finding gets a regression test:
- test_capture_ax_clamps_oversized_max_elements_to_hard_cap
- test_capture_ax_summary_indices_match_returned_elements
- test_capture_multimodal_summary_omits_truncation_note
- test_schema_max_elements_documents_default_and_upper_bound
Verified with `pytest tests/tools/test_computer_use.py` (53 passed,
including the 5 new cases). Confirmed each new test fails on the
pre-fix code path before applying the production change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`computer_use(action='capture', mode='ax')` returned the full AX element
list verbatim in the JSON response. Dense Electron / Obsidian / JetBrains
UIs publish 500+ AX nodes (one reproduction in #22865 returned 597
elements against Obsidian), so a single capture could consume enough
context to trigger compression failures or render the session unusable.
The human-readable `_format_elements` summary is already capped at 40
lines, so the truncation gap was invisible to anyone reading the summary
output.
Add a `max_elements` argument to the tool schema, default 100, that
trims the AX `elements` array. When the cap fires, the response surfaces
`total_elements` and `truncated_elements` and appends a "raise
max_elements or pass app= to narrow" hint to the summary so the model
knows the JSON view is partial and can re-issue with a tighter scope.
Validation is centralized in `_coerce_max_elements`: missing /
non-integer / sub-1 inputs fall back to the default cap, so the
protection can never be silently disabled by a malformed tool-call
argument. The cap only affects AX-mode JSON; `mode='som'` and
`mode='vision'` keep returning a screenshot + image-aware summary
unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the active main model has no vision capability — or when the user
explicitly configured auxiliary.vision in config.yaml — sending the
captured screenshot back to the main model in a multimodal tool-result
envelope is the wrong move: it trips HTTP 404 / 400 at the provider
boundary (e.g. 'No endpoints found that support image input') and the
agent loop reports a hard tool failure for what should have been a
simple capture.
The reporter on #24015 hit this with:
model:
default: tencent/hy3-preview # no vision support
provider: openrouter
auxiliary:
vision:
provider: openrouter
model: google/gemini-2.5-flash # explicitly configured
…and observed:
computer_use(action='capture', mode='som')
→ ⚠️ API call failed (attempt1/3): NotFoundError [HTTP 404]
🔌 Provider: openrouter Model: tencent/hy3-preview
📝 Error: HTTP 404: No endpoints found that support image input
Fix: in tools/computer_use/tool.py::_capture_response, after a
screenshot is captured (modes 'som' / 'vision'), consult the routing
helper introduced earlier in this branch. When it says 'route to aux',
materialise the PNG to $HERMES_HOME/cache/vision/, run vision_analyze
on it (which honours auxiliary.vision via the standard async_call_llm
task='vision' router), and return a text-only JSON tool result that
embeds the analysis alongside the existing AX/SOM index. The main
model never sees the pixels — it sees an actionable text description
plus the same set-of-mark element index it normally uses.
The two new helpers (_should_route_through_aux_vision,
_route_capture_through_aux_vision) keep the policy and the IO
separated so each can be tested in isolation. Both fail open: if the
config import fails, if the aux call raises, or if the analysis is
empty, we fall back to the existing multimodal envelope so the
behaviour is at worst the pre-fix status quo. Temp screenshot files
are cleaned up unconditionally in a finally block — even on aux call
failure — to avoid leaving residue under cache/vision/.
The end-to-end regression for #24015 is added in the next commit.
