Commit graph

26 commits

Author SHA1 Message Date
Teknium
40ec417881
Merge remote-tracking branch 'origin/main' into tests/prune-low-value
# Conflicts:
#	tests/hermes_cli/test_install_cua_driver.py
#	tests/run_agent/test_codex_app_server_integration.py
#	tests/test_tui_gateway_server.py
#	tests/tools/test_computer_use_delivery_ladder.py
#	tests/tools/test_zombie_process_cleanup.py
2026-07-29 14:12:25 -07:00
Teknium
39975613b1
test: prune wave 2 + speed fixes — 28,106 → 19,757 test functions, suite wall 315s → 294s
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.

Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
  (DEFAULT_CONFIG smart-approval leaked in) — pinned approval
  mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
  silently probed live provider catalogs (~2s/test) — stubbed
  cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
  shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
  MagicMock agents — interrupt.side_effect now clears _running_agents
  (22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
  (24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
  patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
  5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
  with rationale comment — the watchdog-dump assertion needs the loop
  blocked well past deadline+grace under parallel load (flaked once in
  the 40-worker verification run at a 0.2s margin).

Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
2026-07-29 13:39:40 -07:00
Francesco Bonacci
847e401b74 feat(computer_use): align cua-driver 0.9 contracts
Salvaged from PR #67807 by @f-trycua onto current main.

- Foreground gate: discover delivery_mode support from the live tools/list
  inputSchema.properties (fail closed), not the never-shipped
  input.delivery_mode capability token
- bring_to_front: standalone strict-schema MCP tool (inject_session=False),
  separate approval scope, requires foreground
- Verdict precedence: confirmed > unverifiable (verify before retry) >
  suspected_noop/refusal (escalate); surfaced as explicit verdict field
- Typed cua_browser_* route inside computer_use (browser_route.py) with
  exact-binding, adapter-injected session, snapshot-scoped refs
- Per-Hermes-session backend isolation + release_computer_use_session seam
  wired into AIAgent.close()
- Recorded 0.9 tools/list fixture replaces fabricated capability tokens
2026-07-29 12:19:37 -07:00
Teknium
c3ffe27383 test: align doctor spawn fixture with stricter health_report validation
Some checks are pending
CI / Detect affected areas (push) Waiting to run
CI / Python tests (push) Blocked by required conditions
CI / Python lints (push) Blocked by required conditions
CI / JS & TS checks (push) Blocked by required conditions
CI / Desktop E2E (push) Blocked by required conditions
CI / Docs Site (push) Blocked by required conditions
CI / Deny unrelated histories (push) Blocked by required conditions
CI / Check contributors (push) Blocked by required conditions
CI / Check uv.lock (push) Blocked by required conditions
CI / Check no committed infographics (push) Blocked by required conditions
CI / package-lock.json diff (push) Blocked by required conditions
CI / Lint Docker scripts (push) Blocked by required conditions
CI / Build&Test Docker image (push) Blocked by required conditions
CI / Supply-chain scan (push) Blocked by required conditions
CI / Review label gate (push) Blocked by required conditions
CI / OSV scan (push) Waiting to run
CI / CI review comment (live) (push) Blocked by required conditions
CI / All required checks pass (push) Blocked by required conditions
CI / CI timing report (push) Blocked by required conditions
Docker Build, Test, and Publish / build (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Waiting to run
Docker Build, Test, and Publish / build (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Docker Build, Test, and Publish / publish (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Blocked by required conditions
Docker Build, Test, and Publish / publish (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
Docker Build, Test, and Publish / merge (push) Blocked by required conditions
auto-fix lint issues & formatting / Generate eslint --fix patch (push) Waiting to run
auto-fix lint issues & formatting / Apply patch (push) Blocked by required conditions
The #74187 doctor fallback requires a checks list in schema_version=1
payloads; add it to the salvaged Windows console-hiding test fixture.
2026-07-29 11:35:33 -07:00
Teknium
28cc6599df test: pin cua-driver resolver in CLI-fallback sanitization test
The test passed on dev boxes where a cua-driver binary is on PATH but
failed in hermetic CI: _call_tool_via_cli hit resolve_cua_driver_cmd()'s
install-hint early exit before reaching the spawn-env assertion. Pin the
resolver so the test exercises the code path it actually asserts.
2026-07-29 11:35:33 -07:00
Teknium
16720dc45b fix(computer-use): hide the --no-overlay help probe console too
Follow-up to the #62821 salvage: the _cua_driver_supports_no_overlay
--help probe is another Windows-reachable spawn; give it the same
windows_hide_flags() treatment. Also adjust the status test to stub
_resolve_driver_cmd (permissions.py resolves via that helper, not
shutil.which).
2026-07-29 11:35:33 -07:00
motoblurr
7bc9956660 fix(computer-use): normalize Windows manifest paths in WSL
A Windows-installed cua-driver can return an absolute
``C:\Users\...\cua-driver.exe`` mcp_invocation.command to a Hermes
process running inside WSL. POSIX spawning can't use the raw Windows
string even though the binary is reachable through DrvFS. Translate
``<drive>:\...`` to ``/mnt/<drive>/...`` in _resolve_mcp_invocation
(before the path-separator check, since backslash is not a separator
on POSIX), only when actually running under WSL.

Salvaged from #63532 by @motoblurr (original commit carried a
placeholder 'Hermes Agent <hermes@local>' identity; re-authored).
Fixes #63938 premise.
2026-07-29 11:35:33 -07:00
ZundamonnoVRChatkaisetu
50e27abdd3 fix(computer-use): hide Windows cua-driver subprocess consoles
Apply windows_hide_flags() (CREATE_NO_WINDOW; 0 on POSIX) at the
Windows-reachable cua-driver subprocess boundaries: manifest probe,
update checker, CLI fallback transport, doctor health-report spawn,
and the permissions/status runner. Prevents OpenConsole/Windows
Terminal windows flashing into the foreground when spawned from
GUI-backed Gateway/Desktop processes.

The env-probe half of the original PR was already implemented on main
and is not re-applied here.

Salvaged from #62821 by @ZundamonnoVRChatkaisetu (original commits
carried a placeholder 'Claude Code Enterprise' identity; re-authored
to the contributor's GitHub identity).
2026-07-29 11:35:33 -07:00
Teknium
21d1d08a2f test: convert doctor env-sanitization check from source-inspection to behavioral
The old assertion read _drive_health_report's source text for the
_sanitized_cua_env() call — a banned source-reading test that broke when
the spawn moved into _open_mcp with identical runtime behavior. Now
intercepts subprocess.Popen at the _open_mcp seam and asserts the env it
actually receives strips secrets and applies the telemetry opt-out.
2026-07-29 11:09:08 -07:00
monerostar
b9215f5bc9 fix(computer_use): surface CLI --version when health_report version lies 2026-07-29 11:09:08 -07:00
camaleonidas
c6db7b0f4f fix(computer-use): fallback doctor when cua-driver health_report is unclassified
cua-driver 0.10.0 marks health_report as risk.class=unclassified and denies
the MCP call with isError. Hermes doctor previously treated the bare
{exit_code:1} structuredContent as a real report and printed
"cua-driver ? on ? — ?" with exit 1.

Detect isError / non-schema payloads, raise HealthReportUnavailable, and
compose a schema_version=1 report from working probes (check_permissions,
list_apps, CLI --version/doctor). Prefer real health_report when present.

Tests cover unclassified denial, schema preference, and overall mapping.
2026-07-29 11:09:08 -07:00
Brooklyn Nicholson
4e49af94be fix(computer_use): stop the cua-driver child on exit
CuaDriverBackend caches a long-lived cua-driver subprocess for the life of
the Hermes process, and stop() was never called from anywhere — the driver
outlived the session that spawned it. #69903 stopped the orphan from pegging
a core by disabling the cursor overlay, but left the process behind; this is
item 3 of #28152 ("Hermes does not keep the driver alive after tool
completion").

Register an atexit hook, mirroring browser_tool's
atexit.register(_emergency_cleanup_all_sessions). atexit only, no signal
handlers, for the prompt_toolkit reason documented there. reset_backend_for_tests
now reuses the same teardown instead of repeating it.
2026-07-26 03:38:23 -05:00
Brooklyn Nicholson
cc1765dce2 Merge remote-tracking branch 'origin/main' into bb/computer-use-perf
# Conflicts:
#	hermes_cli/config.py
#	tools/computer_use/cua_backend.py
2026-07-23 01:39:21 -05:00
Brooklyn Nicholson
69b97a97f7 Merge remote-tracking branch 'origin/main' into bb/salvage-53841-no-overlay
# Conflicts:
#	tools/computer_use/cua_backend.py
2026-07-23 01:26:21 -05:00
Brooklyn Nicholson
8b6d34ad85 test(computer_use): pin resolved driver for update-check tests
cua_driver_update_check now short-circuits to None when no driver
resolves; CI has none installed, so pin resolve_cua_driver_cmd in the
update-check and env-sanitization tests.
2026-07-23 01:10:19 -05:00
Brooklyn Nicholson
12ad13ddca perf(computer_use): cap capture size and cache vision routing
Cut steady-state Computer Use latency without changing default behavior
or waiting on cua-driver:

- Cap screenshots via set_config(max_image_dimension) on session start
  (config: computer_use.max_image_dimension, default 1456)
- Cache aux-vision routing per (provider, model) so captures skip
  repeated load_config()
- Add computer_use.capture_after_mode (default som) so users can opt
  follow-ups down to ax (elements only) for speed
2026-07-23 00:52:43 -05:00
Brooklyn Nicholson
f957fe3760 fix(computer_use): default --no-overlay on macOS for idle CPU
Auto-detect now disables the cursor overlay on darwin as well as
headless/WSL2 Linux. After start_session, also call
set_agent_cursor_enabled(false) when the policy is on so older drivers
without --no-overlay still tear the overlay down.

Co-authored-by: David Metcalfe <80915+DavidMetcalfe@users.noreply.github.com>
2026-07-23 00:45:26 -05:00
David Metcalfe
3d84689714 fix(computer_use): address sweeper feedback on --no-overlay subprocess + manifest probe
The hermes-sweeper review #4701565902 (2026-07-15) flagged two
consistency issues in `_cua_driver_supports_no_overlay` and one
additive-config concern:

1. `cua-backend.py:260` — the `cua-driver --help` support probe
   inherited the full parent environment. cua-driver is a third-party
   binary; every other spawn site in this file (manifest probe at
   `:214`, MCP spawn at `:697`, install probe at `:997`) uses
   `_sanitize_subprocess_env(cua_driver_child_env())`. The `--help`
   probe should match. This was a low-impact leak (only help output
   exits), but inconsistency is the wrong default for a third-party
   subprocess.

2. `cua_backend.py:238` — when the manifest returned a `command`
   different from the input `driver_cmd` parameter (e.g. a relocated
   executable at `/opt/relocated/cua-driver` while the system binary
   is at `/usr/bin/cua-driver`), the support probe ran against
   `_CUA_DRIVER_CMD` (the default) instead of the manifest-discovered
   `command`. Two failure modes:
   - The wrapper binary supports `--no-overlay` but the system binary
     doesn't → probe returns False → overlay kept despite capability.
   - The system binary supports `--no-overlay` but the wrapper doesn't
     → probe returns True → MCP spawn crashes on the unknown flag.

3. The original commit bumped `_config_version` 31→32 for an additive
   default (`computer_use.no_overlay: None`). AGENTS.md specifies that
   additive defaults in existing sections are handled by deep merge
   and should NOT trigger a version bump. After cherry-picking onto
   current `origin/main` (which is already at 33), the bump is
   effectively dropped — resolved to main's 33.

Changes:

- Add `env=_sanitize_subprocess_env(cua_driver_child_env())` to the
  `--help` subprocess (with the same import + rationale comment as
  the manifest probe).
- Pass `driver_cmd=command` (or `driver_cmd=driver_cmd` for the
  fallback path) into `_mcp_args_with_overlay_flag`, so the support
  probe runs against the binary that will actually be launched.

Tests (3 new):

- `test_help_probe_passes_sanitized_env` — verifies `subprocess.run`
  is called with an `env=` kwarg.
- `test_manifest_command_drives_support_probe` — verifies the probe
  runs against the manifest command when it differs from the input
  driver_cmd.
- `test_fallback_uses_input_driver_cmd_for_support_probe` — verifies
  the fallback path (no command in manifest) uses the input
  driver_cmd.
- `test_probe_distinguishes_support_between_binaries` — sanity check
  that the lru_cache key on `driver_cmd` prevents cross-binary
  cache leakage.

File-revert negative test confirmed all three of the new
"manifest/probe" tests are load-bearing: with the pre-fix code, they
fail (probe runs against the default binary instead of the resolved
one); with the fix, they pass. 20/20 tests in
`tests/computer_use/test_cua_no_overlay.py` green.
`TestMcpInvocationResolution` (8/8) still green.

Refs: sweeper review #4701565902
2026-07-23 00:44:43 -05:00
David Metcalfe
8d4f7a0002 fix(computer_use): refine auto-detect to headless/WSL2 only, add driver version probe
Address review feedback from cross-vendor review (Flash + GPT-OSS):

1. Auto-detect now checks for headless Linux (no DISPLAY), WSL2
   (/proc/version contains 'microsoft'), instead of all Linux.
   Desktop Linux with a compositor keeps the overlay.

2. Add _cua_driver_supports_no_overlay() that probes cua-driver --help
   to check if the flag is supported. Older drivers (< 0.6.x) reject
   unknown flags, so passing --no-overlay would crash the MCP spawn.

3. Update tests to cover headless vs desktop Linux, WSL2 detection,
   version probe, and the unsupported-driver fallback path.
2026-07-23 00:44:43 -05:00
David Metcalfe
f43ff5b4bb fix(computer_use): mock _cua_no_overlay in existing tests, fix platform-dependent assertions
- Add autouse fixture to TestMcpInvocationResolution to disable
  --no-overlay flag so existing tests assert baseline args
- Make test_config_load_failure_fails_safe and test_missing_section_enables
  platform-aware (Linux auto-detect returns True, macOS/Windows False)
2026-07-23 00:44:43 -05:00
David Metcalfe
8ceada6e30 fix(computer_use): pass --no-overlay to cua-driver on Linux/WSL2 to prevent idle CPU
cua-driver's cursor overlay rendering loop can consume CPU indefinitely
when idle (#28152, #47032). On Linux/WSL2, the overlay serves no visual
purpose and the rendering path is the primary source of idle CPU usage.

Add computer_use.no_overlay config option (default: auto-detect) that
passes --no-overlay to cua-driver when enabled. Auto-detection disables
the overlay on Linux (covers WSL2, headless, containers) where it has no
benefit, and keeps it enabled on macOS/Windows where it is visually
useful.

Refs: #28152, #47032
2026-07-23 00:44:43 -05:00
Tianqing Yun
2f9d88caee fix: resolve cua-driver across computer-use surfaces 2026-07-23 00:42:46 -05:00
Teknium
0823230545
fix(computer-use): sanitize env on the 4 remaining cua-driver spawn sites (#59165)
PR #58889 fixed the CLI-fallback transport; review of that fix found the
same leak class at four sibling spawn sites of the third-party cua-driver
binary:

- _resolve_mcp_invocation (cua-driver manifest): no env= at all — full
  parent environment inherited
- cua_driver_update_check (check-update --json): telemetry env but no
  secret sanitization
- doctor._drive_health_report (<binary> mcp Popen): telemetry env only
- permissions._run (every macOS/Linux permission probe): telemetry env only

All now route through _sanitize_subprocess_env(cua_driver_child_env()),
matching the sanctioned MCP spawn and the #53503/#55709/#58889 strip-by-
default policy for non-terminal spawns. Sanitization degrades gracefully
(falls back to the telemetry env) so doctor/permission probes never break
on an import error.

4 regression tests covering each site.
2026-07-05 14:48:20 -07:00
srojk34
f10851e3fd fix(computer-use): sanitize subprocess env in cua-driver CLI fallback transport
_CuaDriverSession._call_tool_via_cli() (the EAGAIN/silent-empty MCP
fallback transport) invokes `cua-driver call <tool> <json>` via
subprocess.run() with no env= argument, so the third-party cua-driver
binary inherits the full, unsanitized parent environment. The primary
MCP spawn site (_lifecycle_coro) already applies
_sanitize_subprocess_env(cua_driver_child_env()) before opening the
stdio client, per the same policy #53503/#55709 established for other
subprocess spawn points — this fallback path, added alongside the
EAGAIN/silent-empty-capture hardening, missed it.

Fix: apply the same env=_sanitize_subprocess_env(cua_driver_child_env())
to the subprocess.run() call in _call_tool_via_cli(), mirroring the
sanctioned spawn site exactly (telemetry policy applied first, then
Hermes-managed secrets filtered).
2026-07-05 13:57:20 -07:00
Teknium
f1e6d39a74
feat(computer_use): disable cua-driver telemetry by default, add opt-in (#50842)
* feat(computer_use): disable cua-driver telemetry by default, add opt-in

cua-driver ships anonymous PostHog usage telemetry ENABLED by default
upstream (fires cua_driver_install / cua_driver_doctor events to
eu.i.posthog.com). Hermes now disables it for our users unless they
explicitly opt in.

- New config key `computer_use.cua_telemetry` (default false) in
  DEFAULT_CONFIG.
- `cua_backend.cua_driver_child_env()` injects
  `CUA_DRIVER_RS_TELEMETRY_ENABLED=0` into the child env when telemetry is
  disabled (the default); leaves the var untouched on opt-in so the driver
  uses its own default. Reads config fail-safe — any error defaults to
  telemetry off.
- Routed every cua-driver spawn site through the policy: MCP backend
  (StdioServerParameters env), `cua_driver_update_check`, doctor's
  health_report Popen, the install.sh/install.ps1 runner, and the
  `--version` / status probes.
- Docs: new Telemetry subsection in computer-use.md (EN).
- Tests: tests/computer_use/test_cua_telemetry.py — default disables,
  explicit-false disables, opt-in leaves var untouched, config-failure
  fails safe, inherited-enabled is overridden off.

Verified live on Linux against the real cua-driver-rs 0.6.0 binary: with
the var=0 the driver reports "telemetry: disabled via
CUA_DRIVER_RS_TELEMETRY_ENABLED" and sends no event; with it unset it logs
"sending event: cua_driver_doctor". 213 computer_use + install tests green.

* fix(dashboard): fold computer_use config category into agent tab

The new computer_use.cua_telemetry key created a single-field dashboard
config category, tripping test_no_single_field_categories (web_server's
invariant that categories with <2 fields must be merged to avoid tab
sprawl). Add computer_use -> agent to _CATEGORY_MERGE, matching the
existing onboarding/telegram single-field folds.
2026-06-22 09:57:16 -07:00
Francesco Bonacci
f2e37549c6 feat(computer_use): cross-platform cua-driver (macOS/Windows/Linux)
Make the computer_use toolset platform-agnostic by driving cua-driver on
macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces
(capability discovery, structuredContent AX tree, opaque element_token,
click button enum, explicit mimeType, machine-readable manifest,
structured list_windows, structured health_report), each degrading
gracefully on older drivers.

Adds `hermes computer-use doctor` (drives cua-driver health_report with a
per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full
typed wrappers for the previously-uncovered cua-driver tools plus a generic
call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware
system-prompt guidance (host-deterministic, cache-safe), and honors
HERMES_CUA_DRIVER_CMD end-to-end.

Replaces the macOS-only skills/apple/macos-computer-use skill with a
cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans
docs.

Supersedes #44221 (Windows-enablement salvage of #30660).

Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-06-22 06:42:30 -07:00