hermes-agent/tools/computer_use
Adrian Lastra 13b75e73ff fix(computer_use): re-fetch via CLI when MCP returns silent-empty captures
The first fix handled the EAGAIN McpError path. But the persistent MCP
session (long-running gateway/desktop worker) has a second failure mode:
list_windows or get_window_state 'succeed' over MCP yet return a
degenerate/empty payload (no windows, or no screenshot + blank tree)
WITHOUT raising — typically when the bridge reconnected mid-call and
dropped the heavy response. That surfaced to the model as a silent 0x0
capture with no error and no fallback firing (0.00s empty return).

Fix: detect empty results in capture() and re-fetch over the CLI
transport before giving up:
  - empty list_windows -> CLI re-fetch the window list
  - empty get_window_state (som/ax) -> CLI re-fetch the AX tree + screenshot
  - empty screenshot (vision) -> CLI re-fetch get_window_state for the PNG

Adds 2 regression tests. Full suite: 83 passed.
2026-07-05 05:36:31 -07:00
..
__init__.py feat(computer-use): cua-driver backend, universal any-model schema 2026-05-08 11:07:38 -07:00
backend.py feat(computer_use): cross-platform cua-driver (macOS/Windows/Linux) 2026-06-22 06:42:30 -07:00
cua_backend.py fix(computer_use): re-fetch via CLI when MCP returns silent-empty captures 2026-07-05 05:36:31 -07:00
doctor.py feat(computer_use): disable cua-driver telemetry by default, add opt-in (#50842) 2026-06-22 09:57:16 -07:00
permissions.py revert(windows): roll back terminal-popup PRs #53791 #53810 #53829 (#53853) 2026-06-27 15:59:00 -07:00
schema.py feat(computer-use): add whole-screen/desktop capture target 2026-06-22 12:21:58 -07:00
tool.py fix(computer_use): reconcile Linux gate with stale "gated off" comments 2026-06-22 06:42:30 -07:00
vision_routing.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00