hermes-agent/tools/computer_use
Adrian Lastra 7af9abd174 fix(computer_use): fall back to CLI transport when cua-driver MCP bridge hits EAGAIN
The cua-driver MCP stdio bridge intermittently (and on some machines
persistently) fails to forward heavier calls like get_window_state to
the daemon with POSIX EAGAIN — 'daemon transport error forwarding
get_window_state: Resource temporarily unavailable (os error 35)'.
The wrapper surfaced this as an empty 0x0 capture, so computer_use
returned blank screenshots even though the display, permissions, and
the daemon were all healthy (the direct 'cua-driver call' CLI path
worked fine throughout).

Fix: when the MCP path raises the transient/transport error, fall back
to the 'cua-driver call' subprocess transport, which talks to the
daemon over a different socket. The CLI fallback routes get_window_state
screenshots to a temp file via screenshot_out_file (tiny JSON response
instead of a multi-MB base64 blob that congests the socket), reads the
PNG back, retries with backoff, and remaps the JSON into the same
{data, images, structuredContent, isError} shape the MCP path produces
so capture()/_action() are transport-agnostic.

Adds _is_transient_daemon_error() classifier and 3 regression tests.
Verified live: captures that returned 0x0 now return full
1567x905 screenshots with the AX element tree.
2026-07-05 05:36:31 -07:00
..
__init__.py feat(computer-use): cua-driver backend, universal any-model schema 2026-05-08 11:07:38 -07:00
backend.py feat(computer_use): cross-platform cua-driver (macOS/Windows/Linux) 2026-06-22 06:42:30 -07:00
cua_backend.py fix(computer_use): fall back to CLI transport when cua-driver MCP bridge hits EAGAIN 2026-07-05 05:36:31 -07:00
doctor.py feat(computer_use): disable cua-driver telemetry by default, add opt-in (#50842) 2026-06-22 09:57:16 -07:00
permissions.py revert(windows): roll back terminal-popup PRs #53791 #53810 #53829 (#53853) 2026-06-27 15:59:00 -07:00
schema.py feat(computer-use): add whole-screen/desktop capture target 2026-06-22 12:21:58 -07:00
tool.py fix(computer_use): reconcile Linux gate with stale "gated off" comments 2026-06-22 06:42:30 -07:00
vision_routing.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00