hermes-agent/tools/computer_use
Jeff b062a5d4b9
fix(tools): harden Windows computer_use for daily co-located use
Three failure modes found preparing for real daytime use on a shared
desktop:

1. Vision-node outage broke captures outright. When aux-vision routing
   is requested (the main model cannot consume images) and the aux call
   fails, the old fallthrough returned the multimodal envelope - putting
   a screenshot in front of a text-only model and erroring the capture.
   Degrade to the AX/SOM text payload instead (vision_unavailable flag
   set): element-index actions keep working blind until vision returns.

2. Stale coordinates after a window move. Element bounds are absolute
   screen coords frozen at capture time; dragging the window between
   capture and click landed clicks on whatever sat at the old position.
   Track the captured window rect and translate element centers by the
   origin delta; a resize (interior layout changed) fails with an
   explicit re-capture message instead of guessing.

3. Input collisions with an active user. Synthetic input lands in
   whatever has focus; injecting mid-keystroke sprays input across both
   parties' targets. All input actions now wait for a short user-idle
   window (HERMES_COMPUTER_USE_IDLE_WAIT, default 1.5s, 0 disables),
   capped at 8s so the agent yields but never deadlocks.

Routing tests updated for the new degradation contract; new tests cover
all three behaviors and run dependency-free off-Windows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 06:19:20 -07:00
..
__init__.py feat(computer-use): cua-driver backend, universal any-model schema 2026-05-08 11:07:38 -07:00
backend.py fix(computer-use): add set_value to ComputerUseBackend ABC and _NoopBackend stub 2026-05-22 01:14:15 -07:00
cua_backend.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00
overlay.py feat(tools): on-screen overlay for Windows computer_use 2026-06-15 06:19:20 -07:00
schema.py feat(tools): add Windows UIA backend for computer_use 2026-06-15 06:19:20 -07:00
tool.py fix(tools): harden Windows computer_use for daily co-located use 2026-06-15 06:19:20 -07:00
vision_routing.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00
windows_backend.py fix(tools): harden Windows computer_use for daily co-located use 2026-06-15 06:19:20 -07:00