hermes-agent/tools/computer_use
Jeff 17b35a7d95
fix(tools): release stuck input and prevent UIA index drift on Windows
Addresses review feedback on the Windows computer_use backend (#43927).

1. A failed click/drag/scroll left modifier keys - and, for drag, the
   mouse button - synthetically held down: the release ran after the
   action inside the same try block, so an injection error skipped it.
   Move the release into a finally so Ctrl/Alt/Shift and the button are
   always released and the cursor restored even when an injection raises.

2. capture (_walk_elements) and set_value (_control_at_index) each
   reimplemented the same BFS + interactability filter; if the two ever
   diverged, set_value would resolve an index to a different control than
   the capture advertised. Both now consume one _iter_interactable
   generator, so element #N is the same control in both paths.

3. That shared walk uses collections.deque.popleft() instead of the
   O(n) list.pop(0).

7 new dependency-free tests (modifier/button release on failure,
capture/set_value index agreement, BFS order, Text-pattern filter); they
pass off-Windows. Full computer_use suites: 137 passed on Windows 11.

Thanks to @Icather for spotting all three and proposing the fixes
(originally raised in #45976).

Suggested-by: ChengLong Han <97326386+Icather@users.noreply.github.com>
Co-authored-by: ChengLong Han <97326386+Icather@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 06:19:20 -07:00
..
__init__.py feat(computer-use): cua-driver backend, universal any-model schema 2026-05-08 11:07:38 -07:00
backend.py fix(computer-use): add set_value to ComputerUseBackend ABC and _NoopBackend stub 2026-05-22 01:14:15 -07:00
cua_backend.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00
overlay.py feat(tools): on-screen overlay for Windows computer_use 2026-06-15 06:19:20 -07:00
schema.py feat(tools): add Windows UIA backend for computer_use 2026-06-15 06:19:20 -07:00
tool.py fix(tools): harden Windows computer_use for daily co-located use 2026-06-15 06:19:20 -07:00
vision_routing.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00
windows_backend.py fix(tools): release stuck input and prevent UIA index drift on Windows 2026-06-15 06:19:20 -07:00