hermes-agent/tools/computer_use
Jeff 42fc7d86d3
feat(tools): on-screen overlay for Windows computer_use
Visible "PC use mode": a persistent banner pill while desktop control
is active, the numbered SOM element boxes mirrored onto the real
screen after each capture, click ripples / drag arrows where actions
land, and short action flashes (typing, key combos, scroll).

overlay.py runs as a subprocess: a fullscreen transparent
click-through topmost tkinter window spanning the virtual desktop,
driven over localhost UDP, excluded from screen capture via
SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE) so the model's own
screenshots never contain it (verified by pixel-sampling a capture
taken while a box was on screen). The overlay returns foreground
focus after spawning, and the backend never targets the overlay
process as a capture subject. All overlay traffic is fire-and-forget:
any failure disables the overlay without affecting actions. Disable
with HERMES_COMPUTER_USE_OVERLAY=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-15 06:19:20 -07:00
..
__init__.py feat(computer-use): cua-driver backend, universal any-model schema 2026-05-08 11:07:38 -07:00
backend.py fix(computer-use): add set_value to ComputerUseBackend ABC and _NoopBackend stub 2026-05-22 01:14:15 -07:00
cua_backend.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00
overlay.py feat(tools): on-screen overlay for Windows computer_use 2026-06-15 06:19:20 -07:00
schema.py feat(tools): add Windows UIA backend for computer_use 2026-06-15 06:19:20 -07:00
tool.py feat(tools): add Windows UIA backend for computer_use 2026-06-15 06:19:20 -07:00
vision_routing.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00
windows_backend.py feat(tools): on-screen overlay for Windows computer_use 2026-06-15 06:19:20 -07:00