mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-21 07:55:08 +00:00
Visible "PC use mode": a persistent banner pill while desktop control is active, the numbered SOM element boxes mirrored onto the real screen after each capture, click ripples / drag arrows where actions land, and short action flashes (typing, key combos, scroll). overlay.py runs as a subprocess: a fullscreen transparent click-through topmost tkinter window spanning the virtual desktop, driven over localhost UDP, excluded from screen capture via SetWindowDisplayAffinity(WDA_EXCLUDEFROMCAPTURE) so the model's own screenshots never contain it (verified by pixel-sampling a capture taken while a box was on screen). The overlay returns foreground focus after spawning, and the backend never targets the overlay process as a capture subject. All overlay traffic is fire-and-forget: any failure disables the overlay without affecting actions. Disable with HERMES_COMPUTER_USE_OVERLAY=0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| backend.py | ||
| cua_backend.py | ||
| overlay.py | ||
| schema.py | ||
| tool.py | ||
| vision_routing.py | ||
| windows_backend.py | ||