mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-21 07:55:08 +00:00
Three failure modes found preparing for real daytime use on a shared desktop: 1. Vision-node outage broke captures outright. When aux-vision routing is requested (the main model cannot consume images) and the aux call fails, the old fallthrough returned the multimodal envelope - putting a screenshot in front of a text-only model and erroring the capture. Degrade to the AX/SOM text payload instead (vision_unavailable flag set): element-index actions keep working blind until vision returns. 2. Stale coordinates after a window move. Element bounds are absolute screen coords frozen at capture time; dragging the window between capture and click landed clicks on whatever sat at the old position. Track the captured window rect and translate element centers by the origin delta; a resize (interior layout changed) fails with an explicit re-capture message instead of guessing. 3. Input collisions with an active user. Synthetic input lands in whatever has focus; injecting mid-keystroke sprays input across both parties' targets. All input actions now wait for a short user-idle window (HERMES_COMPUTER_USE_IDLE_WAIT, default 1.5s, 0 disables), capped at 8s so the agent yields but never deadlocks. Routing tests updated for the new degradation contract; new tests cover all three behaviors and run dependency-free off-Windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| backend.py | ||
| cua_backend.py | ||
| overlay.py | ||
| schema.py | ||
| tool.py | ||
| vision_routing.py | ||
| windows_backend.py | ||