mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-06-23 10:42:00 +00:00
807b696295
Vision mode called a `screenshot` MCP tool that cua-driver dropped in 0.5.x (full-window PNG capture was folded into `get_window_state`). The driver replied "Unknown tool: screenshot", so `images` came back empty, `png_b64` stayed None, and capture returned a 0x0 result with no image on every call. `som`/`ax` were unaffected because they already use `get_window_state`, which masked the regression. Route vision by capability: - driver advertises `screenshot` (older builds) -> use it (no AX walk) - otherwise -> call `get_window_state` but discard the AX tree/elements, returning only the PNG so vision stays free of element noise - capabilities not yet discovered -> try `screenshot`, fall back to `get_window_state` on an empty image, so the path self-heals Add `_image_from_tool_result` to pull the PNG from either an MCP image content-part or `structuredContent.screenshot_png_b64`, and use it on the som path too so the image won't silently drop on driver builds that deliver it via structuredContent instead of a content part. Verified live (vision: 1568x954, 0 elements; som: image + 527 elements) and with unit coverage of all four routing cases. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| backend.py | ||
| cua_backend.py | ||
| doctor.py | ||
| schema.py | ||
| tool.py | ||
| vision_routing.py | ||