hermes-agent/tools/computer_use
Koho Zheng f2a4ca9637 fix(computer-use): normalize cua-driver result envelopes
cua-driver 0.7.x can return list_windows/list_apps payloads under
structuredContent.windows, data.windows, data._legacy_windows, or
top-level windows/_legacy_windows (direct CLI responses). The wrapper
only read structuredContent.windows, so discovery came back empty
(capture 0x0, list_apps []) while raw cua-driver calls worked.

- add _windows_from_tool_result(): walks the known envelope shapes in
  priority order, skipping empty higher-priority envelopes
- route _load_windows() (MCP + CLI re-fetch paths) through the helper,
  covering capture() and focus_app()
- harden _ingest_windows(): skip non-dict members, normalize untrusted
  app_name/title/z_index fields
- list_apps(): prefer structuredContent.apps, fall through populated
  data/top-level envelopes, derive unique apps from window-shaped
  payloads via _apps_from_windows(), keep the text-line fallback last
- tests for every envelope shape, precedence, malformed records, and
  app derivation

Salvaged from #63037 (Reaper-Legion), which itself preserved the
original implementation from #57961 (kohoj); #73007 (umi008)
independently proposed the same normalization later.

Fixes #57905

Co-authored-by: Reaper <248977840+Reaper-Forge@users.noreply.github.com>
Co-authored-by: Ulises Millan Guerrero <ulises.millanguerrero@gmail.com>
2026-07-29 11:08:58 -07:00
..
__init__.py feat(computer-use): cua-driver backend, universal any-model schema 2026-05-08 11:07:38 -07:00
backend.py feat(computer_use): follow cua-driver's verify → escalate ladder (#67123) 2026-07-18 13:59:35 -07:00
cua_backend.py fix(computer-use): normalize cua-driver result envelopes 2026-07-29 11:08:58 -07:00
doctor.py fix: repair sweep fallout — duplicate encoding kwargs, non-subprocess call sites, kwarg-snapshot tests 2026-07-24 11:45:57 -07:00
permissions.py fix: add explicit UTF-8 encoding to all subprocess text=True calls (#53428) 2026-07-24 11:45:57 -07:00
schema.py feat(computer_use): follow cua-driver's verify → escalate ladder (#67123) 2026-07-18 13:59:35 -07:00
tool.py fix(computer_use): stop the cua-driver child on exit 2026-07-26 03:38:23 -05:00
vision_routing.py fix(computer_use): honor custom vision routing 2026-06-07 02:09:20 -07:00