mirrors/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-07-17 14:42:06 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6

fix(computer_use): parse (label) and = "value" AX element label forms

Browse source 
The SOM/AX element list dropped labels for two extremely common cua-driver
render forms, leaving the model unable to target elements by name:
  - [79] AXButton (Dark)              -> parenthesised label
  - [4]  AXStaticText = "Wi-Fi"       -> = "value" form
  - [92] AXPopUpButton = "Automatic"  -> = "value" form
The old regex only matched quoted "label" and id=Label, so System Settings
buttons/text/popups all surfaced with empty labels. That's why selecting the
macOS Appearance 'Dark' button by element index required guessing — the
labels weren't available to aim with.

Fix: extend _ELEMENT_LINE_RE to capture all four label forms (= "value",
"quoted", (parenthesised), id=Label), skipping a pure-digit (N) order number
in favour of the id= label. Verified live against System Settings: the
Appearance buttons now surface as Auto/Light/Dark.

Adds a regression test covering all label forms. Full suite: 84 passed.
This commit is contained in:
Adrian Lastra 2026-06-07 12:41:13 -04:00 • committed by Teknium
parent 13b75e73ff
commit 519ec7b3b3
2 changed files with 60 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

43
tools/computer_use/cua_backend.py
View file

@ -191,16 +191,31 @@ def _resolve_mcp_invocation(
# Regex to parse element lines from get_window_state AX tree markdown.
#
# Handles two output formats from different cua-driver versions:
# Classic: " - [N] AXRole \"label\""
# New: "[N] AXRole (order) id=Label"
# cua-driver renders each actionable node as one of:
# - [N] AXRole "label" (quoted label, classic)
# - [N] AXRole = "value" (value form, e.g. AXStaticText/AXPopUpButton)
# - [N] AXRole (label) (parenthesised label, e.g. AXButton (Dark))
# - [N] AXRole (order) id=Label (order number + id= label, newer builds)
# - [N] AXRole id=Label (id= label only)
# - [N] AXRole (no label)
# followed by trailing metadata like [help="..." actions=[...]].
#
# Group 1: element index
# Group 2: AX role
# Group 3: quoted label (classic format)
# Group 4: id= label (new format)
# Earlier the regex only matched the quoted and id= forms, so the very common
# `(label)` and `= "value"` forms (System Settings buttons, static text, popups)
# came back with an empty label — which made label-driven clicking impossible.
# A parenthesised group that is purely digits is an ORDER index, not a label, so
# it is excluded and we fall through to the id= label.
#
# Group 1: element index Group 2: AX role
# Groups 3-6: the label in value / quoted / paren / id= form (whichever matched)
_ELEMENT_LINE_RE = re.compile(
r'^\s*(?:-\s+)?\[(\d+)\]\s+(\w+)(?:\s+"([^"]*)"|(?:\s+\(\d+\))?\s+id=([^\s\[\]]*))?' ,
r'^\s*(?:-\s+)?\[(\d+)\]\s+(\w+)'
r'(?:'
r'\s*=\s*"([^"]*)"' # = "value"
r'|\s+"([^"]*)"' # "value"
r'|\s+\((?!\d+\))([^)]*)\)' # (value) but not a pure-digit (order) number
r')?'
r'(?:\s+(?:\(\d+\)\s+)?id=([^\s\[\]]+))?', # optional id=value (after an optional (order))
re.MULTILINE,
)
@ -323,15 +338,17 @@ def _parse_elements_from_tree(markdown: str) -> List[UIElement]:
``_parse_elements_from_structured`` — Surface 2 of #47072 prefers
that path).
Handles both the classic ``"label"``-quoted format and the newer
``id=Label`` format introduced in cua-driver v0.1.6. Bounds always
Captures the label whichever form cua-driver used: ``= "value"``,
``"quoted"``, ``(parenthesised)``, or ``id=Label``. Bounds always
come back ``(0, 0, 0, 0)`` because the markdown surface doesn't
carry them — yet another reason to prefer the structured path.
carry them — yet another reason to prefer the structured path;
element-index clicks don't need them (the driver resolves the index
to a frame internally).
"""
elements = []
for m in _ELEMENT_LINE_RE.finditer(markdown):
# group(3) = quoted label (classic); group(4) = id= label (new)
label = m.group(3) or m.group(4) or ""
# groups 3-6: value / quoted / paren / id= label (first non-None wins)
label = m.group(3) or m.group(4) or m.group(5) or m.group(6) or ""
elements.append(UIElement(
index=int(m.group(1)),
role=m.group(2),