mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-11 03:31:55 +00:00
feat(computer-use): cua-driver backend, universal any-model schema
Background macOS desktop control via cua-driver MCP — does NOT steal the user's cursor or keyboard focus, works with any tool-capable model. Replaces the Anthropic-native `computer_20251124` approach from the abandoned #4562 with a generic OpenAI function-calling schema plus SOM (set-of-mark) captures so Claude, GPT, Gemini, and open models can all drive the desktop via numbered element indices. - `tools/computer_use/` package — swappable ComputerUseBackend ABC + CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary). - Universal `computer_use` tool with one schema for all providers. Actions: capture (som/vision/ax), click, double_click, right_click, middle_click, drag, scroll, type, key, wait, list_apps, focus_app. - Multimodal tool-result envelope (`_multimodal=True`, OpenAI-style `content: [text, image_url]` parts) that flows through handle_function_call into the tool message. Anthropic adapter converts into native `tool_result` image blocks; OpenAI-compatible providers get the parts list directly. - Image eviction in convert_messages_to_anthropic: only the 3 most recent screenshots carry real image data; older ones become text placeholders to cap per-turn token cost. - Context compressor image pruning: old multimodal tool results have their image parts stripped instead of being skipped. - Image-aware token estimation: each image counts as a flat 1500 tokens instead of its base64 char length (~1MB would have registered as ~250K tokens before). - COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset is active. - Session DB persistence strips base64 from multimodal tool messages. - Trajectory saver normalises multimodal messages to text-only. - `hermes tools` post-setup installs cua-driver via the upstream script and prints permission-grant instructions. - CLI approval callback wired so destructive computer_use actions go through the same prompt_toolkit approval dialog as terminal commands. - Hard safety guards at the tool level: blocked type patterns (curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash, force delete, lock screen, log out). - Skill `apple/macos-computer-use/SKILL.md` — universal (model-agnostic) workflow guide. - Docs: `user-guide/features/computer-use.md` plus reference catalog entries. 44 new tests in tests/tools/test_computer_use.py covering schema shape (universal, not Anthropic-native), dispatch routing, safety guards, multimodal envelope, Anthropic adapter conversion, screenshot eviction, context compressor pruning, image-aware token estimation, run_agent helpers, and universality guarantees. 469/469 pass across tests/tools/test_computer_use.py + the affected agent/ test suites. - `model_tools.py` provider-gating: the tool is available to every provider. Providers without multi-part tool message support will see text-only tool results (graceful degradation via `text_summary`). - Anthropic server-side `clear_tool_uses_20250919` — deferred; client-side eviction + compressor pruning cover the same cost ceiling without a beta header. - macOS only. cua-driver uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that can break on any macOS update. Pin with HERMES_CUA_DRIVER_VERSION. - Requires Accessibility + Screen Recording permissions — the post-setup prints the Settings path. Supersedes PR #4562 (pyautogui/Quartz foreground backend, Anthropic- native schema). Credit @0xbyt4 for the original #3816 groundwork whose context/eviction/token design is preserved here in generic form.
This commit is contained in:
parent
474d1e812b
commit
850413f120
23 changed files with 2861 additions and 27 deletions
|
|
@ -150,6 +150,31 @@ def _append_text_to_content(content: Any, text: str, *, prepend: bool = False) -
|
|||
return text + rendered if prepend else rendered + text
|
||||
|
||||
|
||||
def _strip_image_parts_from_parts(parts: Any) -> Any:
|
||||
"""Strip image parts from an OpenAI-style content-parts list.
|
||||
|
||||
Returns a new list with image_url / image / input_image parts replaced
|
||||
by a text placeholder, or None if the list had no images (callers
|
||||
skip the replacement in that case). Used by the compressor to prune
|
||||
old computer_use screenshots.
|
||||
"""
|
||||
if not isinstance(parts, list):
|
||||
return None
|
||||
had_image = False
|
||||
out = []
|
||||
for part in parts:
|
||||
if not isinstance(part, dict):
|
||||
out.append(part)
|
||||
continue
|
||||
ptype = part.get("type")
|
||||
if ptype in ("image", "image_url", "input_image"):
|
||||
had_image = True
|
||||
out.append({"type": "text", "text": "[screenshot removed to save context]"})
|
||||
else:
|
||||
out.append(part)
|
||||
return out if had_image else None
|
||||
|
||||
|
||||
def _truncate_tool_call_args_json(args: str, head_chars: int = 200) -> str:
|
||||
"""Shrink long string values inside a tool-call arguments JSON blob while
|
||||
preserving JSON validity.
|
||||
|
|
@ -578,10 +603,12 @@ class ContextCompressor(ContextEngine):
|
|||
if msg.get("role") != "tool":
|
||||
continue
|
||||
content = msg.get("content") or ""
|
||||
# Skip multimodal content (list of content blocks)
|
||||
# Multimodal content — dedupe by the text summary if available.
|
||||
if isinstance(content, list):
|
||||
continue
|
||||
if not isinstance(content, str):
|
||||
# Multimodal dict envelopes ({_multimodal: True, content: [...]}) and
|
||||
# other non-string tool-result shapes can't be hashed/deduped by text.
|
||||
continue
|
||||
if len(content) < 200:
|
||||
continue
|
||||
|
|
@ -599,8 +626,20 @@ class ContextCompressor(ContextEngine):
|
|||
if msg.get("role") != "tool":
|
||||
continue
|
||||
content = msg.get("content", "")
|
||||
# Skip multimodal content (list of content blocks)
|
||||
# Multimodal content (base64 screenshots etc.): strip the image
|
||||
# payload — keep a lightweight text placeholder in its place.
|
||||
# Without this, an old computer_use screenshot (~1MB base64 +
|
||||
# ~1500 real tokens) survives every compression pass forever.
|
||||
if isinstance(content, list):
|
||||
stripped = _strip_image_parts_from_parts(content)
|
||||
if stripped is not None:
|
||||
result[i] = {**msg, "content": stripped}
|
||||
pruned += 1
|
||||
continue
|
||||
if isinstance(content, dict) and content.get("_multimodal"):
|
||||
summary = content.get("text_summary") or "[screenshot removed to save context]"
|
||||
result[i] = {**msg, "content": f"[screenshot removed] {summary[:200]}"}
|
||||
pruned += 1
|
||||
continue
|
||||
if not isinstance(content, str):
|
||||
continue
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue