mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-04-25 00:51:20 +00:00
feat(computer-use): cua-driver backend, universal any-model schema
Background macOS desktop control via cua-driver MCP — does NOT steal the user's cursor or keyboard focus, works with any tool-capable model. Replaces the Anthropic-native `computer_20251124` approach from the abandoned #4562 with a generic OpenAI function-calling schema plus SOM (set-of-mark) captures so Claude, GPT, Gemini, and open models can all drive the desktop via numbered element indices. ## What this adds - `tools/computer_use/` package — swappable ComputerUseBackend ABC + CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary). - Universal `computer_use` tool with one schema for all providers. Actions: capture (som/vision/ax), click, double_click, right_click, middle_click, drag, scroll, type, key, wait, list_apps, focus_app. - Multimodal tool-result envelope (`_multimodal=True`, OpenAI-style `content: [text, image_url]` parts) that flows through handle_function_call into the tool message. Anthropic adapter converts into native `tool_result` image blocks; OpenAI-compatible providers get the parts list directly. - Image eviction in convert_messages_to_anthropic: only the 3 most recent screenshots carry real image data; older ones become text placeholders to cap per-turn token cost. - Context compressor image pruning: old multimodal tool results have their image parts stripped instead of being skipped. - Image-aware token estimation: each image counts as a flat 1500 tokens instead of its base64 char length (~1MB would have registered as ~250K tokens before). - COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset is active. - Session DB persistence strips base64 from multimodal tool messages. - Trajectory saver normalises multimodal messages to text-only. - `hermes tools` post-setup installs cua-driver via the upstream script and prints permission-grant instructions. - CLI approval callback wired so destructive computer_use actions go through the same prompt_toolkit approval dialog as terminal commands. - Hard safety guards at the tool level: blocked type patterns (curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash, force delete, lock screen, log out). - Skill `apple/macos-computer-use/SKILL.md` — universal (model-agnostic) workflow guide. - Docs: `user-guide/features/computer-use.md` plus reference catalog entries. ## Tests 44 new tests in tests/tools/test_computer_use.py covering schema shape (universal, not Anthropic-native), dispatch routing, safety guards, multimodal envelope, Anthropic adapter conversion, screenshot eviction, context compressor pruning, image-aware token estimation, run_agent helpers, and universality guarantees. 469/469 pass across tests/tools/test_computer_use.py + the affected agent/ test suites. ## Not in this PR - `model_tools.py` provider-gating: the tool is available to every provider. Providers without multi-part tool message support will see text-only tool results (graceful degradation via `text_summary`). - Anthropic server-side `clear_tool_uses_20250919` — deferred; client-side eviction + compressor pruning cover the same cost ceiling without a beta header. ## Caveats - macOS only. cua-driver uses private SkyLight SPIs (SLEventPostToPid, SLPSPostEventRecordTo, _AXObserverAddNotificationAndCheckRemote) that can break on any macOS update. Pin with HERMES_CUA_DRIVER_VERSION. - Requires Accessibility + Screen Recording permissions — the post-setup prints the Settings path. Supersedes PR #4562 (pyautogui/Quartz foreground backend, Anthropic- native schema). Credit @0xbyt4 for the original #3816 groundwork whose context/eviction/token design is preserved here in generic form.
This commit is contained in:
parent
24f139e16a
commit
b07791db05
23 changed files with 2861 additions and 27 deletions
43
tools/computer_use/__init__.py
Normal file
43
tools/computer_use/__init__.py
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
"""Computer use toolset — universal (any-model) macOS desktop control.
|
||||
|
||||
Architecture
|
||||
------------
|
||||
This toolset drives macOS apps through cua-driver's background computer-use
|
||||
primitive (SkyLight private SPIs for focus-without-raise + pid-scoped event
|
||||
posting). Unlike #4562's pyautogui backend, it does NOT steal the user's
|
||||
cursor, keyboard focus, or Space — the agent and the user can co-work on the
|
||||
same machine.
|
||||
|
||||
Unlike #4562's Anthropic-native `computer_20251124` tool, the schema here is
|
||||
a plain OpenAI function-calling schema that every tool-capable model can
|
||||
drive. Vision models get SOM (set-of-mark) captures — a screenshot with
|
||||
numbered overlays on every interactable element plus the AX tree — so they
|
||||
click by element index instead of pixel coordinates. Non-vision models can
|
||||
drive via the AX tree alone.
|
||||
|
||||
Wiring
|
||||
------
|
||||
* `tool.py` — registers the `computer_use` tool via tools.registry.
|
||||
* `backend.py` — abstract `ComputerUseBackend`; swappable implementation.
|
||||
* `cua_backend.py`— default backend; speaks MCP over stdio to `cua-driver`.
|
||||
* `schema.py` — shared schema + docstring for the generic `computer_use`
|
||||
tool. Model-agnostic.
|
||||
* `capture.py` — screenshot post-processing (PNG coercion, sizing, SOM
|
||||
overlay if the backend did not).
|
||||
|
||||
The outer integration points (multimodal tool-result plumbing, screenshot
|
||||
eviction in the Anthropic adapter, image-aware token estimation, the
|
||||
COMPUTER_USE_GUIDANCE prompt block, approval hook, and the skill) live
|
||||
alongside this package. See agent/anthropic_adapter.py and
|
||||
agent/prompt_builder.py for the salvaged hunks from PR #4562.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# Re-export the public surface so `from tools.computer_use import ...` works.
|
||||
from tools.computer_use.tool import ( # noqa: F401
|
||||
handle_computer_use,
|
||||
set_approval_callback,
|
||||
check_computer_use_requirements,
|
||||
get_computer_use_schema,
|
||||
)
|
||||
Loading…
Add table
Add a link
Reference in a new issue