feat(computer-use): cua-driver backend, universal any-model schema

Background macOS desktop control via cua-driver MCP — does NOT steal the
user's cursor or keyboard focus, works with any tool-capable model.

Replaces the Anthropic-native `computer_20251124` approach from the
abandoned #4562 with a generic OpenAI function-calling schema plus SOM
(set-of-mark) captures so Claude, GPT, Gemini, and open models can all
drive the desktop via numbered element indices.

## What this adds

- `tools/computer_use/` package — swappable ComputerUseBackend ABC +
  CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary).
- Universal `computer_use` tool with one schema for all providers.
  Actions: capture (som/vision/ax), click, double_click, right_click,
  middle_click, drag, scroll, type, key, wait, list_apps, focus_app.
- Multimodal tool-result envelope (`_multimodal=True`, OpenAI-style
  `content: [text, image_url]` parts) that flows through
  handle_function_call into the tool message. Anthropic adapter converts
  into native `tool_result` image blocks; OpenAI-compatible providers
  get the parts list directly.
- Image eviction in convert_messages_to_anthropic: only the 3 most
  recent screenshots carry real image data; older ones become text
  placeholders to cap per-turn token cost.
- Context compressor image pruning: old multimodal tool results have
  their image parts stripped instead of being skipped.
- Image-aware token estimation: each image counts as a flat 1500 tokens
  instead of its base64 char length (~1MB would have registered as
  ~250K tokens before).
- COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset
  is active.
- Session DB persistence strips base64 from multimodal tool messages.
- Trajectory saver normalises multimodal messages to text-only.
- `hermes tools` post-setup installs cua-driver via the upstream script
  and prints permission-grant instructions.
- CLI approval callback wired so destructive computer_use actions go
  through the same prompt_toolkit approval dialog as terminal commands.
- Hard safety guards at the tool level: blocked type patterns
  (curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash,
  force delete, lock screen, log out).
- Skill `apple/macos-computer-use/SKILL.md` — universal (model-agnostic)
  workflow guide.
- Docs: `user-guide/features/computer-use.md` plus reference catalog
  entries.

## Tests

44 new tests in tests/tools/test_computer_use.py covering schema
shape (universal, not Anthropic-native), dispatch routing, safety
guards, multimodal envelope, Anthropic adapter conversion, screenshot
eviction, context compressor pruning, image-aware token estimation,
run_agent helpers, and universality guarantees.

469/469 pass across tests/tools/test_computer_use.py + the affected
agent/ test suites.

## Not in this PR

- `model_tools.py` provider-gating: the tool is available to every
  provider. Providers without multi-part tool message support will see
  text-only tool results (graceful degradation via `text_summary`).
- Anthropic server-side `clear_tool_uses_20250919` — deferred;
  client-side eviction + compressor pruning cover the same cost ceiling
  without a beta header.

## Caveats

- macOS only. cua-driver uses private SkyLight SPIs
  (SLEventPostToPid, SLPSPostEventRecordTo,
  _AXObserverAddNotificationAndCheckRemote) that can break on any macOS
  update. Pin with HERMES_CUA_DRIVER_VERSION.
- Requires Accessibility + Screen Recording permissions — the post-setup
  prints the Settings path.

Supersedes PR #4562 (pyautogui/Quartz foreground backend, Anthropic-
native schema). Credit @0xbyt4 for the original #3816 groundwork whose
context/eviction/token design is preserved here in generic form.
This commit is contained in:
Teknium 2026-04-23 16:44:24 -07:00
parent 24f139e16a
commit b07791db05
No known key found for this signature in database
23 changed files with 2861 additions and 27 deletions

View file

@ -67,6 +67,7 @@ CONFIGURABLE_TOOLSETS = [
("messaging", "📨 Cross-Platform Messaging", "send_message"),
("rl", "🧪 RL Training", "Tinker-Atropos training tools"),
("homeassistant", "🏠 Home Assistant", "smart home device control"),
("computer_use", "🖱️ Computer Use (macOS)", "background desktop control via cua-driver"),
]
# Toolsets that are OFF by default for new installs.
@ -361,6 +362,27 @@ TOOL_CATEGORIES = {
},
],
},
"computer_use": {
"name": "Computer Use (macOS)",
"icon": "🖱️",
"platform_gate": "darwin",
"providers": [
{
"name": "cua-driver (background)",
"badge": "★ recommended · free · local",
"tag": (
"macOS background computer-use via SkyLight SPIs — does "
"NOT steal your cursor or focus. Works with any model."
),
"env_vars": [
# cua-driver reads HOME/TMPDIR from the process env, no
# extra keys required. HERMES_CUA_DRIVER_VERSION is an
# optional pin for reproducibility across macOS updates.
],
"post_setup": "cua_driver",
},
],
},
"rl": {
"name": "RL Training",
"icon": "🧪",
@ -431,6 +453,53 @@ def _run_post_setup(post_setup_key: str):
_print_warning(" Node.js not found. Install Camofox via Docker:")
_print_info(" docker run -p 9377:9377 -e CAMOFOX_PORT=9377 jo-inc/camofox-browser")
elif post_setup_key == "cua_driver":
# cua-driver provides macOS background computer-use (SkyLight SPIs).
# Install via upstream curl script if the binary isn't on $PATH yet.
import platform as _plat
import subprocess
if _plat.system() != "Darwin":
_print_warning(" Computer Use (cua-driver) is macOS-only; skipping.")
return
if shutil.which("cua-driver"):
try:
version = subprocess.run(
["cua-driver", "--version"],
capture_output=True, text=True, timeout=5,
).stdout.strip()
_print_success(f" cua-driver already installed: {version or 'unknown version'}")
except Exception:
_print_success(" cua-driver already installed.")
_print_info(" Grant macOS permissions if not done yet:")
_print_info(" System Settings > Privacy & Security > Accessibility")
_print_info(" System Settings > Privacy & Security > Screen Recording")
return
if not shutil.which("curl"):
_print_warning(" curl not found — install manually:")
_print_info(" https://github.com/trycua/cua/blob/main/libs/cua-driver/README.md")
return
_print_info(" Installing cua-driver (macOS background computer-use)...")
try:
install_cmd = (
"/bin/bash -c \"$(curl -fsSL "
"https://raw.githubusercontent.com/trycua/cua/main/"
"libs/cua-driver/scripts/install.sh)\""
)
result = subprocess.run(install_cmd, shell=True, timeout=300)
if result.returncode == 0 and shutil.which("cua-driver"):
_print_success(" cua-driver installed.")
_print_info(" IMPORTANT — grant macOS permissions now:")
_print_info(" System Settings > Privacy & Security > Accessibility")
_print_info(" System Settings > Privacy & Security > Screen Recording")
_print_info(" Both must allow the terminal / Hermes process.")
else:
_print_warning(" cua-driver install did not complete. Re-run manually:")
_print_info(f" {install_cmd}")
except subprocess.TimeoutExpired:
_print_warning(" cua-driver install timed out. Re-run manually.")
except Exception as e:
_print_warning(f" cua-driver install failed: {e}")
elif post_setup_key == "kittentts":
try:
__import__("kittentts")