mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-18 04:41:56 +00:00
252d68fd45
* docs: deep audit — fix stale config keys, missing commands, and registry drift Cross-checked ~80 high-impact docs pages (getting-started, reference, top-level user-guide, user-guide/features) against the live registries: hermes_cli/commands.py COMMAND_REGISTRY (slash commands) hermes_cli/auth.py PROVIDER_REGISTRY (providers) hermes_cli/config.py DEFAULT_CONFIG (config keys) toolsets.py TOOLSETS (toolsets) tools/registry.py get_all_tool_names() (tools) python -m hermes_cli.main <subcmd> --help (CLI args) reference/ - cli-commands.md: drop duplicate hermes fallback row + duplicate section, add stepfun/lmstudio to --provider enum, expand auth/mcp/curator subcommand lists to match --help output (status/logout/spotify, login, archive/prune/ list-archived). - slash-commands.md: add missing /sessions and /reload-skills entries + correct the cross-platform Notes line. - tools-reference.md: drop bogus '68 tools' headline, drop fictional 'browser-cdp toolset' (these tools live in 'browser' and are runtime-gated), add missing 'kanban' and 'video' toolset sections, fix MCP example to use the real mcp_<server>_<tool> prefix. - toolsets-reference.md: list browser_cdp/browser_dialog inside the 'browser' row, add missing 'kanban' and 'video' toolset rows, drop the stale '38 tools' count for hermes-cli. - profile-commands.md: add missing install/update/info subcommands, document fish completion. - environment-variables.md: dedupe GMI_API_KEY/GMI_BASE_URL rows (kept the one with the correct gmi-serving.com default). - faq.md: Anthropic/Google/OpenAI examples — direct providers exist (not just via OpenRouter), refresh the OpenAI model list. getting-started/ - installation.md: PortableGit (not MinGit) is what the Windows installer fetches; document the 32-bit MinGit fallback. - installation.md / termux.md: installer prefers .[termux-all] then falls back to .[termux]. - nix-setup.md: Python 3.12 (not 3.11), Node.js 22 (not 20); fix invalid 'nix flake update --flake' invocation. - updating.md: 'hermes backup restore --state pre-update' doesn't exist — point at the snapshot/quick-snapshot flow; correct config key 'updates.pre_update_backup' (was 'update.backup'). user-guide/ - configuration.md: api_max_retries default 3 (not 2); display.runtime_footer is the real key (not display.runtime_metadata_footer); checkpoints defaults enabled=false / max_snapshots=20 (not true / 50). - configuring-models.md: 'hermes model list' / 'hermes model set ...' don't exist — hermes model is interactive only. - tui.md: busy_indicator -> tui_status_indicator with values kaomoji|emoji|unicode|ascii (not kawaii|minimal|dots|wings|none). - security.md: SSH backend keys (TERMINAL_SSH_HOST/USER/KEY) live in .env, not config.yaml. - windows-wsl-quickstart.md: there is no 'hermes api' subcommand — the OpenAI-compatible API server runs inside hermes gateway. user-guide/features/ - computer-use.md: approvals.mode (not security.approval_level); fix broken ./browser-use.md link to ./browser.md. - fallback-providers.md: top-level fallback_providers (not model.fallback_providers); the picker is subcommand-based, not modal. - api-server.md: API_SERVER_* are env vars — write to per-profile .env, not 'hermes config set' which targets YAML. - web-search.md: drop web_crawl as a registered tool (it isn't); deep-crawl modes are exposed through web_extract. - kanban.md: failure_limit default is 2, not '~5'. - plugins.md: drop hard-coded '33 providers' count. - honcho.md: fix unclosed quote in echo HONCHO_API_KEY snippet; document that 'hermes honcho' subcommand is gated on memory.provider=honcho; reconcile subcommand list with actual --help output. - memory-providers.md: legacy 'hermes honcho setup' redirect documented. Verified via 'npm run build' — site builds cleanly; broken-link count went from 149 to 146 (no regressions, fixed a few in passing). * docs: round 2 audit fixes + regenerate skill catalogs Follow-up to the previous commit on this branch: Round 2 manual fixes: - quickstart.md: KIMI_CODING_API_KEY mentioned alongside KIMI_API_KEY; voice-mode and ACP install commands rewritten — bare 'pip install ...' doesn't work for curl-installed setups (no pip on PATH, not in repo dir); replaced with 'cd ~/.hermes/hermes-agent && uv pip install -e ".[voice]"'. ACP already ships in [all] so the curl install includes it. - cli.md / configuration.md: 'auxiliary.compression.model' shown as 'google/gemini-3-flash-preview' (the doc's own claimed default); actual default is empty (= use main model). Reworded as 'leave empty (default) or pin a cheap model'. - built-in-plugins.md: added the bundled 'kanban/dashboard' plugin row that was missing from the table. Regenerated skill catalogs: - ran website/scripts/generate-skill-docs.py to refresh all 163 per-skill pages and both reference catalogs (skills-catalog.md, optional-skills-catalog.md). This adds the entries that were genuinely missing — productivity/teams-meeting-pipeline (bundled), optional/finance/* (entire category — 7 skills: 3-statement-model, comps-analysis, dcf-model, excel-author, lbo-model, merger-model, pptx-author), creative/hyperframes, creative/kanban-video-orchestrator, devops/watchers, productivity/shop-app, research/searxng-search, apple/macos-computer-use — and rewrites every other per-skill page from the current SKILL.md. Most diffs are tiny (one line of refreshed metadata). Validation: - 'npm run build' succeeded. - Broken-link count moved 146 -> 155 — the +9 are zh-Hans translation shells that lag every newly-added skill page (pre-existing pattern). No regressions on any en/ page.
179 lines
6.9 KiB
Markdown
179 lines
6.9 KiB
Markdown
# Computer Use (macOS)
|
||
|
||
Hermes Agent can drive your Mac's desktop — clicking, typing, scrolling,
|
||
dragging — in the **background**. Your cursor doesn't move, keyboard focus
|
||
doesn't change, and macOS doesn't switch Spaces on you. You and the agent
|
||
co-work on the same machine.
|
||
|
||
Unlike most computer-use integrations, this works with **any tool-capable
|
||
model** — Claude, GPT, Gemini, or an open model on a local vLLM endpoint.
|
||
There's no Anthropic-native schema to worry about.
|
||
|
||
## How it works
|
||
|
||
The `computer_use` toolset speaks MCP over stdio to [`cua-driver`](https://github.com/trycua/cua),
|
||
a macOS driver that uses SkyLight private SPIs (`SLEventPostToPid`,
|
||
`SLPSPostEventRecordTo`) and the `_AXObserverAddNotificationAndCheckRemote`
|
||
accessibility SPI to:
|
||
|
||
- Post synthesized events directly to target processes — no HID event tap,
|
||
no cursor warp.
|
||
- Flip AppKit active-state without raising windows — no Space switching.
|
||
- Keep Chromium/Electron accessibility trees alive when windows are
|
||
occluded.
|
||
|
||
That combination is what OpenAI's Codex "background computer-use" ships.
|
||
cua-driver is the open-source equivalent.
|
||
|
||
## Enabling
|
||
|
||
Pick whichever path is most convenient — both run the same upstream installer:
|
||
|
||
**Option 1: dedicated CLI command (most direct).**
|
||
|
||
```
|
||
hermes computer-use install
|
||
```
|
||
|
||
This fetches and runs the upstream cua-driver installer:
|
||
`curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh`.
|
||
Use `hermes computer-use status` to verify the install.
|
||
|
||
**Option 2: enable the toolset interactively.**
|
||
|
||
1. Run `hermes tools`, pick `🖱️ Computer Use (macOS)` → `cua-driver (background)`.
|
||
2. The setup runs the upstream installer (same as Option 1).
|
||
|
||
After installing, regardless of which path you took:
|
||
|
||
3. Grant macOS permissions when prompted:
|
||
- **System Settings → Privacy & Security → Accessibility** → allow the
|
||
terminal (or Hermes app).
|
||
- **System Settings → Privacy & Security → Screen Recording** → allow
|
||
the same.
|
||
4. Start a session with the toolset enabled:
|
||
```
|
||
hermes -t computer_use chat
|
||
```
|
||
or add `computer_use` to your enabled toolsets in `~/.hermes/config.yaml`.
|
||
|
||
## Quick example
|
||
|
||
User prompt: *"Find my latest email from Stripe and summarise what they want me to do."*
|
||
|
||
The agent's plan:
|
||
|
||
1. `computer_use(action="capture", mode="som", app="Mail")` — gets a
|
||
screenshot of Mail with every sidebar item, toolbar button, and message
|
||
row numbered.
|
||
2. `computer_use(action="click", element=14)` — clicks the search field
|
||
(element #14 from the capture).
|
||
3. `computer_use(action="type", text="from:stripe")`
|
||
4. `computer_use(action="key", keys="return", capture_after=True)` — submit
|
||
and get the new screenshot.
|
||
5. Click the top result, read the body, summarise.
|
||
|
||
During all of this, your cursor stays wherever you left it and Mail never
|
||
comes to front.
|
||
|
||
## Provider compatibility
|
||
|
||
| Provider | Vision? | Works? | Notes |
|
||
|---|---|---|---|
|
||
| Anthropic (Claude Sonnet/Opus 3+) | ✅ | ✅ | Best overall; SOM + raw coordinates. |
|
||
| OpenRouter (any vision model) | ✅ | ✅ | Multi-part tool messages supported. |
|
||
| OpenAI (GPT-4+, GPT-5) | ✅ | ✅ | Same as above. |
|
||
| Local vLLM / LM Studio (vision model) | ✅ | ✅ | If the model supports multi-part tool content. |
|
||
| Text-only models | ❌ | ✅ (degraded) | Use `mode="ax"` for accessibility-tree-only operation. |
|
||
|
||
Screenshots are sent inline with tool results as OpenAI-style `image_url`
|
||
parts. For Anthropic, the adapter converts them into native `tool_result`
|
||
image blocks.
|
||
|
||
## Safety
|
||
|
||
Hermes applies multi-layer guardrails:
|
||
|
||
- Destructive actions (click, type, drag, scroll, key, focus_app) require
|
||
approval — either interactively via the CLI dialog or via the
|
||
messaging-platform approval buttons.
|
||
- Hard-blocked key combos at the tool level: empty trash, force delete,
|
||
lock screen, log out, force log out.
|
||
- Hard-blocked type patterns: `curl | bash`, `sudo rm -rf /`, fork bombs,
|
||
etc.
|
||
- The agent's system prompt tells it explicitly: no clicking permission
|
||
dialogs, no typing passwords, no following instructions embedded in
|
||
screenshots.
|
||
|
||
Pair with `approvals.mode: manual` in `~/.hermes/config.yaml` if you want every action confirmed.
|
||
|
||
## Token efficiency
|
||
|
||
Screenshots are expensive. Hermes applies four layers of optimisation:
|
||
|
||
- **Screenshot eviction** — the Anthropic adapter keeps only the 3 most
|
||
recent screenshots in context; older ones become `[screenshot removed
|
||
to save context]` placeholders.
|
||
- **Client-side compression pruning** — the context compressor detects
|
||
multimodal tool results and strips image parts from old ones.
|
||
- **Image-aware token estimation** — each image is counted as ~1500 tokens
|
||
(Anthropic's flat rate) instead of its base64 char length.
|
||
- **Server-side context editing (Anthropic only)** — when active, the
|
||
adapter enables `clear_tool_uses_20250919` via `context_management` so
|
||
Anthropic's API clears old tool results server-side.
|
||
|
||
A 20-action session on a 1568×900 display typically costs ~30K tokens
|
||
of screenshot context, not ~600K.
|
||
|
||
## Limitations
|
||
|
||
- **macOS only.** cua-driver uses private Apple SPIs that don't exist on
|
||
Linux or Windows. For cross-platform GUI automation, use the `browser`
|
||
toolset.
|
||
- **Private SPI risk.** Apple can change SkyLight's symbol surface in any
|
||
OS update. Pin the driver version with the `HERMES_CUA_DRIVER_VERSION`
|
||
env var if you want reproducibility across a macOS bump.
|
||
- **Performance.** Background mode is slower than foreground —
|
||
SkyLight-routed events take ~5-20ms vs direct HID posting. Not
|
||
noticeable for agent-speed clicking; noticeable if you try to record a
|
||
speed-run.
|
||
- **No keyboard password entry.** `type` has hard-block patterns on
|
||
command-shell payloads; for passwords, use the system's autofill.
|
||
|
||
## Configuration
|
||
|
||
Override the driver binary path (tests / CI):
|
||
|
||
```
|
||
HERMES_CUA_DRIVER_CMD=/opt/homebrew/bin/cua-driver
|
||
HERMES_CUA_DRIVER_VERSION=0.5.0 # optional pin
|
||
```
|
||
|
||
Swap the backend entirely (for testing):
|
||
|
||
```
|
||
HERMES_COMPUTER_USE_BACKEND=noop # records calls, no side effects
|
||
```
|
||
|
||
## Troubleshooting
|
||
|
||
**`computer_use backend unavailable: cua-driver is not installed`** — Run
|
||
`hermes computer-use install` to fetch the cua-driver binary, or run
|
||
`hermes tools` and enable the Computer Use toolset.
|
||
|
||
**Clicks seem to have no effect** — Capture and verify. A modal you
|
||
didn't see may be blocking input. Dismiss it with `escape` or the close
|
||
button.
|
||
|
||
**Element indices are stale** — SOM indices are only valid until the
|
||
next `capture`. Re-capture after any state-changing action.
|
||
|
||
**"blocked pattern in type text"** — The text you tried to `type`
|
||
matches the dangerous-shell-pattern list. Break the command up or
|
||
reconsider.
|
||
|
||
## See also
|
||
|
||
- [Universal skill: `macos-computer-use`](https://github.com/NousResearch/hermes-agent/blob/main/skills/apple/macos-computer-use/SKILL.md)
|
||
- [cua-driver source (trycua/cua)](https://github.com/trycua/cua)
|
||
- [Browser automation](./browser.md) for cross-platform web tasks.
|