mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-05-18 04:41:56 +00:00

docs: deep audit — fix stale config keys, missing commands, and registry drift (#22784 )

* docs: deep audit — fix stale config keys, missing commands, and registry drift

Cross-checked ~80 high-impact docs pages (getting-started, reference, top-level
user-guide, user-guide/features) against the live registries:

  hermes_cli/commands.py    COMMAND_REGISTRY (slash commands)
  hermes_cli/auth.py        PROVIDER_REGISTRY (providers)
  hermes_cli/config.py      DEFAULT_CONFIG (config keys)
  toolsets.py               TOOLSETS (toolsets)
  tools/registry.py         get_all_tool_names() (tools)
  python -m hermes_cli.main <subcmd> --help (CLI args)

reference/
- cli-commands.md: drop duplicate hermes fallback row + duplicate section,
  add stepfun/lmstudio to --provider enum, expand auth/mcp/curator subcommand
  lists to match --help output (status/logout/spotify, login, archive/prune/
  list-archived).
- slash-commands.md: add missing /sessions and /reload-skills entries +
  correct the cross-platform Notes line.
- tools-reference.md: drop bogus '68 tools' headline, drop fictional
  'browser-cdp toolset' (these tools live in 'browser' and are runtime-gated),
  add missing 'kanban' and 'video' toolset sections, fix MCP example to use
  the real mcp_<server>_<tool> prefix.
- toolsets-reference.md: list browser_cdp/browser_dialog inside the 'browser'
  row, add missing 'kanban' and 'video' toolset rows, drop the stale
  '38 tools' count for hermes-cli.
- profile-commands.md: add missing install/update/info subcommands, document
  fish completion.
- environment-variables.md: dedupe GMI_API_KEY/GMI_BASE_URL rows (kept the
  one with the correct gmi-serving.com default).
- faq.md: Anthropic/Google/OpenAI examples — direct providers exist (not just
  via OpenRouter), refresh the OpenAI model list.

getting-started/
- installation.md: PortableGit (not MinGit) is what the Windows installer
  fetches; document the 32-bit MinGit fallback.
- installation.md / termux.md: installer prefers .[termux-all] then falls
  back to .[termux].
- nix-setup.md: Python 3.12 (not 3.11), Node.js 22 (not 20); fix invalid
  'nix flake update --flake' invocation.
- updating.md: 'hermes backup restore --state pre-update' doesn't exist —
  point at the snapshot/quick-snapshot flow; correct config key
  'updates.pre_update_backup' (was 'update.backup').

user-guide/
- configuration.md: api_max_retries default 3 (not 2); display.runtime_footer
  is the real key (not display.runtime_metadata_footer); checkpoints defaults
  enabled=false / max_snapshots=20 (not true / 50).
- configuring-models.md: 'hermes model list' / 'hermes model set ...' don't
  exist — hermes model is interactive only.
- tui.md: busy_indicator -> tui_status_indicator with values
  kaomoji|emoji|unicode|ascii (not kawaii|minimal|dots|wings|none).
- security.md: SSH backend keys (TERMINAL_SSH_HOST/USER/KEY) live in .env,
  not config.yaml.
- windows-wsl-quickstart.md: there is no 'hermes api' subcommand — the
  OpenAI-compatible API server runs inside hermes gateway.

user-guide/features/
- computer-use.md: approvals.mode (not security.approval_level); fix broken
  ./browser-use.md link to ./browser.md.
- fallback-providers.md: top-level fallback_providers (not
  model.fallback_providers); the picker is subcommand-based, not modal.
- api-server.md: API_SERVER_* are env vars — write to per-profile .env,
  not 'hermes config set' which targets YAML.
- web-search.md: drop web_crawl as a registered tool (it isn't); deep-crawl
  modes are exposed through web_extract.
- kanban.md: failure_limit default is 2, not '~5'.
- plugins.md: drop hard-coded '33 providers' count.
- honcho.md: fix unclosed quote in echo HONCHO_API_KEY snippet; document
  that 'hermes honcho' subcommand is gated on memory.provider=honcho;
  reconcile subcommand list with actual --help output.
- memory-providers.md: legacy 'hermes honcho setup' redirect documented.

Verified via 'npm run build' — site builds cleanly; broken-link count went
from 149 to 146 (no regressions, fixed a few in passing).

* docs: round 2 audit fixes + regenerate skill catalogs

Follow-up to the previous commit on this branch:

Round 2 manual fixes:
- quickstart.md: KIMI_CODING_API_KEY mentioned alongside KIMI_API_KEY;
  voice-mode and ACP install commands rewritten — bare 'pip install ...'
  doesn't work for curl-installed setups (no pip on PATH, not in repo
  dir); replaced with 'cd ~/.hermes/hermes-agent && uv pip install -e
  ".[voice]"'. ACP already ships in [all] so the curl install includes it.
- cli.md / configuration.md: 'auxiliary.compression.model' shown as
  'google/gemini-3-flash-preview' (the doc's own claimed default);
  actual default is empty (= use main model). Reworded as 'leave empty
  (default) or pin a cheap model'.
- built-in-plugins.md: added the bundled 'kanban/dashboard' plugin row
  that was missing from the table.

Regenerated skill catalogs:
- ran website/scripts/generate-skill-docs.py to refresh all 163 per-skill
  pages and both reference catalogs (skills-catalog.md,
  optional-skills-catalog.md). This adds the entries that were genuinely
  missing — productivity/teams-meeting-pipeline (bundled),
  optional/finance/* (entire category — 7 skills:
  3-statement-model, comps-analysis, dcf-model, excel-author, lbo-model,
  merger-model, pptx-author), creative/hyperframes,
  creative/kanban-video-orchestrator, devops/watchers,
  productivity/shop-app, research/searxng-search,
  apple/macos-computer-use — and rewrites every other per-skill page from
  the current SKILL.md. Most diffs are tiny (one line of refreshed
  metadata).

Validation:
- 'npm run build' succeeded.
- Broken-link count moved 146 -> 155 — the +9 are zh-Hans translation
  shells that lag every newly-added skill page (pre-existing pattern).
  No regressions on any en/ page.

2026-05-09 13:19:51 -07:00

6.9 KiB

Raw Blame History

Computer Use (macOS)

Hermes Agent can drive your Mac's desktop — clicking, typing, scrolling, dragging — in the background. Your cursor doesn't move, keyboard focus doesn't change, and macOS doesn't switch Spaces on you. You and the agent co-work on the same machine.

Unlike most computer-use integrations, this works with any tool-capable model — Claude, GPT, Gemini, or an open model on a local vLLM endpoint. There's no Anthropic-native schema to worry about.

How it works

The computer_use toolset speaks MCP over stdio to cua-driver, a macOS driver that uses SkyLight private SPIs (SLEventPostToPid, SLPSPostEventRecordTo) and the _AXObserverAddNotificationAndCheckRemote accessibility SPI to:

Post synthesized events directly to target processes — no HID event tap, no cursor warp.
Flip AppKit active-state without raising windows — no Space switching.
Keep Chromium/Electron accessibility trees alive when windows are occluded.

That combination is what OpenAI's Codex "background computer-use" ships. cua-driver is the open-source equivalent.

Enabling

Pick whichever path is most convenient — both run the same upstream installer:

Option 1: dedicated CLI command (most direct).

hermes computer-use install

This fetches and runs the upstream cua-driver installer: curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh. Use hermes computer-use status to verify the install.

Option 2: enable the toolset interactively.

Run hermes tools, pick 🖱️ Computer Use (macOS) → cua-driver (background).
The setup runs the upstream installer (same as Option 1).

After installing, regardless of which path you took:

Grant macOS permissions when prompted:
- System Settings → Privacy & Security → Accessibility → allow the terminal (or Hermes app).
- System Settings → Privacy & Security → Screen Recording → allow the same.
Start a session with the toolset enabled:
```
hermes -t computer_use chat
```
or add computer_use to your enabled toolsets in ~/.hermes/config.yaml.

Quick example

User prompt: "Find my latest email from Stripe and summarise what they want me to do."

The agent's plan:

computer_use(action="capture", mode="som", app="Mail") — gets a screenshot of Mail with every sidebar item, toolbar button, and message row numbered.
computer_use(action="click", element=14) — clicks the search field (element #14 from the capture).
computer_use(action="type", text="from:stripe")
computer_use(action="key", keys="return", capture_after=True) — submit and get the new screenshot.
Click the top result, read the body, summarise.

During all of this, your cursor stays wherever you left it and Mail never comes to front.

Provider compatibility

Provider	Vision?	Works?	Notes
Anthropic (Claude Sonnet/Opus 3+)	✅	✅	Best overall; SOM + raw coordinates.
OpenRouter (any vision model)	✅	✅	Multi-part tool messages supported.
OpenAI (GPT-4+, GPT-5)	✅	✅	Same as above.
Local vLLM / LM Studio (vision model)	✅	✅	If the model supports multi-part tool content.
Text-only models	❌	✅ (degraded)	Use `mode="ax"` for accessibility-tree-only operation.

Screenshots are sent inline with tool results as OpenAI-style image_url parts. For Anthropic, the adapter converts them into native tool_result image blocks.

Safety

Hermes applies multi-layer guardrails:

Destructive actions (click, type, drag, scroll, key, focus_app) require approval — either interactively via the CLI dialog or via the messaging-platform approval buttons.
Hard-blocked key combos at the tool level: empty trash, force delete, lock screen, log out, force log out.
Hard-blocked type patterns: curl | bash, sudo rm -rf /, fork bombs, etc.
The agent's system prompt tells it explicitly: no clicking permission dialogs, no typing passwords, no following instructions embedded in screenshots.

Pair with approvals.mode: manual in ~/.hermes/config.yaml if you want every action confirmed.

Token efficiency

Screenshots are expensive. Hermes applies four layers of optimisation:

Screenshot eviction — the Anthropic adapter keeps only the 3 most recent screenshots in context; older ones become [screenshot removed to save context] placeholders.
Client-side compression pruning — the context compressor detects multimodal tool results and strips image parts from old ones.
Image-aware token estimation — each image is counted as ~1500 tokens (Anthropic's flat rate) instead of its base64 char length.
Server-side context editing (Anthropic only) — when active, the adapter enables clear_tool_uses_20250919 via context_management so Anthropic's API clears old tool results server-side.

A 20-action session on a 1568×900 display typically costs ~30K tokens of screenshot context, not ~600K.

Limitations

macOS only. cua-driver uses private Apple SPIs that don't exist on Linux or Windows. For cross-platform GUI automation, use the browser toolset.
Private SPI risk. Apple can change SkyLight's symbol surface in any OS update. Pin the driver version with the HERMES_CUA_DRIVER_VERSION env var if you want reproducibility across a macOS bump.
Performance. Background mode is slower than foreground — SkyLight-routed events take ~5-20ms vs direct HID posting. Not noticeable for agent-speed clicking; noticeable if you try to record a speed-run.
No keyboard password entry. type has hard-block patterns on command-shell payloads; for passwords, use the system's autofill.

Configuration

Override the driver binary path (tests / CI):

HERMES_CUA_DRIVER_CMD=/opt/homebrew/bin/cua-driver
HERMES_CUA_DRIVER_VERSION=0.5.0    # optional pin

Swap the backend entirely (for testing):

HERMES_COMPUTER_USE_BACKEND=noop   # records calls, no side effects

Troubleshooting

computer_use backend unavailable: cua-driver is not installed — Run hermes computer-use install to fetch the cua-driver binary, or run hermes tools and enable the Computer Use toolset.

Clicks seem to have no effect — Capture and verify. A modal you didn't see may be blocking input. Dismiss it with escape or the close button.

Element indices are stale — SOM indices are only valid until the next capture. Re-capture after any state-changing action.

"blocked pattern in type text" — The text you tried to type matches the dangerous-shell-pattern list. Break the command up or reconsider.

6.9 KiB Raw Blame History Unescape Escape