mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-21 16:18:55 +00:00
9d6d772837
Hermes' computer_use wrapper dropped cua-driver's structured action verdicts, exposed no delivery_mode, and injected background-only guidance — so the agent reported unverified no-ops as success and concluded cua-driver 'cannot drive' Electron/Chromium surfaces (observed live on tldraw offline). Fixes #67052. Phase A — preserve the result contract: - ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode - CuaDriverBackend._action() reads structuredContent (was data-only); a helper normalizes it, additive and None-safe on old drivers - _text_response surfaces the fields additively (ok stays transport-only) Phase B — bounded, model-reachable foreground: - delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher, ABC, and all input methods - foreground is capability-gated (input.delivery_mode); old drivers get a structured foreground_unsupported refusal, never a silent background downgrade - no automatic/hidden foreground retry — the model selects it from the signal Phase C — guidance + isolation: - system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven by returned effect/escalation, not predicted from the app being Electron - foreground approval scoped by (action, delivery_mode): a background approval never silently authorizes foreground - approval state keyed per session_id so concurrent gateway runs don't leak unlocks Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/ unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating + foreground_unsupported, and session-scoped foreground approval. Existing 265 computer_use tests still green. Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground request returned code='foreground_unsupported' — correct on a driver that predates the input.delivery_mode capability. |
||
|---|---|---|
| .. | ||
| apple | ||
| autonomous-ai-agents | ||
| computer-use | ||
| creative | ||
| data-science | ||
| dogfood | ||
| github | ||
| hermes-desktop-plugins | ||
| index-cache | ||
| media | ||
| mlops | ||
| note-taking | ||
| productivity | ||
| research | ||
| smart-home | ||
| social-media | ||
| software-development | ||
| yuanbao | ||