diff --git a/website/docs/user-guide/features/api-server.md b/website/docs/user-guide/features/api-server.md index 0311e2a228bc..8380d6313145 100644 --- a/website/docs/user-guide/features/api-server.md +++ b/website/docs/user-guide/features/api-server.md @@ -375,6 +375,17 @@ Statuses are retained briefly after terminal states (`completed`, `failed`, or ` Server-Sent Events stream of the run's tool-call progress, token deltas, and lifecycle events. Designed for dashboards and thick clients that want to attach/detach without losing state. +When the agent delegates work to background subagents, the stream also carries +`subagent.start` and `subagent.complete` lifecycle events, so clients can +observe delegation outcomes — including timeouts and failures — instead of the +run going silent while a child works. The `subagent.complete` payload carries +the child's status, summary, duration, token/cost figures, and a +`child_session_id` for correlation; free-text fields pass forced secret +redaction before leaving the process. Per-tool child events +(`subagent.tool`, progress ticks) are intentionally **not** forwarded — they +are high-volume UI noise; use the per-child live transcript files for +play-by-play. + Unconsumed event buffers expire after five minutes so a detached client cannot grow memory indefinitely. This expires transport state only: a run that is still executing remains visible to status polling, approval, stop control, and diff --git a/website/docs/user-guide/features/delegation.md b/website/docs/user-guide/features/delegation.md index 4790ce3b4200..d40a0dfc107b 100644 --- a/website/docs/user-guide/features/delegation.md +++ b/website/docs/user-guide/features/delegation.md @@ -191,10 +191,54 @@ delegation: A positive value enforces a hard wall-clock limit on each child; `0` or a negative value disables it. +When a configured cap fires, the child's result carries structured timeout +metadata alongside the error message so parents and hooks can distinguish a +stopwatch kill from other failures without parsing text: `timeout_seconds` +(the configured cap), `timed_out_after_seconds` (actual wall clock), and +`timeout_phase` (`before_first_llm_call` when the child never reached its +first request, `after_llm_calls` otherwise). All three are `null` on +non-timeout errors. + :::tip Diagnostic dump on zero-call timeout -With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout--.log` containing the subagent's config snapshot, credential-resolution trace, and any early error messages. Much easier to root-cause than the previous silent-timeout behavior. +With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout--.log` containing the subagent's config snapshot, credential-resolution trace, any early error messages, and stack traces for **all** live threads (not just the child's own) — a child parked waiting on a nested helper thread is indistinguishable from a slow provider without the full picture. ::: +## Stall Detection for Background Subagents + +Background delegations (`delegate_task(background=true)`) are watched by a +**progress-based stall monitor** — on by default, zero config. Unlike a +wall-clock timeout, it never touches a child that is making progress, no +matter how long it runs. + +The monitor samples each detached child's progress signals — API-call count, +current tool, and last-activity timestamp (which ticks on **every streamed +token**, tool transition, and API-call boundary, so a child mid-stream on a +long response always counts as alive): + +1. **Progressing children are never touched.** Any advancing signal resets + the clock. +2. A child whose progress is completely frozen past the stale threshold + (450s idle, 1200s while inside a tool — legitimately slow terminal + commands and web fetches get the higher ceiling) is **interrupted** and + given a 120s grace window. A child that unwinds in time delivers its + partial results through the normal completion path. +3. A child that never returns is force-finalized with a terminal `stalled` + completion event, so the owning session hears an outcome instead of + going silent, and the async slot frees for new work. + +The `stalled` event carries structured metadata mirroring the sync-path +timeout fields: `stalled_after_quiet_seconds`, `stall_threshold_seconds`, +`stall_phase` (`idle` / `in_tool`), and `stall_grace_seconds`. + +This closed a long-standing failure mode where a wedged background child +left its session looking dead until a process restart. The underlying wedge +(children hanging at their first API call after multi-day gateway uptime) +was also fixed at the root: delegated children now run their OpenAI-wire +API requests inline on their own conversation thread instead of a nested +worker thread — the layer where the wedge lived. The stall monitor remains +as the safety net for anything else. + + ## Monitoring Running Subagents (`/agents`) The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegate_task` fan-out into a first-class audit surface: @@ -206,6 +250,22 @@ The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegat The classic CLI just prints `/agents` as a text summary; the TUI is where the overlay shines. See [TUI — Slash commands](/user-guide/tui#slash-commands). +On the classic CLI and every gateway platform (Telegram, Discord, Slack, ...), +`/agents` also lists **background delegations with live per-child activity**, +sampled directly from each running child: + +``` +Background delegations: 1 running +- deleg_ab12cd34 · running · research the delegation stall monitor + - child 1: 4 api calls · in web_search · active 12s ago + - child 2: 7 api calls · between turns · active 3s ago +``` + +A delegation the stall monitor has flagged shows as +`stalling · no progress 450s — interrupting`, and long-quiet-but-healthy +children show their quiet time so you can tell "slow" from "stuck" at a +glance. + ## Live Transcripts Every `delegate_task` dispatch also creates one **append-only, human-readable log per task** so you (or the parent agent) can watch a subagent work in real time instead of waiting for the consolidated summary: diff --git a/website/sidebars.ts b/website/sidebars.ts index 7314af5a2b38..87f7a7e8d898 100644 --- a/website/sidebars.ts +++ b/website/sidebars.ts @@ -750,6 +750,7 @@ const sidebars: SidebarsConfig = { link: {type: 'doc', id: 'developer-guide/plugins/index'}, items: [ 'developer-guide/plugin-llm-access', + 'developer-guide/subagent-lifecycle-api', 'developer-guide/desktop-plugin-sdk', 'developer-guide/memory-provider-plugin', 'developer-guide/context-engine-plugin',