docs(delegation,api): document stall detection, timeout metadata, /agents live status, runs-stream subagent lifecycle

Catch the docs up to this week's delegation work:

- delegation.md: structured timeout metadata fields (timeout_seconds/
  timed_out_after_seconds/timeout_phase, #72403); new 'Stall Detection
  for Background Subagents' section (progress-based monitor, thresholds,
  grace window, stalled event metadata, root-cause fix note — #72227/
  #72300/#72412); /agents per-child live activity on CLI + gateway;
  all-thread diagnostic dump note.
- api-server.md: /v1/runs events stream now documents subagent.start/
  subagent.complete forwarding, redaction, child_session_id, and the
  deliberate exclusion of per-tool child events (#72406).
- sidebars.ts: register the subagent-lifecycle-api developer guide page
  (shipped in #72501 but absent from the sidebar).

Docusaurus build verified.
This commit is contained in:
teknium1 2026-07-26 23:44:25 -07:00 committed by Teknium
parent ea2499bcfe
commit 4b4d2ae4cd
3 changed files with 73 additions and 1 deletions

View file

@ -375,6 +375,17 @@ Statuses are retained briefly after terminal states (`completed`, `failed`, or `
Server-Sent Events stream of the run's tool-call progress, token deltas, and lifecycle events. Designed for dashboards and thick clients that want to attach/detach without losing state.
When the agent delegates work to background subagents, the stream also carries
`subagent.start` and `subagent.complete` lifecycle events, so clients can
observe delegation outcomes — including timeouts and failures — instead of the
run going silent while a child works. The `subagent.complete` payload carries
the child's status, summary, duration, token/cost figures, and a
`child_session_id` for correlation; free-text fields pass forced secret
redaction before leaving the process. Per-tool child events
(`subagent.tool`, progress ticks) are intentionally **not** forwarded — they
are high-volume UI noise; use the per-child live transcript files for
play-by-play.
Unconsumed event buffers expire after five minutes so a detached client cannot
grow memory indefinitely. This expires transport state only: a run that is
still executing remains visible to status polling, approval, stop control, and

View file

@ -191,10 +191,54 @@ delegation:
A positive value enforces a hard wall-clock limit on each child; `0` or a negative value disables it.
When a configured cap fires, the child's result carries structured timeout
metadata alongside the error message so parents and hooks can distinguish a
stopwatch kill from other failures without parsing text: `timeout_seconds`
(the configured cap), `timed_out_after_seconds` (actual wall clock), and
`timeout_phase` (`before_first_llm_call` when the child never reached its
first request, `after_llm_calls` otherwise). All three are `null` on
non-timeout errors.
:::tip Diagnostic dump on zero-call timeout
With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout-<session>-<timestamp>.log` containing the subagent's config snapshot, credential-resolution trace, and any early error messages. Much easier to root-cause than the previous silent-timeout behavior.
With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout-<session>-<timestamp>.log` containing the subagent's config snapshot, credential-resolution trace, any early error messages, and stack traces for **all** live threads (not just the child's own) — a child parked waiting on a nested helper thread is indistinguishable from a slow provider without the full picture.
:::
## Stall Detection for Background Subagents
Background delegations (`delegate_task(background=true)`) are watched by a
**progress-based stall monitor** — on by default, zero config. Unlike a
wall-clock timeout, it never touches a child that is making progress, no
matter how long it runs.
The monitor samples each detached child's progress signals — API-call count,
current tool, and last-activity timestamp (which ticks on **every streamed
token**, tool transition, and API-call boundary, so a child mid-stream on a
long response always counts as alive):
1. **Progressing children are never touched.** Any advancing signal resets
the clock.
2. A child whose progress is completely frozen past the stale threshold
(450s idle, 1200s while inside a tool — legitimately slow terminal
commands and web fetches get the higher ceiling) is **interrupted** and
given a 120s grace window. A child that unwinds in time delivers its
partial results through the normal completion path.
3. A child that never returns is force-finalized with a terminal `stalled`
completion event, so the owning session hears an outcome instead of
going silent, and the async slot frees for new work.
The `stalled` event carries structured metadata mirroring the sync-path
timeout fields: `stalled_after_quiet_seconds`, `stall_threshold_seconds`,
`stall_phase` (`idle` / `in_tool`), and `stall_grace_seconds`.
This closed a long-standing failure mode where a wedged background child
left its session looking dead until a process restart. The underlying wedge
(children hanging at their first API call after multi-day gateway uptime)
was also fixed at the root: delegated children now run their OpenAI-wire
API requests inline on their own conversation thread instead of a nested
worker thread — the layer where the wedge lived. The stall monitor remains
as the safety net for anything else.
## Monitoring Running Subagents (`/agents`)
The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegate_task` fan-out into a first-class audit surface:
@ -206,6 +250,22 @@ The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegat
The classic CLI just prints `/agents` as a text summary; the TUI is where the overlay shines. See [TUI — Slash commands](/user-guide/tui#slash-commands).
On the classic CLI and every gateway platform (Telegram, Discord, Slack, ...),
`/agents` also lists **background delegations with live per-child activity**,
sampled directly from each running child:
```
Background delegations: 1 running
- deleg_ab12cd34 · running · research the delegation stall monitor
- child 1: 4 api calls · in web_search · active 12s ago
- child 2: 7 api calls · between turns · active 3s ago
```
A delegation the stall monitor has flagged shows as
`stalling · no progress 450s — interrupting`, and long-quiet-but-healthy
children show their quiet time so you can tell "slow" from "stuck" at a
glance.
## Live Transcripts
Every `delegate_task` dispatch also creates one **append-only, human-readable log per task** so you (or the parent agent) can watch a subagent work in real time instead of waiting for the consolidated summary:

View file

@ -750,6 +750,7 @@ const sidebars: SidebarsConfig = {
link: {type: 'doc', id: 'developer-guide/plugins/index'},
items: [
'developer-guide/plugin-llm-access',
'developer-guide/subagent-lifecycle-api',
'developer-guide/desktop-plugin-sdk',
'developer-guide/memory-provider-plugin',
'developer-guide/context-engine-plugin',