mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-29 18:46:59 +00:00
docs(delegation,api): document stall detection, timeout metadata, /agents live status, runs-stream subagent lifecycle
Catch the docs up to this week's delegation work: - delegation.md: structured timeout metadata fields (timeout_seconds/ timed_out_after_seconds/timeout_phase, #72403); new 'Stall Detection for Background Subagents' section (progress-based monitor, thresholds, grace window, stalled event metadata, root-cause fix note — #72227/ #72300/#72412); /agents per-child live activity on CLI + gateway; all-thread diagnostic dump note. - api-server.md: /v1/runs events stream now documents subagent.start/ subagent.complete forwarding, redaction, child_session_id, and the deliberate exclusion of per-tool child events (#72406). - sidebars.ts: register the subagent-lifecycle-api developer guide page (shipped in #72501 but absent from the sidebar). Docusaurus build verified.
This commit is contained in:
parent
ea2499bcfe
commit
4b4d2ae4cd
3 changed files with 73 additions and 1 deletions
|
|
@ -375,6 +375,17 @@ Statuses are retained briefly after terminal states (`completed`, `failed`, or `
|
|||
|
||||
Server-Sent Events stream of the run's tool-call progress, token deltas, and lifecycle events. Designed for dashboards and thick clients that want to attach/detach without losing state.
|
||||
|
||||
When the agent delegates work to background subagents, the stream also carries
|
||||
`subagent.start` and `subagent.complete` lifecycle events, so clients can
|
||||
observe delegation outcomes — including timeouts and failures — instead of the
|
||||
run going silent while a child works. The `subagent.complete` payload carries
|
||||
the child's status, summary, duration, token/cost figures, and a
|
||||
`child_session_id` for correlation; free-text fields pass forced secret
|
||||
redaction before leaving the process. Per-tool child events
|
||||
(`subagent.tool`, progress ticks) are intentionally **not** forwarded — they
|
||||
are high-volume UI noise; use the per-child live transcript files for
|
||||
play-by-play.
|
||||
|
||||
Unconsumed event buffers expire after five minutes so a detached client cannot
|
||||
grow memory indefinitely. This expires transport state only: a run that is
|
||||
still executing remains visible to status polling, approval, stop control, and
|
||||
|
|
|
|||
|
|
@ -191,10 +191,54 @@ delegation:
|
|||
|
||||
A positive value enforces a hard wall-clock limit on each child; `0` or a negative value disables it.
|
||||
|
||||
When a configured cap fires, the child's result carries structured timeout
|
||||
metadata alongside the error message so parents and hooks can distinguish a
|
||||
stopwatch kill from other failures without parsing text: `timeout_seconds`
|
||||
(the configured cap), `timed_out_after_seconds` (actual wall clock), and
|
||||
`timeout_phase` (`before_first_llm_call` when the child never reached its
|
||||
first request, `after_llm_calls` otherwise). All three are `null` on
|
||||
non-timeout errors.
|
||||
|
||||
:::tip Diagnostic dump on zero-call timeout
|
||||
With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout-<session>-<timestamp>.log` containing the subagent's config snapshot, credential-resolution trace, and any early error messages. Much easier to root-cause than the previous silent-timeout behavior.
|
||||
With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout-<session>-<timestamp>.log` containing the subagent's config snapshot, credential-resolution trace, any early error messages, and stack traces for **all** live threads (not just the child's own) — a child parked waiting on a nested helper thread is indistinguishable from a slow provider without the full picture.
|
||||
:::
|
||||
|
||||
## Stall Detection for Background Subagents
|
||||
|
||||
Background delegations (`delegate_task(background=true)`) are watched by a
|
||||
**progress-based stall monitor** — on by default, zero config. Unlike a
|
||||
wall-clock timeout, it never touches a child that is making progress, no
|
||||
matter how long it runs.
|
||||
|
||||
The monitor samples each detached child's progress signals — API-call count,
|
||||
current tool, and last-activity timestamp (which ticks on **every streamed
|
||||
token**, tool transition, and API-call boundary, so a child mid-stream on a
|
||||
long response always counts as alive):
|
||||
|
||||
1. **Progressing children are never touched.** Any advancing signal resets
|
||||
the clock.
|
||||
2. A child whose progress is completely frozen past the stale threshold
|
||||
(450s idle, 1200s while inside a tool — legitimately slow terminal
|
||||
commands and web fetches get the higher ceiling) is **interrupted** and
|
||||
given a 120s grace window. A child that unwinds in time delivers its
|
||||
partial results through the normal completion path.
|
||||
3. A child that never returns is force-finalized with a terminal `stalled`
|
||||
completion event, so the owning session hears an outcome instead of
|
||||
going silent, and the async slot frees for new work.
|
||||
|
||||
The `stalled` event carries structured metadata mirroring the sync-path
|
||||
timeout fields: `stalled_after_quiet_seconds`, `stall_threshold_seconds`,
|
||||
`stall_phase` (`idle` / `in_tool`), and `stall_grace_seconds`.
|
||||
|
||||
This closed a long-standing failure mode where a wedged background child
|
||||
left its session looking dead until a process restart. The underlying wedge
|
||||
(children hanging at their first API call after multi-day gateway uptime)
|
||||
was also fixed at the root: delegated children now run their OpenAI-wire
|
||||
API requests inline on their own conversation thread instead of a nested
|
||||
worker thread — the layer where the wedge lived. The stall monitor remains
|
||||
as the safety net for anything else.
|
||||
|
||||
|
||||
## Monitoring Running Subagents (`/agents`)
|
||||
|
||||
The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegate_task` fan-out into a first-class audit surface:
|
||||
|
|
@ -206,6 +250,22 @@ The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegat
|
|||
|
||||
The classic CLI just prints `/agents` as a text summary; the TUI is where the overlay shines. See [TUI — Slash commands](/user-guide/tui#slash-commands).
|
||||
|
||||
On the classic CLI and every gateway platform (Telegram, Discord, Slack, ...),
|
||||
`/agents` also lists **background delegations with live per-child activity**,
|
||||
sampled directly from each running child:
|
||||
|
||||
```
|
||||
Background delegations: 1 running
|
||||
- deleg_ab12cd34 · running · research the delegation stall monitor
|
||||
- child 1: 4 api calls · in web_search · active 12s ago
|
||||
- child 2: 7 api calls · between turns · active 3s ago
|
||||
```
|
||||
|
||||
A delegation the stall monitor has flagged shows as
|
||||
`stalling · no progress 450s — interrupting`, and long-quiet-but-healthy
|
||||
children show their quiet time so you can tell "slow" from "stuck" at a
|
||||
glance.
|
||||
|
||||
## Live Transcripts
|
||||
|
||||
Every `delegate_task` dispatch also creates one **append-only, human-readable log per task** so you (or the parent agent) can watch a subagent work in real time instead of waiting for the consolidated summary:
|
||||
|
|
|
|||
|
|
@ -750,6 +750,7 @@ const sidebars: SidebarsConfig = {
|
|||
link: {type: 'doc', id: 'developer-guide/plugins/index'},
|
||||
items: [
|
||||
'developer-guide/plugin-llm-access',
|
||||
'developer-guide/subagent-lifecycle-api',
|
||||
'developer-guide/desktop-plugin-sdk',
|
||||
'developer-guide/memory-provider-plugin',
|
||||
'developer-guide/context-engine-plugin',
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue