Commit graph

169 commits

Author SHA1 Message Date
atakan g
3d30232eba fix(delegate): isolate async batches from parent interrupts
Detached background delegation batches (_batch_runner) no longer honor
the foreground parent's interrupt flag — a busy-submit interrupt in the
TUI/desktop previously fabricated 'interrupted' results for background
children that should outlive the turn. Explicit cancellation still works
via _batch_interrupt.

Rebased onto current main from PR #65040; both interrupt-suppression
regression tests aligned with the current _session test helper.

Original work by @AtakanGs in #65040.
2026-07-28 14:20:10 -07:00
Alex Fournier
14bed44c8c Reapply "feat(observability): integrate NeMo Relay runtime and shared metrics"
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 21:10:51 -07:00
Jeffrey Quesnelle
841a5a744a
Revert "feat(observability): integrate NeMo Relay runtime and shared metrics" 2026-07-27 22:28:08 -04:00
Alex Fournier
8ac686fa27 Merge remote-tracking branch 'origin/main' into fix/hermes-relay-review-round3
Signed-off-by: Alex Fournier <afournier@nvidia.com>

# Conflicts:
#	agent/chat_completion_helpers.py
2026-07-27 10:11:53 -07:00
rob-maron
02d5e23085 nous portal anthropic wire 2026-07-27 11:53:48 -04:00
Alex Fournier
4fe4b0dca7 Merge origin/main into feat/hermes-relay-shared-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-27 08:12:37 -07:00
Tony Simons
f60abd6e37 fix subagent lifecycle ownership invariants 2026-07-26 23:27:30 -07:00
teknium1
ece050ac30 fix(delegation): route delegated-child API calls inline to avoid nested-pool wedge (#60203)
Root cause: delegate_task children run through three nested daemon-thread
layers (async-delegation executor -> per-child timeout executor -> the
interrupt worker interruptible_api_call spawns). After multi-day gateway
uptime the deepest layer wedges BEFORE the socket opens — the same
fingerprint as the gateway-cron hang (#62151): zero stale-detector output
(the worker never reaches dispatch), all providers, foreground/restart
works. The cron fix (should_use_direct_api_call) explicitly excluded
delegation 'for lack of evidence' — #60203 is that evidence.

- should_use_direct_api_call: extend the inline gate to delegated
  children, detected via the delegation ContextVar set by
  _run_single_child (platform='subagent' stamp as fallback). Scope
  unchanged otherwise: chat_completions wire only; Codex/Anthropic/
  Bedrock/MoA keep their established workers. Interrupts still work —
  the inline path registers _active_request_abort, which interrupt()
  invokes cross-thread (same mechanism the #72227 stall monitor uses).
- _dump_subagent_timeout_diagnostic: dump ALL thread stacks (bounded,
  40), not just the conversation worker — a pre-HTTP wedge is
  indistinguishable from a slow provider without seeing where the
  nested helper threads sit.
2026-07-26 20:37:51 -07:00
embwl0x
a8c9ad0bcc fix(delegate): strip URL userinfo from tool history 2026-07-26 20:36:47 -07:00
embwl0x
e369d6ea3f feat(delegate): expose redacted child tool history 2026-07-26 20:36:47 -07:00
Jeffrey Quesnelle
9216198601
Merge branch 'main' into feat/hermes-relay-shared-metrics 2026-07-26 23:14:26 -04:00
mahdiwafy
8e163852d8 fix(delegate): include explicit timeout metadata in subagent results
Add timeout_seconds, timed_out_after_seconds, and timeout_phase to timeout
results so parent agents and users can distinguish timeouts before the
first LLM call from timeouts after one or more API calls.

Also attach diagnostic_path to the N>0 API-call timeout error message,
matching the existing zero-API-call timeout path.

Addresses part of #51690 and #17308.
2026-07-26 17:13:52 -07:00
teknium1
5f5afb1eef fix(delegation): count streamed tokens as liveness in the stale monitor
Include last_activity_ts in the progress token sampled from each child.
_touch_activity ticks on every streamed chunk ('receiving stream
response'), every tool transition, and API-call start/completion — so a
child mid-stream on a long response is alive even though api_call_count
only advances when the call completes. Same liveness signal as the
compaction inactivity budget (PR #71508): if tokens are flowing it never
dies; staleness is measured from the last streamed token / tool activity
/ API call.
2026-07-26 16:15:29 -07:00
teknium1
99a381f310 fix(delegation): progress-based stale detection for detached async runners
Replace the wall-clock timeout watchdog (from #60234) with progress-based
staleness detection, on by default with zero config:

- The async registry now accepts a progress_fn per dispatch; delegate_task
  wires a sampler over the batch's child agents (api_call_count +
  current_tool from get_activity_summary()).
- A single monitor thread sweeps running delegations: a child whose
  progress token keeps advancing is never touched, no matter how long it
  runs. A frozen token past the stale threshold (450s idle / 1200s
  in-tool, mirroring the sync-path heartbeat monitor) marks the record
  'stalling' and interrupts the child.
- A stalling child that unwinds within the grace window (120s) finalizes
  through the NORMAL path, preserving its partial results. One that never
  returns is force-finalized with a terminal 'stalled' completion event so
  the owning session hears an outcome and the async slot frees.
- Late runner returns after force-finalization are deduped by the
  begin/push/finish finalization split (kept from #60234).

Why not a timeout: delegation.child_timeout_seconds defaults to 0 by
deliberate design (DEFAULT_CHILD_TIMEOUT rationale) — a timeout-based
watchdog never arms for default configs, leaving the reported silent-
profile symptom (#60203) unfixed, and when armed it kills legitimately
slow heavy subagents mid-task. Progress detection distinguishes 'wedged
at first API call' from 'grinding through a 2h review'.

Builds on izumi0uu's finalization-atomicity work from #60234.
2026-07-26 16:15:29 -07:00
izumi0uu
65420cdecd fix(delegation): timeout stuck async child runners
Async background delegation can leave gateway sessions holding only a dispatched handle when the detached runner wedges before it can return and enqueue a completion. Enforce the configured child timeout in the async registry so the parent observes a terminal timeout event and the async slot is released.

Constraint: Issue #60203 reports long-lived gateway processes with background child delegates that never produce completion events despite child_timeout_seconds being configured.

Rejected: Relying only on _run_single_child timeout handling | it cannot finalize the async registry when the outer runner thread itself never reaches normal completion.

Confidence: high

Scope-risk: narrow

Directive: Keep background delegation completion owned by the async registry whenever detached workers can outlive the caller's immediate control.

Tested: .venv/bin/python -m pytest tests/tools/test_async_delegation.py tests/tools/test_delegate_subagent_timeout_diagnostic.py tests/tools/test_delegate.py -q

Tested: .venv/bin/python -m ruff check tools/async_delegation.py tools/delegate_tool.py tests/tools/test_async_delegation.py

Tested: git diff --check

Not-tested: Multi-day real gateway degradation; covered with deterministic stuck-runner registry tests.
2026-07-26 16:15:29 -07:00
Alex Fournier
fb1e417367 Merge remote-tracking branch 'origin/main' into merge/relay-metrics-upstream-20260723 2026-07-23 14:10:49 -07:00
Alex Fournier
a6fbff1d7e fix(relay): retain active child sessions
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-23 13:22:50 -07:00
Ian Ker-Seymer
246eacea7b fix(gateway): deliver kanban/delegate wake-ups to api_server sessions
Wake-ups for kanban notifications and background delegation completions were
injected via handle_message() using a build_session_key()-derived key, which
can never match the raw X-Hermes-Session-Id key that api_server sessions run
under — so the wake landed in a session nobody was reading. On top of that,
ApiServerAdapter.send() reports failure without raising, and that was treated
as a successful delivery, so the notify cursor advanced past events that were
permanently lost; and background delegation was forced synchronous on
api_server since there was no way to wake the session afterward.

Fix: route wake-ups for non-push adapters through a self-post to
/v1/chat/completions with the original session id, treat non-raising send
failures as failures (rewind instead of advancing the cursor), and re-enable
background delegation whenever a session id is available to wake.

The origin session id is captured from the request-scoped api_server chat_id
binding rather than HERMES_SESSION_ID: constructing a child agent calls
set_current_session_id() with the subagent's internal id, clobbering that
variable right before dispatch would read it and misrouting the wake into
the subagent's own session.

Related: #56580, #64609, #53027, #63169, #56531, #50319, #64113
2026-07-23 11:55:17 -07:00
Mustafa
dc3e4e8428 fix(kanban): keep delegated results in worker turn
Dispatcher-spawned Kanban workers are finite one-shot processes, so detached delegation completions can outlive their only consumer. Mark that runtime as unable to deliver async completions and reuse the synchronous delegation fallback, returning required child results before the worker exits.\n\nAlso make unsupported-session notes runtime-generic and cover the delayed-child lifecycle regression.\n\nRefs #63169
2026-07-23 08:33:55 -07:00
trkim
47bbc12e18 fix(kanban): isolate delegated children from parent task 2026-07-23 07:33:36 -07:00
Alex Fournier
761fce7f79 Merge origin/main into feat/hermes-relay-shared-metrics
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-22 08:55:25 -07:00
Teknium
295e20358c
feat(delegation): let subagents use execute_code (#69325)
Children inherit the parent's env, repo, and toolsets but were denied
execute_code ('children should reason step-by-step, not write scripts').
That forces subagents doing mechanical multi-step work (batch file
reads, fetch-N-pages loops, filter-before-context reductions) to burn
reasoning iterations one tool call at a time.

- Remove execute_code from DELEGATE_BLOCKED_TOOLS
- Stop stripping the code_execution toolset from child bundles
- No recursion risk: the sandbox bridges only the 7 SANDBOX_ALLOWED_TOOLS
  (web/file/terminal) — delegate_task and execute_code itself are not
  reachable from inside a sandbox script
- Update schema text, AGENTS.md, and delegation docs
- Tests: blocked-constant, strip, and child-assembly tests updated;
  new test pins execute_code as intentionally unblocked
2026-07-22 06:58:45 -07:00
Alex Fournier
67863777ca Merge remote-tracking branch 'origin/main' into tmp/hermes-relay-pr-merge-20260717
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-20 11:07:39 -07:00
Teknium
299e409f15
feat(delegation): live-viewable subagent transcripts — tail your subagents while they work (#67479)
* feat(delegation): live-viewable subagent transcripts for delegate_task

Each child now streams an append-only, human-readable log to
<hermes_home>/cache/delegation/live/<delegation_id>/task-<n>.log while it
runs, and the dispatch return includes the paths so the caller can tail
them immediately instead of waiting blind for the consolidated summary.

- New tools/delegation_live_log.py: LiveTranscriptWriter (per-event append
  + flush, one-line rendering with truncation, never raises into the agent
  loop), wrap_progress_callback (tees the child's existing
  tool_progress_callback events into the log, preserves the _flush
  contract), dispatch-time creation with pre-headered files so tail -f
  attaches immediately, manifest.json (goals/task count/per-task status),
  and 7-day retention pruning on new dispatches.
- delegate_task: wraps each child's progress callback with the writer;
  sync results and background dispatch responses gain live_transcripts
  (+ hint field on dispatch); per-task result entries carry
  live_transcript; transcripts finalized with exit-reason markers.
- async_delegation: dispatch_async_delegation_batch accepts an optional
  delegation_id so the live/ dir name matches the returned handle; the
  completion event carries live_transcripts.
- process_registry: consolidated batch-completion block references each
  task's live transcript path.
- Tool schema description documents the live_transcripts return surface;
  docs gain a 'Live Transcripts' section with a tail -f example.

Placement under cache/delegation means the logs are mounted read-only
into remote terminal backends for free. Side-channel only: zero changes
to message content, so prompt caching is unaffected. Transcript-OUT only
— no overlap with the subagent control surfaces of PR #66046.

* fix(delegation): label the kickoff transcript line as user — it is the child's one user message
2026-07-19 10:29:14 -07:00
Alex Fournier
4dedaa4237 refactor(runtime): consolidate Relay lifecycle ownership
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:59:06 -04:00
Alex Fournier
9bc521b138 refactor(runtime): manage Relay LLM tool and subagent execution
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:57:38 -04:00
Alex Fournier
056e7df0e0 fix(observability): harden Relay metrics isolation and aggregation
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-07-19 08:55:07 -04:00
Mason Tanguay
1aadb02eaf fix(delegation): stop mixed platform bundles from re-exposing blocked tools to leaf children
A leaf subagent is meant to be denied delegate_task, execute_code, memory,
clarify, cronjob, and send_message. _strip_blocked_tools() only drops a
toolset when EVERY tool in it is blocked, so mixed platform bundles
(hermes-cli, hermes-telegram, and every other gateway bundle) survived
stripping and re-exposed the blocked tools after composite expansion. A
leaf child spawned from any gateway platform could recursively delegate,
run code, and write memory.

Pass exact one-tool deny toolsets into the child's disabled_toolsets so
model_tools subtracts the blocked names AFTER composite expansion, and the
restriction survives later registry/MCP refreshes. Orchestrators regain
only delegate_task.

Salvaged from #66036 by Mason Tanguay (@DictatorBacon); scoped to the
authority fix + its regressions (docs/interrupt changes dropped).

Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
2026-07-18 01:30:17 -07:00
Clifford Garwood
3d9be27895 fix(delegate): declare stateless channel in one-shot and cron so delegate_task returns results
run_agent._dispatch_delegate_task forces background=True for every top-level
delegation, and async_delivery_supported() returns True for any session that
never binds the capability. On runners that cannot receive a completion after
their turn ends, that combination silently discards every subagent result: the
model gets a dispatch handle, ends its turn, and reports 'waiting for results'.

Two such runners never bind the capability:

* hermes -z (one-shot) prints one final response and exits. It bypasses cli.py,
  so nothing drains process_registry.completion_queue (only the interactive
  process_loop and the gateway watchers do).

* cron run_job clears the HERMES_SESSION_* routing keys, so a completion event
  carries session_key="" — _enrich_async_delegation_routing cannot resolve it
  and _inject_watch_notification drops it ("no routing metadata"). By then
  run_job has already shipped the job's final response via _deliver_result;
  there is no turn left to re-enter. Worse, get_current_session_key() can fall
  back to the ambient os.environ HERMES_SESSION_KEY, so a cron subagent's output
  can be routed into an unrelated user chat rather than merely dropped.

Add declare_stateless_channel() and bind it in both runners, routing
delegate_task to its existing inline/synchronous path — the same fallback the
stateless HTTP adapter already relies on, and the fix suggested in #63142. The
helper binds only the capability: set_session_vars() would also latch
_session_context_engaged, which a pure single-process one-shot must not trigger.

Also correct two agent-facing strings that hardcoded 'stateless HTTP API' as the
only channel without async delivery (delegate_tool, terminal_tool); they now name
the actual condition.

Repro (before): hermes -z 'Use delegate_task to spawn a subagent that replies
BANANA. Report its reply.' -> "Waiting for the subagent's response...", exit 0,
no BANANA. After: BANANA is returned in-turn.

Fixes #53027
Fixes #63142
2026-07-18 00:05:25 -07:00
teknium1
b9267b50ea docs(delegation): fix stale internal batch-lifecycle comments
Two internal comments in delegate_tool.py still described the superseded
"N independent handles, no combined wait" model, contradicting the
authoritative batch contract (one async unit, one consolidated result
when all children finish). Aligns the comments with the runtime path in
_execute_and_aggregate / dispatch_async_delegation_batch.
2026-07-17 15:55:49 -07:00
Gille
35c578177b docs(delegation): align guidance with current contract 2026-07-17 15:55:49 -07:00
ethernet
5461e0e098 refactor(file-tools): resolve paths against per-session cwd records (step 2)
Flips the read side of the cwd rearchitecture onto the _session_cwd
store introduced in the previous commit.

_authoritative_workspace_root now resolves:
  1. the session's own cwd record (get_session_cwd) — per-session by
     construction, so one session's cd can never leak into another
     session's file resolution, with no ownership heuristics at all
  2. registered override (fallback for cleared/never-written records)
  3. legacy shared-env live cwd + preserved anchor (transition-only,
     for commands that ran before this code loaded)
  4. sentinel-free absolute TERMINAL_CWD

delegate_task children get their record seeded from the parent's at
spawn: they keep starting in the parent's directory (current behavior)
but their subsequent cds stay isolated in their own record instead of
bleeding back through the shared env.

The wrong-worktree leak class is now solved structurally on this path —
there is no shared cwd for sessions to inherit. The legacy env-side
tracking (cwd_owner, _live_cwd_if_owned, _last_known_cwd) remains only
as a transition fallback and is deleted in the next step.
2026-07-16 00:18:38 -07:00
Teknium
47d853fdf2 fix(delegation): fail closed on restored completions + stamp CLI dispatch identity
Three-layer companion to the salvaged CLI drain-ownership fix (#64240):

1. restore_undelivered_completions stamps restored=True (in-memory only)
   on every durable completion re-enqueued at process start.
2. drain_notifications' legacy unfiltered branch re-queues restored
   events instead of consuming them — a fresh process can no longer
   adopt a dead session's delegation results (#64484). Same-process
   keyless events keep the legacy behavior.
3. delegate_tool's async dispatch now falls back to the parent agent's
   durable session_id when the approval-context key resolves empty (the
   CLI case), so the CLI's new positive-ownership drain can actually
   claim its own completions instead of failing closed on ''.
2026-07-14 22:14:10 -07:00
kshitijk4poor
8727e67295 fix(runtime): preserve resolved fork metadata
Some checks are pending
CI / Detect affected areas (push) Waiting to run
CI / Python tests (push) Blocked by required conditions
CI / Python lints (push) Blocked by required conditions
CI / TypeScript (push) Blocked by required conditions
CI / Docs Site (push) Blocked by required conditions
CI / Deny unrelated histories (push) Blocked by required conditions
CI / Check contributors (push) Blocked by required conditions
CI / Check uv.lock (push) Blocked by required conditions
CI / Lint Docker scripts (push) Blocked by required conditions
CI / Build&Test Docker image (push) Blocked by required conditions
CI / Supply-chain scan (push) Blocked by required conditions
CI / OSV scan (push) Waiting to run
CI / All required checks pass (push) Blocked by required conditions
CI / CI timing report (push) Blocked by required conditions
Deploy Site / deploy-vercel (push) Waiting to run
Deploy Site / deploy-docs (push) Waiting to run
2026-07-10 18:50:28 +05:30
kshitijk4poor
a0032f5f92 fix(routing): preserve profile and delegation parity 2026-07-10 13:10:45 +05:30
nankingjing
d39c62409b fix(delegate): pin async completion to spawning parent session (#57498)
Background delegate_task completions only carried session_key. When multiple
active sessions shared a routing peer, get_or_create_session could recover the
latest ended_at IS NULL row and inject the subagent result into the wrong
session.

Capture parent_agent.session_id at dispatch time, include it on async-delegation
completion events, and pin gateway routing via switch_session when the
synthetic completion message is handled.

Fixes #57498
2026-07-08 08:10:28 -07:00
Dan Schnurbusch
aab351bfa6 fix(delegation): route async results to origin session
Carry the live TUI session id with async delegation completion events and prefer the commissioning UI session when desktop pollers share the completion queue. Resolve compressed session keys to their continuation before treating events as orphaned, and capture the live parent agent session id for TUI/ACP dispatch.
2026-07-08 07:06:15 -07:00
Shannon Sands
bf7639138e Use read-only config loader and honor HERMES_IGNORE_USER_CONFIG in delegation config 2026-07-07 17:17:38 -07:00
Shannon Sands
0263f1d12e Fix delegation config precedence 2026-07-07 17:17:38 -07:00
Teknium
55e3ee1ab8
fix: remove dead f-string prefixes via ruff F541 (216 sites) (#52336)
ruff check --fix --select F541 . on current main. Pure prefix removals;
adjacent-string concatenations keep the f only on interpolating fragments.
No string content or live placeholder altered.
2026-07-05 13:42:46 -07:00
Eugeniusz Gilewski
e4dbb67bf5 fix(security): remove model-controlled delegate ACP transport
Source: https://github.com/NousResearch/hermes-agent/pull/52346
Related prior work: https://github.com/NousResearch/hermes-agent/pull/39462
Related prior work: https://github.com/NousResearch/hermes-agent/pull/27426
Maintainer direction: https://github.com/NousResearch/hermes-agent/pull/52346#issuecomment-4854881612

Remove acp_command and acp_args from the model-facing delegate_task schema and
dispatch paths. Child agents can still use ACP subprocess transport when it
comes from trusted delegation config or parent inheritance, but a model tool
call can no longer choose the command or arguments that reach child
construction.

This is salvageable because the risky boundary is model control over child ACP
transport, not ACP itself. The patch follows the maintainer direction from the
source discussion by preserving trusted ACP configuration and prior integration
work while removing the untrusted tool-call fields from both top-level and
per-task delegate inputs.

Reproduced on main by passing acp_command through delegate_task and observing it
reach _build_child_agent. Verified after the fix that model dispatch strips the
hidden top-level fields and per-task hidden fields are ignored before child
construction.

Co-authored-by: Carlosian <claudlos@agentmail.to>
Co-authored-by: ssiweifnag <120658181+ssiweifnag@users.noreply.github.com>
Co-authored-by: nikshepsvn <23241247+nikshepsvn@users.noreply.github.com>
2026-07-03 03:27:47 -07:00
Brooklyn Nicholson
5a6720b884 fix(desktop,tui-gateway,zai): stop thinking-off from reverting to medium
A Z.ai desktop user reported thinking reverting to medium after one turn,
burning ~200% of a week's credits in 4 days despite reasoning_effort: false
in config.yaml. Four compounding bugs:

- _session_info reported reasoning_effort "" for disabled reasoning,
  indistinguishable from unset — the desktop adopted it after the first
  turn, wiping its sticky "thinking off" pick so every later chat
  reverted to the default effort.
- config.set key=reasoning always wrote agent.reasoning_effort to global
  config.yaml, so every desktop model-menu selection (preset.effort ??
  'medium') clobbered the user's configured value. Now session-scoped
  like the messaging gateway's /reasoning, landing on
  create_reasoning_override so lazily-built sessions keep it too.
- YAML `reasoning_effort: false`/`off`/`no` (boolean False) was coerced
  to "" by every loader's `str(x or "")`, silently re-enabling thinking.
  parse_reasoning_effort now treats False/"false"/"disabled" as
  {"enabled": False}; loaders (tui gateway, gateway, cli, cron,
  delegate) pass the raw value through. The desktop config reader also
  crashed on the boolean (false.trim()), aborting voice/STT settings.
- The zai provider profile never sent thinking on the wire, and GLM-4.5+
  defaults to thinking ON server-side — so disabling reasoning was a
  silent no-op on direct Z.ai, the actual token burner. The profile now
  emits extra_body.thinking {"type": "enabled"|"disabled"} for
  thinking-capable GLM models, mirroring the DeepSeek profile.

Also: /new (session reset) now carries reasoning_config across the
rebuild like model_override; config.get reasoning prefers the session's
live value and maps a config False to "none"; Settings shows "Off"
instead of a blank select for hand-written false.
2026-07-02 15:23:47 -05:00
Teknium
3f2a56d1a4
fix(cli): reliable interrupts, bounded exit, and exit feedback (#57000)
Three CLI reliability fixes:

1. Interrupt reliability: chat() only re-queued the user's interrupt
   message when the turn result carried interrupted=True. When the agent
   thread raced past its last interrupt check (or finished) before the
   interrupt landed, the message was silently dropped — and the stale
   _interrupt_requested flag left on the agent instantly aborted the
   NEXT turn. Un-acknowledged interrupt messages are now re-queued as
   the next turn and the stale flag is cleared (only when the agent
   thread actually exited). The clarify-race path also parks the message
   in _pending_input instead of dropping it.

2. Slow exit (5+ min): stdlib ThreadPoolExecutor workers are non-daemon
   and joined unconditionally by concurrent.futures' atexit hook — even
   after shutdown(wait=False). One wedged tool worker (abandoned after
   interrupt/timeout) held the process open forever. Promoted
   async_delegation's daemon executor to a shared tools/daemon_pool
   module and adopted it in tool_executor (concurrent tool batches),
   memory_manager (background sync), delegate_tool (child timeout wrapper
   + batch fan-out), and skills_hub (source fan-out). Added a 30s exit
   watchdog (HERMES_EXIT_WATCHDOG_S) armed at _run_cleanup start as a
   backstop for wedged cleanup steps.

3. Exit jank: after prompt_toolkit tears down the input/status bars the
   terminal sat silent for the whole cleanup window, looking hung. Print
   'Shutting down… (finalizing session)' immediately at exit start.

E2E: live PTY interrupt of a foreground 'sleep 120' terminal tool now
aborts in ~1s and the typed message runs as the next turn; wedged-worker
+ wedged-cleanup subprocess exits in 5.8s (watchdog) instead of hanging.
2026-07-02 04:20:43 -07:00
Teknium
6e369a3762
feat(delegation): unify concurrency caps — deprecate max_async_children (#56955)
delegation.max_concurrent_children is now the single cap for both a
batch's parallelism and concurrent background delegation units.

- _get_max_async_children() delegates to _get_max_concurrent_children();
  a leftover max_async_children key logs a one-time deprecation warning
- config v32→33 migration removes the stale key, folding a raised
  max_async_children into max_concurrent_children (max wins, no lost
  headroom)
- capacity error messages now point at max_concurrent_children
- pool-at-capacity sync fallback now attaches an explanatory note so
  the model/user know why the call blocked instead of dispatching async

Previously users who raised max_concurrent_children (e.g. to 15) still
hit the invisible default-3 async cap: the 4th background delegate_task
silently ran inline, blocking the turn with no signal.
2026-07-02 02:53:39 -07:00
Teknium
ba0bc01d1f
feat(delegate): remove model-facing toolsets arg — subagents always inherit parent's (#56386)
The model could pass `toolsets` (top-level and per-task) to delegate_task,
letting it choose which toolsets a subagent got. Toolset selection is a
capability-scoping decision the model should not control; subagents inherit
the parent's enabled toolsets, period.

- Remove `toolsets` from the delegate_task() signature, the registry handler,
  the top-level + per-task JSON schema, and the live dispatch path
  (run_agent._dispatch_delegate_task — this forwarded it on every model call).
- Single-task and per-task child builds now pass toolsets=None so
  _build_child_agent resolves to pure parent inheritance.
- Drop the now-dead _SUBAGENT_TOOLSETS / _TOOLSET_LIST_STR schema-hint block.
- _build_child_agent keeps its internal toolsets param + intersection helpers
  (internal API; fed the inherited value only).
- Tests: schema assertions flipped to assertNotIn; added a regression test
  proving the dispatch path never forwards a smuggled model `toolsets`.
- Docs: update delegate_task signature refs in the autonomous-ai-agents skill.
2026-07-01 05:35:26 -07:00
Justin Ohms
8f21311906 fix(delegation): route native-SDK providers through runtime resolver; fail on '(empty)' sentinel
Two related bugs caused subagent delegation to silently return empty summaries
with 0 tokens when the user configured delegation.provider=bedrock alongside
delegation.base_url=https://bedrock-runtime.<region>.amazonaws.com.

Root cause #1 — misrouting in _resolve_delegation_credentials():
  The configured_base_url branch unconditionally forced provider='custom' and
  api_mode='chat_completions', only specializing for chatgpt.com, anthropic,
  and kimi hosts. Bedrock (and other native-SDK providers) fell through as
  'custom' + chat_completions, which then POSTed OpenAI-shaped JSON at
  Bedrock's native API. Bedrock rejected the payload and returned nothing,
  which looked like an empty LLM response to the child agent.

  Fix: when provider is one of {bedrock, vertex, google, google-genai}, skip
  the base_url short-circuit and fall through to resolve_runtime_provider(),
  which knows how to construct the proper SDK client. base_url can still be
  forwarded through that path for regional overrides.

Root cause #2 — '(empty)' sentinel accepted as success:
  After N retries of empty LLM responses, run_agent.py emits the literal
  string '(empty)' as final_response. _run_single_child then hit
  `elif summary:` — '(empty)' is truthy, so status became 'completed' and
  the parent surfaced a blank result with no error. Users saw api_calls=4,
  tokens=0, duration~0.4s, status=completed.

  Fix: treat final_response.strip() == '(empty)' as a failure so the parent
  surfaces it instead of silently accepting zero-content 'success'.

Both paths were reproduced in a live Hermes TUI session on us-west-2 Bedrock
(provider=bedrock, model=us.anthropic.claude-sonnet-4-6) and are covered by
new tests in tests/tools/test_delegate.py.
2026-07-01 00:45:31 -07:00
nikshepsvn
d82a69b624 fix(tools): prune acp_command from delegate_task schema when no ACP CLI is on PATH
Defense-in-depth follow-up to the runtime guard added in the previous commit.
Models on headless hosts (Railway / Fly / Docker / fresh VPS) without any ACP
CLI installed occasionally hallucinate ``acp_command="copilot"`` from the
schema description, despite the explicit "Do NOT set" instruction. The runtime
guard prevented the crash but the model still wasted a tool turn and got an
opaque silent fallback.

This commit removes the temptation at its source: ``_build_dynamic_schema_overrides``
now strips ``acp_command`` and ``acp_args`` from both the top-level and per-task
schemas when none of the known ACP CLIs (``copilot``, ``claude``, ``codex``) are
detectable on PATH. The model literally never sees the fields, so it cannot
pass them.

The runtime guard from the previous commit stays in place as defense-in-depth
for internal callers, tests, and any future code path that bypasses the schema.

``_acp_binary_available`` is intentionally NOT cached: ``shutil.which`` is
cheap, and avoiding the cache means the schema reacts to mid-session installs
without requiring a process restart.

Tests:
- ``test_schema_prunes_acp_command_when_no_acp_binary``
- ``test_schema_keeps_acp_command_when_binary_available``
- ``test_acp_binary_available_checks_known_clis``

Full ``test_delegate.py`` suite: 136/136 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-30 03:41:46 -07:00
nikshepsvn
2e0b591076 fix(tools): validate acp_command binary exists before forcing copilot-acp transport
When a model passes `acp_command="copilot"` (or any other binary name) in a
`delegate_task` tool call, `_build_child_agent` unconditionally sets
`effective_provider = "copilot-acp"`, which routes the subagent through
`CopilotACPClient`. That client spawns the named binary via subprocess; if it
isn't on PATH, every retry raises RuntimeError and an asyncio cleanup race
during error delivery can take the entire gateway down.

This is a real failure mode on headless deploys (Railway / Fly / VPS / Docker)
where `copilot` / `claude` / etc. aren't installed. The schema does say
"Do NOT set unless the user explicitly told you an ACP CLI is installed,"
but models occasionally pass it anyway — particularly for X (Twitter) search
prompts where Grok seems to associate ACP with "search assistance."

Reproduction:
- Headless install (no `copilot` binary on PATH)
- Set provider to xai-oauth + model grok-4.3
- Telegram prompt: "Search X for crypto twitter trends"
- Grok decides to delegate and passes `acp_command="copilot"`
- Subagent crashes 3x, gateway crashes on the 3rd retry teardown

Fix: validate the binary exists on PATH via `shutil.which` before honoring
the override. If missing, log a warning and fall through to the parent's
default transport. No behavior change when the binary IS present (covered
by `test_build_child_agent_honors_acp_command_when_binary_present`).

Tests:
- `test_build_child_agent_ignores_acp_command_when_binary_missing`
- `test_build_child_agent_honors_acp_command_when_binary_present`

Verified on Python 3.11 (macOS) and 3.12 (Debian 13 container).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-30 03:41:46 -07:00
Markus Phan
cd9f5cc671 fix(delegate): route subagent progress lines through _safe_print for ACP stdio
delegate_task's per-task completion display emitted lines like
"✓ [1/3] Research done (17.92s)" via a bare print(). Under ACP (and any
headless JSON-RPC stdio host where AIAgent routes human output to stderr
via a custom _print_fn), these landed on stdout and corrupted the
protocol frame stream, surfacing as "Failed to parse JSON message: ✓
[3/3] …" in the ACP adapter.

Add _emit_parent_console() which prefers parent_agent._safe_print (the
same hook AIAgent uses for every other user-facing print) and falls back
to print() only when no router is wired up or it raises. CLI behavior is
unchanged.

The PR's other fix (preset toolset expansion) is already covered on main
by _expand_parent_toolsets(), so only the stdio-safe printing change is
salvaged here.
2026-06-30 03:16:22 -07:00
teknium1
35a0803a3b fix(delegation): budget subagent summaries against parent context headroom
Batch delegation returned each subagent's full final_response verbatim
into the parent's context. A fan-out of N children could dump 60k+ tokens
at once, blowing the parent's context window and — on rate-limited
providers — triggering a compression/429 death spiral (429 misread as
context-too-large -> window step-down -> retry loop -> conversation dies).

Cap each summary against the parent's *remaining* context headroom split
across the batch (not a magic char count). When trimming, mirror the
web_extract convention: spill the full text to cache/delegation (mounted
into remote backends via credential_files._CACHE_DIRS) and return a
head+tail window (75/25, line-snapped) plus a footer with the exact
read_file offset to page the omitted middle. Both the subagent's opening
AND its closing (outcomes / files-changed / issues, which live at the end)
survive in-context, and nothing is lost — the parent can read_file the
full version on any backend.

delegation.max_summary_chars (default 24000) is a static ceiling layered
on top as belt-and-suspenders for models that ignore 'be concise'; 0
disables it. Child prompt tightened to lead with outcomes / bullets.

Co-authored-by: rc-int <rcint@klaith.com>
2026-06-30 03:07:40 -07:00