When a user explicitly configures a platform with its native composite
(e.g. platform_toolsets.discord: [hermes-discord]), the discord and
discord_admin toolsets were silently stripped by _DEFAULT_OFF_TOOLSETS
even though the composite contains those tools. The strip could not tell
an explicit composite opt-in apart from the unconfigured default.
Track whether the platform was explicitly configured and, when it was,
exempt toolsets that are both default-off and platform-restricted to the
current platform from the strip. Only discord/discord_admin are affected
(the sole entries in both _DEFAULT_OFF_TOOLSETS and
_TOOLSET_PLATFORM_RESTRICTIONS). Unconfigured and empty-list platforms
keep the security default-off behaviour.
/api/status (loopback/insecure binds only) now includes:
- profiles: every profile on the host (default + named)
- gateway_mode: none | single | multiple | multiplex
- gateways: one entry per live gateway with the host ports its
port-binding platforms listen on, plus served_profiles when the
default gateway is multiplexing
Ports resolve from each profile's config.yaml (top-level platforms:
wins over gateway.platforms:, matching load_gateway_config precedence)
with adapter defaults as fallback. Topology enumeration runs in an
executor so the profile scan + process-table probes stay off the event
loop, and the whole block is gated behind the same loopback-only split
as hermes_home/gateway_pid so gated binds leak nothing new.
signal.SIGKILL / os.killpg don't exist on Windows. The watchdog is only
spawned on POSIX (wrap site gates on os.name), but guard via getattr with
a plain terminate/kill fallback so an accidental Windows import can't
AttributeError.
The salvaged test predates the parked-server self-probe
(_PARKED_RETRY_INTERVAL, landed on main after the PR branched): after the
final failed retry, run() parks in a real asyncio.wait that the patched
asyncio.sleep doesn't cover, stalling the test 300s. Signal shutdown once
the retry budget is exhausted so the park exits immediately.
Two fixes on top of the salvaged parent-death watchdog:
- Apply the watchdog wrap AFTER the OSV malware preflight so the check
inspects the real npx/uvx package instead of the python wrapper
(the wrap previously made the preflight a silent no-op for every
stdio server).
- The real server runs in its own process group under the watchdog, so
the graceful-shutdown killpg no longer reached it; the watchdog now
forwards SIGTERM/SIGINT to the child's group, keeping wedged servers
killable on clean shutdown.
A stdio MCP server (e.g. `npx -y mcp-remote <url>`) is spawned as a direct
child of the Hermes process. Existing teardown (MCPServerTask.shutdown() /
_kill_orphaned_mcp_children()) reaps it correctly on a clean exit, but a
kill -9 / crash / force-quit of the Hermes process skips that path entirely
-- the child (and its own descendants, e.g. mcp-remote's spawned node
process) is orphaned and keeps running. Repeated ungraceful restarts pile up
N orphaned processes racing to hold the same upstream SSE session, producing
errors like 'Invalid request parameters' on legitimate reconnects.
macOS/Linux have no portable equivalent of prctl(PR_SET_PDEATHSIG) at the
Python subprocess level, so this adds a thin supervisor
(tools/mcp_stdio_watchdog.py) that:
- execs the real command as its own child in its own process group
- passes stdin/stdout/stderr through untouched (MCP stdio protocol
talks directly over those streams)
- polls the original spawning PID with the same orphan-detection
algorithm already proven in tui_gateway/slash_worker.py (ppid
comparison + psutil creation-time guard against PID reuse)
- SIGTERM-then-SIGKILL's the child's process group the moment the
original parent is gone
Wired into _run_stdio via a new _wrap_command_with_watchdog() helper,
POSIX-only (matches the existing killpg-based cleanup's platform scope),
fails open (any error resolving pid/create-time falls back to the
unwrapped command) so this can never be the reason a working MCP server
stops starting.
Verified: reproduced the exact orphan scenario standalone (fake parent
process spawns watchdog + fake long-running MCP child, kill -9 the fake
parent, confirm the watchdog reaps the child within its poll window with
zero leaked processes). Updated test_mcp_tool_issue_948.py's resolved-path
assertion to check the watchdog-wrapped command instead of the raw
resolved binary. Full test_mcp_tool.py + test_mcp_stability.py +
test_mcp_tool_issue_948.py suite: 232 passed. Full -k mcp sweep across the
whole test tree: 1003 passed, 2 skipped, 0 failed.
Sibling sites of the same bug class as the salvaged stdio fix:
- SSE, streamable-HTTP (new + deprecated API) initialize() calls are now
bounded by the same connect_timeout, so an endpoint that accepts the
connection but never answers the handshake cannot park the run() task
forever.
- start() now cancels its ensure_future'd run() task when the caller's
connect timeout cancels start() itself — the orphaned-task leak was
the root mechanism behind #59349, and this closes the class for any
future pre-ready hang.
A stdio MCP server that never completes `initialize` (e.g. emits a
non-JSON-RPC frame and then blocks on stdin) leaks a child process plus its
stdio pipes/pidfd on every discovery-retry cycle — unbounded, until the
gateway hits EMFILE and every new open()/spawn fails (#59349).
Root cause (confirmed by instrumenting the live repro, and different from the
issue's own hypothesis): the spawned child IS captured in `new_pids`, so the
report's "new_pids empty at finally" guess is not it. The real cause is that
`session.initialize()` hangs forever on the garbage stream. `connect_timeout`
only bounds the caller's `.result()` wait on the foreground thread — it does
NOT cancel the `_run_stdio` coroutine on the background MCP loop. So the
coroutine is stuck at `await session.initialize()` permanently, its cleanup
`finally` never runs, the child is never reaped, and it stays invisible to the
orphan-reaper (whose `_orphan_stdio_pids` set never gets populated).
Fix: wrap `session.initialize()` in `asyncio.wait_for(..., connect_timeout)`
so a stalled handshake fails instead of hanging. The TimeoutError unwinds
through the SDK context managers (closing the child's stdin -> EOF -> exit)
and lets the existing `finally` reap any straggler. Cross-platform — no
signals/pgid/proc.
Scope: stdio only. The HTTP path has the same `await session.initialize()`
shape but spawns no subprocess (so it can't cause this leak) and already has
httpx transport timeouts.
Verified: the reporter's repro goes from unbounded growth to draining to zero;
added a hermetic regression test (fake transport whose `initialize()` hangs,
asserts the connect is bounded by connect_timeout) that fails on the pre-fix
code and passes on the fix; 566 existing MCP tests pass; ruff clean.
Repro confirmed on macOS (pipe FDs); the Linux-specific pidfd growth in the
report should be equivalent — the reporter offered to validate on Linux.
Closes#59349
Merge the two cherry-picked reap call sites into one unscoped sweep at
the top of _run_stdio (the unscoped sweep is a superset of the
per-server one), and run it via asyncio.to_thread so the 2s
SIGTERM->SIGKILL escalation cannot stall the shared MCP event loop.
When an MCP stdio subprocess fails to connect (token expiry, port
contention, timeout), the run() reconnect loop retries with backoff.
Each retry calls _run_stdio() which spawns a new process pair, but the
previous failed pair was only detected as orphaned (added to
_orphan_stdio_pids) — never actually killed. This caused rapid zombie
accumulation: 5 failed attempts × 2 procs each = 10 orphans competing
for the same port.
Add a _kill_orphaned_mcp_children() call at the top of _run_stdio(),
before the _snapshot_child_pids() baseline, so any orphans from prior
failed attempts are reaped before a new subprocess is spawned.
Fixes#57355
ChatPage sends ?attach=<localStorage token> so /chat reattaches to its
live PTY across refresh. onclose: 4410=process-exit (session ended),
4409=superseded (quiet), else transient -> auto-reconnect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Keep-alive path when ?attach=<token> is present: PTY outlives the socket
via PTY_REGISTRY, reattaches on reconnect. No token = unchanged legacy
pump (_legacy_pump). detach (not close) on disconnect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_run_approval_gate's gateway branch only queued via submit_pending, so
plugin-escalated approvals never sent the interactive embed+buttons on
Discord/Telegram/Slack (#59413) - the user was never notified and the
action stayed silently blocked. Mirror check_dangerous_command's path:
when a session notify callback is registered, run the blocking
_await_gateway_decision round-trip (redacted payload, once/session/
always persistence, deny/timeout produce definitive BLOCKED outcomes);
fall back to submit_pending only when no callback exists.
Fixes#59413.
Follow-up on the #56966 salvage:
- is_provider_explicitly_configured(): an env-seeded credential-pool entry
only counts as explicit while its env var still resolves to a usable
secret. A stale auth.json entry left behind after the user deletes the
var no longer keeps the provider in the picker forever (#55790).
- TUI modelPicker + dashboard ModelPickerDialog/api.getModelOptions pass
include_unconfigured=true explicitly, preserving their full-universe
setup-affordance behavior now that the backend defaults to the
configured subset.
- desktop lib/model-options.ts routes explicit_only through the shared
requestModelOptions() helper (added on main after the PR branched).
- regression tests for ambient (gh_cli) pool sources, explicit manual/
device-code sources, and stale vs live env-seeded entries.
* feat(trace): upload sessions to HF Agent Trace Viewer
Salvage trace upload as a smaller CLI-first feature: deterministic Claude Code JSONL export, fail-closed redaction, lazy Hugging Face dependency, and no gateway slash-command wiring.
* chore(trace): drop external porting references from docstrings
Describe the trace-upload design in Hermes' own terms.
* feat(sessions): fold trace upload into 'sessions export --format trace'
Integrates the HF Agent Trace Viewer exporter (PR #36145) onto the
unified export surface instead of a separate 'hermes trace' subcommand:
- --format trace: Claude Code JSONL to stdout/file, or one
<id>.trace.jsonl per session for filtered bulk export; defaults to
the most recent session when no --session-id/filters given.
- --upload pushes to the user's private HF traces dataset (--public to
opt out of private); reads HF_TOKEN with guided setup when missing.
- traces are secret-redacted by default (force mode); --no-redact opts
out after review; redaction failure blocks export (fail closed).
- hermes_cli/trace.py + subcommands/trace.py removed; agent/trace_upload.py
is the single engine. Docs EN + zh-Hans; 4 new CLI tests.
The salvaged SelfHostedBackend made self-hosted servers reachable via
mem0.json / MEM0_HOST, but the setup wizard still offered only Platform
and OSS — exactly the gap users hit (Discord report: 'At memory setup
there's only 2 options'). Adds a third wizard mode:
- interactive picker: Platform / Self-hosted server / Open Source
- non-interactive: hermes memory setup mem0 --mode selfhosted
--host http://... [--api-key ...] [--dry-run]
- host -> mem0.json (behavioral), API key -> .env as MEM0_API_KEY
(secret), optional key for AUTH_DISABLED servers
- best-effort reachability check against the server, non-fatal
- README + memory-providers docs updated with the wizard path
`_save_anthropic_oauth_creds` wrote the Anthropic OAuth token file with
`os.replace(tmp, path)` followed by a post-hoc `chmod(0o600)`. Between the
rename and the chmod the token file existed at the default umask (0o644 on most
hosts) — a window in which another local user could read the access/refresh
tokens.
Write via `utils.atomic_json_write(..., mode=0o600)`, which creates the temp
with mode 0o600 *before* any content is written, fsyncs, atomically replaces,
preserves the existing file's owner, and cleans up its temp on failure. This
matches the `atomic_json_write(mode=0o600)` call already used elsewhere in this
module for the credential-pool write, and #56644's owner preservation.
Tests updated for the new mechanism, plus a check that the write goes through
`atomic_json_write(mode=0o600)` (mutation-verified).
Opt-in discord.approval_mentions (config.yaml, bridged to
DISCORD_APPROVAL_MENTIONS) prepends <@id> mentions for numeric
allowlist entries to exec-approval prompts, with a scoped
AllowedMentions override (users only). Default off - no surprise
pings. Reapplied onto the content-mirror layout from #60245: mentions
prepend to the visible content block and its truncation budget.
Original implementation from PR #39719; commits arrived bot-authored,
re-attributed to the contributor.
Salvage follow-up integrating PR #30481 (@simplast) and PR #57683
(@catbearlove1-lang) into the unified export surface:
- --format html: standalone self-contained HTML transcript (single
session or multi-session with sidebar), works with all shared filters
and --redact; requires a file output path.
- --only user-prompts: prompt-only export (jsonl records or md sections)
via the shared session_export renderer; the separate export-prompts
subcommand from the original PR is subsumed by this flag.
- AUTHOR_MAP entries for both contributors; docs EN + zh-Hans.
Implements a professional, standalone HTML export feature for Hermes sessions.
Key changes:
- Adds 'hermes sessions export <file>.html' support to the CLI.
- Implements a dark-mode-first, responsive HTML generator in 'hermes_cli/session_export_html.py'.
- Single session export features a focused, centered 90% width layout.
- Multi-session export adds a fixed sidebar with session switching and real-time search filtering.
- ZERO external dependencies; all styles and JS are embedded for offline portability.
Review follow-ups on the salvage:
- get_all() pruned from the ABC and all three backends: mem0_list (its
only caller) was removed by the recall-tuning commit, leaving new,
tested, unreachable code — including SelfHostedBackend's _MAX_TOP_K
over-fetch workaround. Tests for it dropped; fake-class stubs remain
harmlessly. (The #52921 true-total fix lives on in the PR history if
a lister ever returns.)
- The persisted rerank config key was write-only (setup prompted for it,
nothing read it). initialize() now parses it into _rerank_default and
mem0_search uses it when the model doesn't pass rerank explicitly;
per-call args still win. Guard test added.
- Platform-mode setup now warns when MEM0_HOST is set in the environment:
the json host-clear can't help there (_load_config seeds host from the
env var, docs tell users to put it in .env) — the user would silently
keep routing to the self-hosted server.
- SelfHostedBackend: connect-level retries (httpx.HTTPTransport(retries=2))
so a single transient blip doesn't count toward the provider breaker;
transport now injectable and the test helper uses the real __init__
instead of mirroring it via __new__.
- plugin.yaml description no longer leads with reranking (off by default,
platform-only); docs em-dash typo fixed.
Follow-up on the salvaged #55614. The PR added host-based routing to
_create_backend (precedence: oss > host > platform) but two sibling surfaces
didn't mirror it:
- system_prompt_block() checked host before oss, so an oss+host config ran
OSS but told the model it was self-hosted HTTP. Reordered to match routing.
- Platform setup (hermes memory setup mem0 --mode platform) left a stale host
in mem0.json; since host beats platform, the user kept routing to the
self-hosted server. save_config merges (no delete), so clear host to ""
rather than pop() so the merge actually overwrites it.
Adds regression tests for both (mutation-checked).
Salvage of #55614 by @kartik-mem0 (mem0 maintainer). Adds a SelfHostedBackend
that talks to a self-hosted Mem0 Docker server over httpx (X-API-Key auth,
/search + /memories routes), gated behind `host`. Also folds in the mem0
research-team recall tuning that rides with it: rerank defaults to false across
all modes, the mem0_list tool is removed (5->4 tools), search guidance is
de-shouted, and self-hosted get_all reports the true stored total (#52921).
Supersedes the self-hosted portion of #52487 (@liuhao1024, first-submitted).
Closes#52478Fixes#52921
Review findings on the salvaged shim: (a) OpenAI callers may pass stop as
a bare string but Converse's stopSequences requires a list — normalize;
(b) call_llm(stream=True) (MoA aggregator) can reach this client and the
shim silently returned a complete response — keep that behavior (the
streaming consumer's got-final-object path downgrades gracefully) but log
it, and log dropped tool_choice, instead of silently ignoring both.
+2 regression tests.
Follow-up to the salvage of #60217 by @xxxigm.
Auxiliary Bedrock resolution always used the Anthropic Bedrock SDK, which
only works for Claude foundation-model IDs. Non-Claude models such as
openai.gpt-oss-20b-1:0 now use a Bedrock Converse adapter, matching the
main agent's bedrock_converse transport.
Review findings on the salvaged #60332 breaker, fixed as follow-ups:
- restore_primary_runtime() now resets the streak (third provider-swap
path; without it a recovered primary was short-circuited before a
single attempt and could never be re-proven healthy except via /model).
- interruptible_api_call (non-streaming) now carries the same breaker
(guard at entry, bump on stale_call_kill, reset on success). Quiet-mode
/ subagent / headless sessions — the profile most like #58962's
unattended 494-failure session — take this path and had the identical
infinite stale-retry class.
- Partial-stream stub return now resets the streak (chunks were received,
provider demonstrably responsive).
- Consolidated the triple-duplicated counter arithmetic into shared
helpers (_stale_streak/_bump_stale_streak/_reset_stale_streak/
_check_stale_giveup) with one canonical comment block; error message
now says 'consecutive stale attempts' (the counter counts kills, not
turns — a single turn can produce several).
4 new tests (restore resets / no-op restore keeps latch / non-streaming
short-circuit / non-streaming success reset).
