The connector now depends on the single multiplexed gateway for per-profile
relay routing, so hosted deployments need to FORCE multiplexing on regardless
of the image's config.yaml. gateway.multiplex_profiles was config.yaml-only,
which a user could leave unset or flip off.
Add GATEWAY_MULTIPLEX_PROFILES as a standard operator override on top of the
existing config key — the same 'config.yaml is canonical, env is the operator
override' pattern the Telegram/Signal require_mention bridges use:
env (recognized token) > config.yaml (top-level or nested gateway.*) > False
- gateway/config.py: _env_multiplex_profiles_override() resolves the env var
tri-state — recognized truthy/falsy token → bool; unset/blank/unrecognized
→ None (fall through to config). Blank is deliberately None, not False, so a
provisioned-but-unpopulated Fly secret ('') can't shadow a config.yaml opt-in
(the empty-secret trap). Wired into GatewayConfig.from_dict so every consumer
(run.py, session.py via self.config) sees the resolved value.
- hermes_cli/gateway.py: the named-profile-start guard
(_guard_named_profile_under_multiplexer) reads config.yaml directly, so it
gets the SAME env precedence — otherwise env-forced multiplex would leave the
guard blind and someone could start a conflicting per-profile gateway that
double-binds a bot token. Env-forced-on trips the guard even with no
config.yaml key; env-forced-off disables it over a config opt-in.
Tests: full 3-tier precedence in test_config.py (incl. the discriminating
env-overrides-config cases + the empty/whitespace/unrecognized fall-through
trap + resolver tri-state), mutation-verified (flipping precedence fails
exactly the two env-wins tests); guard env cases in test_multiplex_lifecycle.py.
Force-on is safe on a single-profile instance: session keys stay byte-identical
(agent:main) and the _run_agent wrapper installs the per-turn secret scope, so
the fail-closed get_secret() path is satisfied.
Strict charset allowlist (alnum + - _, max 64) on the {name} path param of
the memory-provider config/setup endpoints. Prevents traversal-shaped names
from reaching find_provider_dir(), and setup now 404s when neither a
loadable provider nor a plugin manifest exists, so the command-running path
is only reachable for discoverable plugins. Adds regression tests.
The multiplex machinery already routes an inbound message to a profile via
SessionSource.profile (build_session_key namespacing + the per-turn
config/credential scope in SessionStore._resolve_profile_for_key). But the
relay path never populated it: _event_from_wire rebuilt the SessionSource
field-by-field and dropped any 'profile' the connector sent, so a
Team-Gateway (connector + relay) message could not be routed to a specific
profile the way the /p/<profile>/ HTTP prefix and per-credential polling
adapters already can.
Stamp source.profile from the wire payload in _event_from_wire. This is the
last missing link for NAS-driven per-profile routing over the relay in
multiplex mode; the connector populating the field ships separately
(gateway-gateway contract adds the optional wire field).
Back-compat: absent 'profile' → None → legacy agent:main namespace,
byte-identical to today for every single-profile gateway.
The profile+gateway topology added in #60537 sits entirely behind the
loopback/--insecure auth gate. But a hosted agent (Hermes Cloud) binds
non-loopback with OAuth, so should_require_auth is True, and NAS reads
/api/status over the network (fly-provider.ts getInstanceRuntimeStatus)
with no session token. On that gated path the whole topology block was
omitted, so the Portal could never render the profile list.
Split the topology readout by sensitivity:
- profile NAMES (profiles) + gateway_mode are low-sensitivity product
surface and now ride the always-public status body, surviving the auth
gate so NAS/the Portal can enumerate profiles.
- the per-gateway detail (gateways[], carrying host ports) is deployment
recon and stays gated alongside hermes_home / config_path / env_path /
gateway_pid / gateway_health_url.
The collector now runs unconditionally (still in the executor, off the
event loop). No new fields; only the gate placement changes.
Headless/hosted deploys run the dashboard server without COLORTERM in
the process environment, so chalk inside the PTY-spawned TUI child
downgraded every skin hex color to the xterm 256 palette — the default
skin's bronze banner border (#CD7F32) snapped to palette 173 (#D7875F,
salmon red) and the gold caduceus rendered red/yellow on fresh cloud
instances. Local launches never reproduced it because the operator's
interactive terminal leaks COLORTERM=truecolor into the server env.
xterm.js always renders 24-bit RGB, so the dashboard PTY child should
always advertise truecolor: backfill COLORTERM=truecolor in
_resolve_chat_argv via setdefault (an explicit operator value wins).
Verified with a clean-env PTY probe of the real TUI binary:
no COLORTERM -> 0 truecolor SGRs / 165 palette-256 (salmon 38;5;173);
with the backfill -> 166 truecolor SGRs, exact bronze 38;2;205;127;50.
Session-based channel discovery resurrected historical origins for
platforms with no connected adapter, exposing stale send_message
targets that can no longer deliver. Gate both the enum loop and the
plugin-registry loop on the live adapter set.
Surgical reapply of the channel-directory portion of PR #25959 (branch
was 6.5k commits stale; the text-batching delay changes bundled there
were dropped - separate concern, defaults have since been retuned on
main).
Co-authored-by: Marco-Olivier Lavoie <marcolivier@gmail.com>
Log a one-shot structured warning when Discord denies traffic because
no allowlist/policy is configured, and correct the setup wizard's
inverted warning text. The fail-closed default itself is unchanged.
Fixes#58682.
When a user explicitly configures a platform with its native composite
(e.g. platform_toolsets.discord: [hermes-discord]), the discord and
discord_admin toolsets were silently stripped by _DEFAULT_OFF_TOOLSETS
even though the composite contains those tools. The strip could not tell
an explicit composite opt-in apart from the unconfigured default.
Track whether the platform was explicitly configured and, when it was,
exempt toolsets that are both default-off and platform-restricted to the
current platform from the strip. Only discord/discord_admin are affected
(the sole entries in both _DEFAULT_OFF_TOOLSETS and
_TOOLSET_PLATFORM_RESTRICTIONS). Unconfigured and empty-list platforms
keep the security default-off behaviour.
/api/status (loopback/insecure binds only) now includes:
- profiles: every profile on the host (default + named)
- gateway_mode: none | single | multiple | multiplex
- gateways: one entry per live gateway with the host ports its
port-binding platforms listen on, plus served_profiles when the
default gateway is multiplexing
Ports resolve from each profile's config.yaml (top-level platforms:
wins over gateway.platforms:, matching load_gateway_config precedence)
with adapter defaults as fallback. Topology enumeration runs in an
executor so the profile scan + process-table probes stay off the event
loop, and the whole block is gated behind the same loopback-only split
as hermes_home/gateway_pid so gated binds leak nothing new.
The salvaged test predates the parked-server self-probe
(_PARKED_RETRY_INTERVAL, landed on main after the PR branched): after the
final failed retry, run() parks in a real asyncio.wait that the patched
asyncio.sleep doesn't cover, stalling the test 300s. Signal shutdown once
the retry budget is exhausted so the park exits immediately.
A stdio MCP server (e.g. `npx -y mcp-remote <url>`) is spawned as a direct
child of the Hermes process. Existing teardown (MCPServerTask.shutdown() /
_kill_orphaned_mcp_children()) reaps it correctly on a clean exit, but a
kill -9 / crash / force-quit of the Hermes process skips that path entirely
-- the child (and its own descendants, e.g. mcp-remote's spawned node
process) is orphaned and keeps running. Repeated ungraceful restarts pile up
N orphaned processes racing to hold the same upstream SSE session, producing
errors like 'Invalid request parameters' on legitimate reconnects.
macOS/Linux have no portable equivalent of prctl(PR_SET_PDEATHSIG) at the
Python subprocess level, so this adds a thin supervisor
(tools/mcp_stdio_watchdog.py) that:
- execs the real command as its own child in its own process group
- passes stdin/stdout/stderr through untouched (MCP stdio protocol
talks directly over those streams)
- polls the original spawning PID with the same orphan-detection
algorithm already proven in tui_gateway/slash_worker.py (ppid
comparison + psutil creation-time guard against PID reuse)
- SIGTERM-then-SIGKILL's the child's process group the moment the
original parent is gone
Wired into _run_stdio via a new _wrap_command_with_watchdog() helper,
POSIX-only (matches the existing killpg-based cleanup's platform scope),
fails open (any error resolving pid/create-time falls back to the
unwrapped command) so this can never be the reason a working MCP server
stops starting.
Verified: reproduced the exact orphan scenario standalone (fake parent
process spawns watchdog + fake long-running MCP child, kill -9 the fake
parent, confirm the watchdog reaps the child within its poll window with
zero leaked processes). Updated test_mcp_tool_issue_948.py's resolved-path
assertion to check the watchdog-wrapped command instead of the raw
resolved binary. Full test_mcp_tool.py + test_mcp_stability.py +
test_mcp_tool_issue_948.py suite: 232 passed. Full -k mcp sweep across the
whole test tree: 1003 passed, 2 skipped, 0 failed.
A stdio MCP server that never completes `initialize` (e.g. emits a
non-JSON-RPC frame and then blocks on stdin) leaks a child process plus its
stdio pipes/pidfd on every discovery-retry cycle — unbounded, until the
gateway hits EMFILE and every new open()/spawn fails (#59349).
Root cause (confirmed by instrumenting the live repro, and different from the
issue's own hypothesis): the spawned child IS captured in `new_pids`, so the
report's "new_pids empty at finally" guess is not it. The real cause is that
`session.initialize()` hangs forever on the garbage stream. `connect_timeout`
only bounds the caller's `.result()` wait on the foreground thread — it does
NOT cancel the `_run_stdio` coroutine on the background MCP loop. So the
coroutine is stuck at `await session.initialize()` permanently, its cleanup
`finally` never runs, the child is never reaped, and it stays invisible to the
orphan-reaper (whose `_orphan_stdio_pids` set never gets populated).
Fix: wrap `session.initialize()` in `asyncio.wait_for(..., connect_timeout)`
so a stalled handshake fails instead of hanging. The TimeoutError unwinds
through the SDK context managers (closing the child's stdin -> EOF -> exit)
and lets the existing `finally` reap any straggler. Cross-platform — no
signals/pgid/proc.
Scope: stdio only. The HTTP path has the same `await session.initialize()`
shape but spawns no subprocess (so it can't cause this leak) and already has
httpx transport timeouts.
Verified: the reporter's repro goes from unbounded growth to draining to zero;
added a hermetic regression test (fake transport whose `initialize()` hangs,
asserts the connect is bounded by connect_timeout) that fails on the pre-fix
code and passes on the fix; 566 existing MCP tests pass; ruff clean.
Repro confirmed on macOS (pipe FDs); the Linux-specific pidfd growth in the
report should be equivalent — the reporter offered to validate on Linux.
Closes#59349
When an MCP stdio subprocess fails to connect (token expiry, port
contention, timeout), the run() reconnect loop retries with backoff.
Each retry calls _run_stdio() which spawns a new process pair, but the
previous failed pair was only detected as orphaned (added to
_orphan_stdio_pids) — never actually killed. This caused rapid zombie
accumulation: 5 failed attempts × 2 procs each = 10 orphans competing
for the same port.
Add a _kill_orphaned_mcp_children() call at the top of _run_stdio(),
before the _snapshot_child_pids() baseline, so any orphans from prior
failed attempts are reaped before a new subprocess is spawned.
Fixes#57355
Keep-alive path when ?attach=<token> is present: PTY outlives the socket
via PTY_REGISTRY, reattaches on reconnect. No token = unchanged legacy
pump (_legacy_pump). detach (not close) on disconnect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up on the #56966 salvage:
- is_provider_explicitly_configured(): an env-seeded credential-pool entry
only counts as explicit while its env var still resolves to a usable
secret. A stale auth.json entry left behind after the user deletes the
var no longer keeps the provider in the picker forever (#55790).
- TUI modelPicker + dashboard ModelPickerDialog/api.getModelOptions pass
include_unconfigured=true explicitly, preserving their full-universe
setup-affordance behavior now that the backend defaults to the
configured subset.
- desktop lib/model-options.ts routes explicit_only through the shared
requestModelOptions() helper (added on main after the PR branched).
- regression tests for ambient (gh_cli) pool sources, explicit manual/
device-code sources, and stale vs live env-seeded entries.
* feat(trace): upload sessions to HF Agent Trace Viewer
Salvage trace upload as a smaller CLI-first feature: deterministic Claude Code JSONL export, fail-closed redaction, lazy Hugging Face dependency, and no gateway slash-command wiring.
* chore(trace): drop external porting references from docstrings
Describe the trace-upload design in Hermes' own terms.
* feat(sessions): fold trace upload into 'sessions export --format trace'
Integrates the HF Agent Trace Viewer exporter (PR #36145) onto the
unified export surface instead of a separate 'hermes trace' subcommand:
- --format trace: Claude Code JSONL to stdout/file, or one
<id>.trace.jsonl per session for filtered bulk export; defaults to
the most recent session when no --session-id/filters given.
- --upload pushes to the user's private HF traces dataset (--public to
opt out of private); reads HF_TOKEN with guided setup when missing.
- traces are secret-redacted by default (force mode); --no-redact opts
out after review; redaction failure blocks export (fail closed).
- hermes_cli/trace.py + subcommands/trace.py removed; agent/trace_upload.py
is the single engine. Docs EN + zh-Hans; 4 new CLI tests.
The salvaged SelfHostedBackend made self-hosted servers reachable via
mem0.json / MEM0_HOST, but the setup wizard still offered only Platform
and OSS — exactly the gap users hit (Discord report: 'At memory setup
there's only 2 options'). Adds a third wizard mode:
- interactive picker: Platform / Self-hosted server / Open Source
- non-interactive: hermes memory setup mem0 --mode selfhosted
--host http://... [--api-key ...] [--dry-run]
- host -> mem0.json (behavioral), API key -> .env as MEM0_API_KEY
(secret), optional key for AUTH_DISABLED servers
- best-effort reachability check against the server, non-fatal
- README + memory-providers docs updated with the wizard path
`_save_anthropic_oauth_creds` wrote the Anthropic OAuth token file with
`os.replace(tmp, path)` followed by a post-hoc `chmod(0o600)`. Between the
rename and the chmod the token file existed at the default umask (0o644 on most
hosts) — a window in which another local user could read the access/refresh
tokens.
Write via `utils.atomic_json_write(..., mode=0o600)`, which creates the temp
with mode 0o600 *before* any content is written, fsyncs, atomically replaces,
preserves the existing file's owner, and cleans up its temp on failure. This
matches the `atomic_json_write(mode=0o600)` call already used elsewhere in this
module for the credential-pool write, and #56644's owner preservation.
Tests updated for the new mechanism, plus a check that the write goes through
`atomic_json_write(mode=0o600)` (mutation-verified).
Opt-in discord.approval_mentions (config.yaml, bridged to
DISCORD_APPROVAL_MENTIONS) prepends <@id> mentions for numeric
allowlist entries to exec-approval prompts, with a scoped
AllowedMentions override (users only). Default off - no surprise
pings. Reapplied onto the content-mirror layout from #60245: mentions
prepend to the visible content block and its truncation budget.
Original implementation from PR #39719; commits arrived bot-authored,
re-attributed to the contributor.
Salvage follow-up integrating PR #30481 (@simplast) and PR #57683
(@catbearlove1-lang) into the unified export surface:
- --format html: standalone self-contained HTML transcript (single
session or multi-session with sidebar), works with all shared filters
and --redact; requires a file output path.
- --only user-prompts: prompt-only export (jsonl records or md sections)
via the shared session_export renderer; the separate export-prompts
subcommand from the original PR is subsumed by this flag.
- AUTHOR_MAP entries for both contributors; docs EN + zh-Hans.
Review follow-ups on the salvage:
- get_all() pruned from the ABC and all three backends: mem0_list (its
only caller) was removed by the recall-tuning commit, leaving new,
tested, unreachable code — including SelfHostedBackend's _MAX_TOP_K
over-fetch workaround. Tests for it dropped; fake-class stubs remain
harmlessly. (The #52921 true-total fix lives on in the PR history if
a lister ever returns.)
- The persisted rerank config key was write-only (setup prompted for it,
nothing read it). initialize() now parses it into _rerank_default and
mem0_search uses it when the model doesn't pass rerank explicitly;
per-call args still win. Guard test added.
- Platform-mode setup now warns when MEM0_HOST is set in the environment:
the json host-clear can't help there (_load_config seeds host from the
env var, docs tell users to put it in .env) — the user would silently
keep routing to the self-hosted server.
- SelfHostedBackend: connect-level retries (httpx.HTTPTransport(retries=2))
so a single transient blip doesn't count toward the provider breaker;
transport now injectable and the test helper uses the real __init__
instead of mirroring it via __new__.
- plugin.yaml description no longer leads with reranking (off by default,
platform-only); docs em-dash typo fixed.
Follow-up on the salvaged #55614. The PR added host-based routing to
_create_backend (precedence: oss > host > platform) but two sibling surfaces
didn't mirror it:
- system_prompt_block() checked host before oss, so an oss+host config ran
OSS but told the model it was self-hosted HTTP. Reordered to match routing.
- Platform setup (hermes memory setup mem0 --mode platform) left a stale host
in mem0.json; since host beats platform, the user kept routing to the
self-hosted server. save_config merges (no delete), so clear host to ""
rather than pop() so the merge actually overwrites it.
Adds regression tests for both (mutation-checked).
Salvage of #55614 by @kartik-mem0 (mem0 maintainer). Adds a SelfHostedBackend
that talks to a self-hosted Mem0 Docker server over httpx (X-API-Key auth,
/search + /memories routes), gated behind `host`. Also folds in the mem0
research-team recall tuning that rides with it: rerank defaults to false across
all modes, the mem0_list tool is removed (5->4 tools), search guidance is
de-shouted, and self-hosted get_all reports the true stored total (#52921).
Supersedes the self-hosted portion of #52487 (@liuhao1024, first-submitted).
Closes#52478Fixes#52921
Review findings on the salvaged shim: (a) OpenAI callers may pass stop as
a bare string but Converse's stopSequences requires a list — normalize;
(b) call_llm(stream=True) (MoA aggregator) can reach this client and the
shim silently returned a complete response — keep that behavior (the
streaming consumer's got-final-object path downgrades gracefully) but log
it, and log dropped tool_choice, instead of silently ignoring both.
+2 regression tests.
Follow-up to the salvage of #60217 by @xxxigm.
Review findings on the salvaged #60332 breaker, fixed as follow-ups:
- restore_primary_runtime() now resets the streak (third provider-swap
path; without it a recovered primary was short-circuited before a
single attempt and could never be re-proven healthy except via /model).
- interruptible_api_call (non-streaming) now carries the same breaker
(guard at entry, bump on stale_call_kill, reset on success). Quiet-mode
/ subagent / headless sessions — the profile most like #58962's
unattended 494-failure session — take this path and had the identical
infinite stale-retry class.
- Partial-stream stub return now resets the streak (chunks were received,
provider demonstrably responsive).
- Consolidated the triple-duplicated counter arithmetic into shared
helpers (_stale_streak/_bump_stale_streak/_reset_stale_streak/
_check_stale_giveup) with one canonical comment block; error message
now says 'consecutive stale attempts' (the counter counts kills, not
turns — a single turn can produce several).
4 new tests (restore resets / no-op restore keeps latch / non-streaming
short-circuit / non-streaming success reset).
Follow-up for the salvaged #60332 circuit breaker. The breaker latches:
once the streak trips, interruptible_streaming_api_call raises before any
stream is attempted, so the on-success reset can never run again. The
error text tells the user to switch models and retry — but neither
switch_model() nor try_activate_fallback() cleared the streak, so a
freshly selected healthy provider kept short-circuiting forever (only
/new recovered), and the automatic fallback chain was wedged the same way.
Reset the streak at both swap sites (after a successful rebuild only;
rollback/exhaustion paths keep the latch). 4 tests.
A session wedged against an unresponsive OpenAI-compatible provider can hit the stale-stream detector on every turn and loop forever, burning the full 180s x retries each turn with no response. Issue #58962 reports 494 consecutive failures over 3+ days on a single session.
The streaming retry path already caps retries WITHIN a turn (HERMES_STREAM_RETRIES, default 2) but has no cross-turn cap. Once a session's conversation state makes every turn stale, it retries indefinitely across turns and never notifies the user.
Add a per-session consecutive-stale-stream counter on the agent:
- incremented on every stale-stream kill in the outer poll loop;
- reset to 0 only when a stream actually completes;
- when it reaches HERMES_STREAM_STALE_GIVEUP (default 5), the next turn aborts immediately with a clear, actionable RuntimeError instead of spending 180s x retries again.
This is distinct from the existing stale-stream work (local-provider hard ceiling #44938, backoff/parse-error #60031): those bound a single hung stream, while this bounds repeated cross-turn staleness and surfaces a user-visible error.
Adds tests/run_agent/test_stream_stale_circuit_breaker.py covering the short-circuit, the success-reset, and the increment.
Same 10062 degrade-gracefully pattern as _run_simple_slash: create the
thread anyway, skip the ephemeral followups that need a live
interaction token. Non-expiry defer errors still raise.
- export now shares _add_session_filter_args / build_prune_filters with
prune/archive: AGE grammar (5h/2d/1w/ISO) on --older-than plus the full
filter set (--model, --provider, --min-messages, --min-cost, --branch,
--chat-id, ...) for both JSONL and md/qmd bulk exports; --dry-run works
on JSONL too; removes the one-off list_export_candidates helper.
- new --redact flag runs exported message content and tool output through
force-mode secret redaction (agent.redact) for jsonl, md, and qmd.
- docs EN + zh-Hans updated; new tests for AGE grammar, extended filters,
filtered JSONL, and redaction.
Replaces the parallel export-md subcommand from the salvaged commit with
a --format jsonl|md|qmd flag on the existing export subcommand, so all
session export formats share one surface. Adds AUTHOR_MAP entry.
Salvage follow-up for PR #59542 by @web3blind.
_handle_active_session_busy_message (the busy_session_handler most
platform adapters register) demotes busy_input_mode='interrupt' to
queue semantics for two reasons: active subagents (#30170) and, as of
this week, context compression in flight (#56391) — interrupting while
compression holds the state.db lock races a new turn against the
pre-rotation parent session, and if that new turn also grows past the
compression threshold it starts its own uncancellable compression on
the same stale parent, forking orphaned compression siblings.
_handle_message has its own, independent inline "PRIORITY" busy-path
(reached directly with a live running agent — see the `if _quick_key in
self._running_agents:` guard, exercised end-to-end by the existing
tests/gateway/test_running_agent_session_toggles.py harness). Its own
comment says it mirrors _handle_active_session_busy_message's subagent-
demotion rationale verbatim, and it does demote for active subagents,
but it never checked _session_has_compression_in_flight, so a plain-text
follow-up landing on this path while compression is mid-flight still
called running_agent.interrupt() unconditionally.
Fix: add the same _session_has_compression_in_flight(session_key) check
before the PRIORITY interrupt call, demoting to queue exactly like the
sibling path.
Tests: tests/gateway/test_priority_path_compression_demotion_56391.py
drives _handle_message end-to-end (reusing the test_running_agent_
session_toggles.py harness pattern) with a live running agent and a
mocked compression lock. Mutation-verified: reverting the fix makes the
demotion test fail (interrupt() gets called) against the pre-fix code;
a control test pins the unchanged default-interrupt behavior when no
compression lock is held.
