The busy-session branch of _notification_poller_loop re-queued the
completion event and immediately re-polled it with no sleep, spinning
at full speed (100% CPU, ~1100 futex/s of GIL churn) for as long as
the session stayed running. This starved the dashboard asyncio loop:
/api/status went from 0.14s to 3-6s with 10s timeouts.
Sleep 0.25s outside history_lock before re-polling, mirroring the
0.1s back-off already used for foreign-session events.
Three fixes for the silent post-restart ticker stall:
1. _jobs_lock() bounds its cross-process flock: LOCK_NB polled against a
30s deadline instead of an unbounded LOCK_EX taken while holding the
process-wide RLock. On timeout it logs at ERROR and degrades to
in-process-only locking (the existing fallback path), so a sibling
process wedged while holding .jobs.lock can no longer freeze every
cron function - including the ticker's get_due_jobs() and thus the
heartbeat - forever with zero logging.
2. fire_claim/run_claim freshness checks are bounded on both sides
(0 <= age < ttl): a claim stamped in the future (clock/TZ skew across
a restart) was previously fresh forever, making the job permanently
unfireable and every manual run report 'already being fired'.
3. _execute_job_now distinguishes paused/disabled/missing jobs from a
genuinely held claim instead of mislabeling them all as 'already
being fired'.
Review findings (hermes-pr-review Phase 2, 3-angle):
- _save_auth_store() does real filesystem I/O (mkdir, O_EXCL create, fsync,
atomic replace) and can raise on disk-full/permissions/lock-timeout. The
persist ran bare in the success path, so a persist failure aborted
_resolve_zai_base_url() after detection had already succeeded. Wrap the
persist in try/except: log a warning and still return the detected URL
(worst case: next start re-probes).
- Readability: stage the payload in a local detected_endpoint instead of
writing through the stale pre-lock 'state' dict, which is no longer what
gets persisted.
_save_provider_state() sets auth_store['active_provider'] as a side effect.
The Z.AI endpoint probe runs from credential-pool env seeding for any user
with a Z.AI key in env — persisting the probe cache must not silently make
zai the active provider. Use _store_provider_state(set_active=False).
Follow-up to PR #41201 salvage.
For air-gapped / self-hosted-IdP deploys with NO Nous Portal, let the gateway
obtain its caller-identity bearer from a generic OAuth2 client_credentials grant
against the operator's own IdP (e.g. Microsoft Entra ID) instead of only
resolve_nous_access_token(). The connector's OIDC tenant resolver reads a claim
(default tid) off that token as the tenant.
- gateway/relay: new canonical _resolve_relay_identity_token() — client_credentials
when gateway.idp.token_url (or GATEWAY_RELAY_IDP_* env) is set, else Nous Portal
(unchanged default). Wired into self_provision_relay().
- hermes_cli/gateway_enroll: _resolve_identity_token() delegates to the canonical
resolver so the enroll CLI and the runtime self-provision path share ONE impl.
Config via gateway.idp.{token_url,client_id,client_secret,scope} in config.yaml
(env override GATEWAY_RELAY_IDP_*). No behaviour change when unset.
Tests: tests/gateway/relay/test_identity_token_resolver.py (6 — mode selection,
request shape, config/env precedence, fail-closed). Relay suite 162 pass.
Validated via the cross-repo gateway<->connector live E2E (provision, managed
self-provision, inbound round-trip, /link) against a connector running the OIDC
tenant resolver with zero NAS config.
The salvaged guard used a hand-maintained frozenset of 14 platform names —
several of which (line, wechat, facebook, imessage, googlechat) aren't
actual Hermes Platform values, while real ones (whatsapp_cloud, feishu,
wecom, dingtalk, qqbot, yuanbao, plugin platforms like irc) were missing.
Resolve the source through gateway.config.Platform instead (built-ins +
registered plugin platforms via _missing_), with an explicit exclusion set
for self-owned/local sources. Adds tests for the guard and both reap paths.
Guard _finalize_session's db.end_session() call against gateway-owned
sessions (telegram, bluebubbles, discord, etc.). The TUI is a viewer
for these sessions, not the lifecycle owner. Unconditionally ending
them in state.db creates a Groundhog Day routing loop: the gateway's
#54878 self-heal detects the stale entry, recovers to the parent
session, context compression splits back to the reaped child, and the
cycle repeats on every inbound message — causing complete conversational
context amnesia.
Fixes#60609
Keep #60631's get_running_job_ids() snapshot + _active_cron_job_count()
(import-guarded for minimal test doubles) as the single read path, and
retarget #60612's drain tests at it. Drops the redundant
cron_jobs_in_flight() helper so there is one surface, not two.
Follow-up to the previous commit on #60432. The status-write guard
(_consume_interrupted_flag, checked right before mark_job_run) closes
the false-success bookkeeping gap, but run_one_job delivers its result
BEFORE that check: delivery happens right after run_job() returns,
mark_job_run happens at the very end. A job whose tool subprocess was
killed mid-flight can still produce a plausible-looking final_response
from the truncated output, and that response would reach the user via
_deliver_result before the interrupted flag was ever consulted --
correct status in jobs.json, wrong message already sent.
Adds _is_interrupted(), a non-destructive peek at the same
_interrupted_job_ids set (_consume_interrupted_flag stays as the
consuming, authoritative check right before the status write -- this
needed a peek instead since the flag has to still be visible there).
Checked right after save_job_output, before the deliver_content
decision: if the run looked successful but was flagged interrupted,
force success=False with an explicit interruption message. This
routes delivery through the existing _summarize_cron_failure_for_delivery
path (the same one a real failure already uses) instead of the raw
final_response, so the user gets an honest "this run was interrupted"
instead of a truncated/misleading result.
Testing: 4 new tests in tests/cron/test_shutdown_interrupt.py --
_is_interrupted peek semantics (false/true/does-not-clear, as opposed
to the consuming _consume_interrupted_flag), and the delivery-gate
test itself, which mocks run_job to return a normal-looking success
with a "plausible final response" while the job is pre-marked
interrupted, and asserts _deliver_result receives the failure summary
("This run was interrupted.") instead, with the summarizer's error
argument confirmed to mention the interruption.
Fail-then-pass: reverted cron/scheduler.py only, the 4 new tests fail
(3 on the missing _is_interrupted attribute, 1 -- the delivery-gate
test -- on _summarize_cron_failure_for_delivery never being called,
i.e. the raw response would have gone out); restored, all 16 tests in
the file pass.
Regression: tests/cron/ (683 tests) + test_cron_active_work_drain.py +
test_gateway_shutdown.py + test_shutdown_cache_cleanup.py -- 11
pre-existing failures (Unix file-permission-bit and path-tilde
assertions that don't apply on this Windows dev box), matching the
same set already established as pre-existing in the prior commit's
regression check. Zero new failures.
Continues #60432
Cron jobs run through cron/scheduler.py's own ThreadPoolExecutor via a
standalone AIAgent (run_job/run_one_job), entirely outside
GatewayRunner._running_agents -- the dict _drain_active_agents() and
every other active-work check on that class reads. A gateway shutdown
(/update, /restart, and SIGUSR1 all funnel through the same stop())
could log active_at_start=0 and immediately kill tool subprocesses
while a cron job's terminal command was still running, with no wait
and no indication anything was interrupted.
Real-world impact (from the issue): a scheduled daily briefing cron
job was in flight during /update, its tool subprocess got killed
by the unconditional shutdown cleanup, and the job was never marked
failed -- it simply never completed or delivered, with no error
surfaced anywhere. A repro with a 30-minute `sleep` cron job in flight
during /update reproduced the same pattern: subprocess killed at
+0.22s of drain (active_at_start=0), the job's agent thread continued
in-process and produced a plausible-looking final response from the
truncated tool output, and the scheduler marked the run successful.
Root cause is layered, not a single line:
1. GatewayRunner._drain_active_agents() only waits on _running_agents.
Cron work was invisible to it, so drain returned instantly whenever
the only active work was a cron job.
2. Even with visibility, the shutdown's final tool-subprocess kill
(process_registry.kill_all()) is a global, unconditional sweep with
no per-job targeting -- a long-running cron job that outlives the
drain timeout still gets its subprocess killed.
3. cron/scheduler.py had no way to detect that a job's tool subprocess
was killed out from under it mid-run; the agent thread kept going
and its eventual (often degraded but plausible-looking) response
got reported as a normal successful completion.
Fix, three parts:
- cron/scheduler.py: expose get_running_job_ids() (thread-safe
snapshot of the existing _running_job_ids set, already used to
prevent double-dispatch) so the gateway can read cron's in-flight
state without reaching into private module internals.
- gateway/run.py: GatewayRunner._active_cron_job_count() reads that
snapshot. _drain_active_agents() now waits on
(_running_agents OR active cron jobs), so a cron-only workload gets
the same bounded wait chat sessions already get instead of an
instant active_at_start=0. Shutdown drain logging gains
cron_active_at_start/cron_active_now fields alongside the existing
ones (unchanged, for compat).
- cron/scheduler.py: mark_running_jobs_interrupted(reason), called by
gateway/run.py's _kill_tool_subprocesses() right after
process_registry.kill_all(), marks every job still in
_running_job_ids at that instant as failed/interrupted via the
existing mark_job_run() -- and records the job IDs in
_interrupted_job_ids BEFORE writing, so run_one_job()'s own
eventual completion for the same run (racing in its own thread)
checks that flag and skips its normal write instead of clobbering
the interrupted status with a false "ok" produced from the
now-truncated tool output. This does not attempt to correlate a
killed PID to a specific job ID (process_registry tracks PIDs, not
job IDs) -- any job still dispatched at the moment of a forced kill
is treated as interrupted, matching the existing coarser precedent
set by _interrupt_running_agents(), which interrupts every entry in
_running_agents on a drain timeout without per-agent correlation
either.
Deliberately out of scope (flagged in the issue as a separate,
lower-priority concern): startup-time reconciliation of cron runs that
started but never reached a terminal status.
Testing:
- tests/cron/test_shutdown_interrupt.py (12 tests): get_running_job_ids
snapshot semantics, mark_running_jobs_interrupted marking/no-op/
partial-failure behavior, and -- the core race guard -- run_one_job
skipping its own last_status write (both the success path and the
exception path) when the shutdown path already marked the run
interrupted, with a control test proving ordinary un-interrupted
completions are unaffected.
- tests/gateway/test_cron_active_work_drain.py (9 tests):
_active_cron_job_count reading cron state and failing closed (0) if
the cron module is unavailable; _drain_active_agents waiting for an
in-flight cron job the same way it waits for chat sessions, timing
out if the job outruns the window, and leaving existing chat-session
drain behavior unchanged; a full runner.stop() integration test
(drain-timeout path) proving mark_running_jobs_interrupted actually
fires with the right job ID when a tool subprocess is force-killed,
plus a no-op control when nothing cron-related is in flight.
- tests/gateway/test_shutdown_cache_cleanup.py: added
_active_cron_job_count() to that file's hand-rolled _FakeGateway test
double, which stop() now calls -- without it those 8 pre-existing
tests AttributeError (caught by fail-then-pass below, not a
production bug).
Fail-then-pass: reverted gateway/run.py + cron/scheduler.py, all 21
new tests fail (fixture/attribute errors -- the feature doesn't exist
yet); restored, all 21 pass.
Regression check: ran the full plausibly-affected surface --
tests/gateway/{test_gateway_shutdown,test_restart_drain,
test_restart_notification,test_restart_redelivery_dedup,
test_restart_resume_pending,test_restart_service_detection,
test_shutdown_cache_cleanup,test_stuck_loop,test_clean_shutdown_marker,
test_external_drain_control,test_session_state_cleanup,
test_update_command,test_update_streaming}.py plus tests/cron/ (944
tests) -- against a clean upstream/main checkout and against this
branch. Diffed the two FAILED lists: identical, 20 pre-existing
failures on both sides (Windows-locale/cp1252 file-encoding issues and
Unix-permission-bit assertions that don't apply on this Windows dev
box), zero new failures, zero fixed-by-accident. The 8
test_shutdown_cache_cleanup.py failures found mid-development were
from the _FakeGateway gap above, fixed in the same commit and
confirmed clean on the final rerun (diff against baseline: exit 0).
Fixes#60432
/update and other shutdown paths only waited on gateway session agents,
so active cron tool work was killed immediately in final-cleanup while
the scheduler could still mark the job successful (#60432).
safe_load() raises ComposerError on multi-document streams (k8s manifests)
and ConstructorError on application-defined tags (CloudFormation !Sub,
Ansible !vault) — both valid YAML syntax. Now that the linter's verdict is
a fail-closed write gate, those false positives would refuse legitimate
writes outright. Switch to yaml.parse() (scanner+parser only), which still
catches real syntax failures.
write_file() previously called _atomic_write() first and only ran the
JSON/YAML/TOML/Python syntax check afterward as an informational lint
delta -- a parse failure never set the top-level `error` key, so a
corrupt structured-data write still landed on disk (and file_tools.py's
files_modified gating, which keys off `error`, silently reported it as
a successful modification).
Move the in-process syntax check for JSON/YAML/TOML ahead of
_atomic_write() and refuse the write outright on a parse failure: no
temp file, no rename, nothing touches disk, and the result carries a
top-level `error` so callers correctly see it as unmodified.
Deliberately scoped to _FAIL_CLOSED_INPROC_EXTS (JSON/YAML/TOML), not
all of LINTERS_INPROC -- .py is excluded because this codebase's own
test fixtures (TestPatchReplacePostWriteVerification et al.) write
arbitrary non-Python text through *.py paths purely to exercise
write-mechanics; a hard block there broke 3 previously-passing tests
during development. Python keeps its pre-existing non-blocking
lint-delta report.
Adds tests/tools/test_write_file_syntax_gate.py: invalid JSON/YAML/YML/
TOML refused with nothing written (new file) and nothing modified
(existing file); valid JSON/YAML still written byte-for-byte; a
non-linted extension with garbage content is unaffected; invalid Python
is confirmed NOT hard-refused (still just reported).
* feat(install): warn pip/Homebrew installs are unsupported (CLI, TUI, desktop)
pip and Homebrew are now Unsupported install methods per
website/docs/getting-started/platform-support.md. Surface a
warn-don't-block deprecation notice everywhere the install method is
already shown, pointing at the platform-support docs and noting these
installs will not receive further updates. NixOS (Tier 2) is untouched.
- hermes_cli/config.py: shared is_unsupported_install_method() /
format_unsupported_install_warning() helpers so the wording and docs
link stay consistent across every surface.
- hermes_cli/banner.py: generalize the existing pip-only banner
warning to also cover Homebrew.
- hermes_cli/main.py: hermes update and hermes update --check print
the warning before proceeding (still update; warn, don't block).
- tui_gateway/server.py: session.info gains install_warning.
- ui-tui: SessionPanel renders install_warning alongside the existing
'N commits behind' notice.
- apps/desktop: SessionRuntimeInfo/GatewayEventPayload gain
install_warning; applyRuntimeInfo + the live session.info event fire
a snoozable warning toast via a new reportInstallMethodWarning(),
mirroring the existing backend-contract-skew toast pattern. i18n
strings added for en/zh/zh-hant/ja.
- Tests: updated pip banner assertions for the new wording, added a
Homebrew banner test, and two tui_gateway session_info tests
(install_warning present for pip, absent for git).
* fix(nix): make `hermes` in developement environment actually work
install modules as editable overlay with uv
* feat: print install method when running --version
* fix: correct detect install method when running from a subtree
The April 2026 pin to WhiskeySockets/Baileys#01047deb existed only to
pick up the abprops bad-request fix (Baileys PR #2473) before it was
released. That fix shipped in v7.0.0-rc11 (May 2026); our pinned commit
is now 48 commits behind rc13.
The git pin forced npm to clone the repo and compile Baileys from
TypeScript source on every fresh install (~3 min), which blew past the
dashboard pairing flow's timeout. Registry install takes ~3s.
Validation: all 9 bridge.js imports present in rc13, bridge.native.test.mjs
passes (13/13), live bridge boot renders pairing QR against real WA servers.
The chat PTY launch path landed on main after PR #50558 and still called
_session_latest_descendant() with the old one-arg signature. Open the
requested profile's state DB (matching the REST endpoint) so profile-scoped
resume resolves descendants in the right database.
The connector now depends on the single multiplexed gateway for per-profile
relay routing, so hosted deployments need to FORCE multiplexing on regardless
of the image's config.yaml. gateway.multiplex_profiles was config.yaml-only,
which a user could leave unset or flip off.
Add GATEWAY_MULTIPLEX_PROFILES as a standard operator override on top of the
existing config key — the same 'config.yaml is canonical, env is the operator
override' pattern the Telegram/Signal require_mention bridges use:
env (recognized token) > config.yaml (top-level or nested gateway.*) > False
- gateway/config.py: _env_multiplex_profiles_override() resolves the env var
tri-state — recognized truthy/falsy token → bool; unset/blank/unrecognized
→ None (fall through to config). Blank is deliberately None, not False, so a
provisioned-but-unpopulated Fly secret ('') can't shadow a config.yaml opt-in
(the empty-secret trap). Wired into GatewayConfig.from_dict so every consumer
(run.py, session.py via self.config) sees the resolved value.
- hermes_cli/gateway.py: the named-profile-start guard
(_guard_named_profile_under_multiplexer) reads config.yaml directly, so it
gets the SAME env precedence — otherwise env-forced multiplex would leave the
guard blind and someone could start a conflicting per-profile gateway that
double-binds a bot token. Env-forced-on trips the guard even with no
config.yaml key; env-forced-off disables it over a config opt-in.
Tests: full 3-tier precedence in test_config.py (incl. the discriminating
env-overrides-config cases + the empty/whitespace/unrecognized fall-through
trap + resolver tri-state), mutation-verified (flipping precedence fails
exactly the two env-wins tests); guard env cases in test_multiplex_lifecycle.py.
Force-on is safe on a single-profile instance: session keys stay byte-identical
(agent:main) and the _run_agent wrapper installs the per-turn secret scope, so
the fail-closed get_secret() path is satisfied.
Strict charset allowlist (alnum + - _, max 64) on the {name} path param of
the memory-provider config/setup endpoints. Prevents traversal-shaped names
from reaching find_provider_dir(), and setup now 404s when neither a
loadable provider nor a plugin manifest exists, so the command-running path
is only reachable for discoverable plugins. Adds regression tests.
The multiplex machinery already routes an inbound message to a profile via
SessionSource.profile (build_session_key namespacing + the per-turn
config/credential scope in SessionStore._resolve_profile_for_key). But the
relay path never populated it: _event_from_wire rebuilt the SessionSource
field-by-field and dropped any 'profile' the connector sent, so a
Team-Gateway (connector + relay) message could not be routed to a specific
profile the way the /p/<profile>/ HTTP prefix and per-credential polling
adapters already can.
Stamp source.profile from the wire payload in _event_from_wire. This is the
last missing link for NAS-driven per-profile routing over the relay in
multiplex mode; the connector populating the field ships separately
(gateway-gateway contract adds the optional wire field).
Back-compat: absent 'profile' → None → legacy agent:main namespace,
byte-identical to today for every single-profile gateway.
The profile+gateway topology added in #60537 sits entirely behind the
loopback/--insecure auth gate. But a hosted agent (Hermes Cloud) binds
non-loopback with OAuth, so should_require_auth is True, and NAS reads
/api/status over the network (fly-provider.ts getInstanceRuntimeStatus)
with no session token. On that gated path the whole topology block was
omitted, so the Portal could never render the profile list.
Split the topology readout by sensitivity:
- profile NAMES (profiles) + gateway_mode are low-sensitivity product
surface and now ride the always-public status body, surviving the auth
gate so NAS/the Portal can enumerate profiles.
- the per-gateway detail (gateways[], carrying host ports) is deployment
recon and stays gated alongside hermes_home / config_path / env_path /
gateway_pid / gateway_health_url.
The collector now runs unconditionally (still in the executor, off the
event loop). No new fields; only the gate placement changes.
Headless/hosted deploys run the dashboard server without COLORTERM in
the process environment, so chalk inside the PTY-spawned TUI child
downgraded every skin hex color to the xterm 256 palette — the default
skin's bronze banner border (#CD7F32) snapped to palette 173 (#D7875F,
salmon red) and the gold caduceus rendered red/yellow on fresh cloud
instances. Local launches never reproduced it because the operator's
interactive terminal leaks COLORTERM=truecolor into the server env.
xterm.js always renders 24-bit RGB, so the dashboard PTY child should
always advertise truecolor: backfill COLORTERM=truecolor in
_resolve_chat_argv via setdefault (an explicit operator value wins).
Verified with a clean-env PTY probe of the real TUI binary:
no COLORTERM -> 0 truecolor SGRs / 165 palette-256 (salmon 38;5;173);
with the backfill -> 166 truecolor SGRs, exact bronze 38;2;205;127;50.
Session-based channel discovery resurrected historical origins for
platforms with no connected adapter, exposing stale send_message
targets that can no longer deliver. Gate both the enum loop and the
plugin-registry loop on the live adapter set.
Surgical reapply of the channel-directory portion of PR #25959 (branch
was 6.5k commits stale; the text-batching delay changes bundled there
were dropped - separate concern, defaults have since been retuned on
main).
Co-authored-by: Marco-Olivier Lavoie <marcolivier@gmail.com>
Docs portion of PR #57067: 'bot connects but never replies' section
pointing at the gateway.log warning and the allowlist/policy knobs.
Co-authored-by: ooovenenoso <120500656+ooovenenoso@users.noreply.github.com>
Log a one-shot structured warning when Discord denies traffic because
no allowlist/policy is configured, and correct the setup wizard's
inverted warning text. The fail-closed default itself is unchanged.
Fixes#58682.
Restructures the five parallel export sections into a single 'Export
Sessions' section: a format table (jsonl/md/qmd/html/trace + --only
user-prompts), one shared-filters paragraph covering all formats, and
per-format subsections nested beneath. EN + zh-Hans.
When a user explicitly configures a platform with its native composite
(e.g. platform_toolsets.discord: [hermes-discord]), the discord and
discord_admin toolsets were silently stripped by _DEFAULT_OFF_TOOLSETS
even though the composite contains those tools. The strip could not tell
an explicit composite opt-in apart from the unconfigured default.
Track whether the platform was explicitly configured and, when it was,
exempt toolsets that are both default-off and platform-restricted to the
current platform from the strip. Only discord/discord_admin are affected
(the sole entries in both _DEFAULT_OFF_TOOLSETS and
_TOOLSET_PLATFORM_RESTRICTIONS). Unconfigured and empty-list platforms
keep the security default-off behaviour.
/api/status (loopback/insecure binds only) now includes:
- profiles: every profile on the host (default + named)
- gateway_mode: none | single | multiple | multiplex
- gateways: one entry per live gateway with the host ports its
port-binding platforms listen on, plus served_profiles when the
default gateway is multiplexing
Ports resolve from each profile's config.yaml (top-level platforms:
wins over gateway.platforms:, matching load_gateway_config precedence)
with adapter defaults as fallback. Topology enumeration runs in an
executor so the profile scan + process-table probes stay off the event
loop, and the whole block is gated behind the same loopback-only split
as hermes_home/gateway_pid so gated binds leak nothing new.
signal.SIGKILL / os.killpg don't exist on Windows. The watchdog is only
spawned on POSIX (wrap site gates on os.name), but guard via getattr with
a plain terminate/kill fallback so an accidental Windows import can't
AttributeError.
The salvaged test predates the parked-server self-probe
(_PARKED_RETRY_INTERVAL, landed on main after the PR branched): after the
final failed retry, run() parks in a real asyncio.wait that the patched
asyncio.sleep doesn't cover, stalling the test 300s. Signal shutdown once
the retry budget is exhausted so the park exits immediately.
Two fixes on top of the salvaged parent-death watchdog:
- Apply the watchdog wrap AFTER the OSV malware preflight so the check
inspects the real npx/uvx package instead of the python wrapper
(the wrap previously made the preflight a silent no-op for every
stdio server).
- The real server runs in its own process group under the watchdog, so
the graceful-shutdown killpg no longer reached it; the watchdog now
forwards SIGTERM/SIGINT to the child's group, keeping wedged servers
killable on clean shutdown.
A stdio MCP server (e.g. `npx -y mcp-remote <url>`) is spawned as a direct
child of the Hermes process. Existing teardown (MCPServerTask.shutdown() /
_kill_orphaned_mcp_children()) reaps it correctly on a clean exit, but a
kill -9 / crash / force-quit of the Hermes process skips that path entirely
-- the child (and its own descendants, e.g. mcp-remote's spawned node
process) is orphaned and keeps running. Repeated ungraceful restarts pile up
N orphaned processes racing to hold the same upstream SSE session, producing
errors like 'Invalid request parameters' on legitimate reconnects.
macOS/Linux have no portable equivalent of prctl(PR_SET_PDEATHSIG) at the
Python subprocess level, so this adds a thin supervisor
(tools/mcp_stdio_watchdog.py) that:
- execs the real command as its own child in its own process group
- passes stdin/stdout/stderr through untouched (MCP stdio protocol
talks directly over those streams)
- polls the original spawning PID with the same orphan-detection
algorithm already proven in tui_gateway/slash_worker.py (ppid
comparison + psutil creation-time guard against PID reuse)
- SIGTERM-then-SIGKILL's the child's process group the moment the
original parent is gone
Wired into _run_stdio via a new _wrap_command_with_watchdog() helper,
POSIX-only (matches the existing killpg-based cleanup's platform scope),
fails open (any error resolving pid/create-time falls back to the
unwrapped command) so this can never be the reason a working MCP server
stops starting.
Verified: reproduced the exact orphan scenario standalone (fake parent
process spawns watchdog + fake long-running MCP child, kill -9 the fake
parent, confirm the watchdog reaps the child within its poll window with
zero leaked processes). Updated test_mcp_tool_issue_948.py's resolved-path
assertion to check the watchdog-wrapped command instead of the raw
resolved binary. Full test_mcp_tool.py + test_mcp_stability.py +
test_mcp_tool_issue_948.py suite: 232 passed. Full -k mcp sweep across the
whole test tree: 1003 passed, 2 skipped, 0 failed.
Sibling sites of the same bug class as the salvaged stdio fix:
- SSE, streamable-HTTP (new + deprecated API) initialize() calls are now
bounded by the same connect_timeout, so an endpoint that accepts the
connection but never answers the handshake cannot park the run() task
forever.
- start() now cancels its ensure_future'd run() task when the caller's
connect timeout cancels start() itself — the orphaned-task leak was
the root mechanism behind #59349, and this closes the class for any
future pre-ready hang.
A stdio MCP server that never completes `initialize` (e.g. emits a
non-JSON-RPC frame and then blocks on stdin) leaks a child process plus its
stdio pipes/pidfd on every discovery-retry cycle — unbounded, until the
gateway hits EMFILE and every new open()/spawn fails (#59349).
Root cause (confirmed by instrumenting the live repro, and different from the
issue's own hypothesis): the spawned child IS captured in `new_pids`, so the
report's "new_pids empty at finally" guess is not it. The real cause is that
`session.initialize()` hangs forever on the garbage stream. `connect_timeout`
only bounds the caller's `.result()` wait on the foreground thread — it does
NOT cancel the `_run_stdio` coroutine on the background MCP loop. So the
coroutine is stuck at `await session.initialize()` permanently, its cleanup
`finally` never runs, the child is never reaped, and it stays invisible to the
orphan-reaper (whose `_orphan_stdio_pids` set never gets populated).
Fix: wrap `session.initialize()` in `asyncio.wait_for(..., connect_timeout)`
so a stalled handshake fails instead of hanging. The TimeoutError unwinds
through the SDK context managers (closing the child's stdin -> EOF -> exit)
and lets the existing `finally` reap any straggler. Cross-platform — no
signals/pgid/proc.
Scope: stdio only. The HTTP path has the same `await session.initialize()`
shape but spawns no subprocess (so it can't cause this leak) and already has
httpx transport timeouts.
Verified: the reporter's repro goes from unbounded growth to draining to zero;
added a hermetic regression test (fake transport whose `initialize()` hangs,
asserts the connect is bounded by connect_timeout) that fails on the pre-fix
code and passes on the fix; 566 existing MCP tests pass; ruff clean.
Repro confirmed on macOS (pipe FDs); the Linux-specific pidfd growth in the
report should be equivalent — the reporter offered to validate on Linux.
Closes#59349
Merge the two cherry-picked reap call sites into one unscoped sweep at
the top of _run_stdio (the unscoped sweep is a superset of the
per-server one), and run it via asyncio.to_thread so the 2s
SIGTERM->SIGKILL escalation cannot stall the shared MCP event loop.
