Surgical reapply of PR #37523 onto current main (the original branch
predates the SecretSource registry refactor). _build_safe_env() now
forwards env vars tagged in env_loader._SECRET_SOURCES — widened from
Bitwarden-only to any registered secret source (Bitwarden, 1Password,
plugin backends), since the provenance map is source-agnostic.
Explicit server env: config still wins; untagged secrets stay filtered.
Fixes#37499.
Builds on diffen77's #66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.
Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
Builds on trevorgordon981's #50589 (cherry-picked as the previous commit).
The #50394 cooldown reset only ran inside the async _shutdown coroutine,
which is skipped on the empty-_servers fast path — the most common state
when a server failed to connect (failed servers are never recorded in
_servers). It was also skipped when the MCP loop wasn't running.
Clear _server_connect_retry_after/_server_connect_failures on the fast
path and in a final unconditional sweep so a full shutdown/restart always
re-attempts every configured server immediately. Adds regression tests
for both paths.
Log-storm hygiene for the reconnect machinery (#65673, #66092):
- State transitions carry exactly one WARNING each:
connected → degraded (keepalive failure), degraded → parked
(budget exhausted / rapid-drop park / permanent error),
parked → connected (revival proven healthy, via
_mark_session_proven).
- Per-attempt retry logs (initial-connect attempts, connection-lost
reconnect attempts, per-cycle rebuild notices, park-wake probes)
demoted to DEBUG.
- Backoff sleeps get +/-20% uniform jitter (_jittered) so a herd of
servers that lost the same backend doesn't retry — and log — in
lockstep.
- Add module-level _unwrap_exception_group (ported from
hermes_cli/mcp_config.py, adapted): handles nested groups, prefers
non-cancellation leaves over the CancelledErrors anyio sprays across
sibling tasks, and re-raises KeyboardInterrupt/SystemExit leaves.
- Add _classify_mcp_failure(exc) -> 'permanent'|'transient'.
Permanent: auth 401/403, NonMcpEndpointError, InvalidMcpUrlError,
FileNotFoundError/ENOENT on the stdio command. Transient: everything
else (network/EOF/ClosedResource/TaskGroup drops). Permanent failures
park immediately without burning the retry ladder — every retry
against a missing binary or a revoked credential hits the same wall.
- Every log site in run() and the keepalive failure log now logs the
unwrapped root cause as 'TypeName: message', so a dead stdio pipe
says 'BrokenPipeError' instead of the opaque
'unhandled errors in a TaskGroup (1 sub-exception)' (and empty
str(exc) dead-pipe errors are no longer blank).
Harden the immediate-reconnect path from #66271:
- _reconnect_or_reraise_group re-raises when the transport TaskGroup
carries a KeyboardInterrupt / SystemExit leaf — fatal signals must
propagate to the interpreter, never be converted into a reconnect.
- Rapid-drop budget: a completed handshake alone no longer clears
_reconnect_retries/backoff on clean transport return. A session is
UNPROVEN until it demonstrates real health — survived >=1 full
keepalive interval (keepalive success path) or served >=1 successful
tool call (_mark_session_proven). Unproven clean returns are charged
against _MAX_RECONNECT_RETRIES, so a flapping transport that
handshakes fine and drops moments later still reaches the park
instead of respawning forever (#62212: 6212 spawns in 63h).
Proven sessions keep the #57604 behaviour: budget clears, transient
blips over a long-lived session never accumulate toward parking.
Streamable-HTTP / SSE MCP transports run their stream pump inside an anyio
TaskGroup. A transient stream drop (idle timeout, brief backend blip,
server-side TCP close) surfaces as a BaseExceptionGroup escaping the
transport context manager. It reached run()'s error path, which applied
exponential backoff (1s..16s) and eventually parked the server for 300s and
deregistered its tools — turning a sub-second glitch into a multi-minute
tool outage even though the POST path was still healthy.
_run_http now wraps all three transport branches (SSE, new + deprecated
Streamable-HTTP) and routes a BaseExceptionGroup through a new
_reconnect_or_reraise_group() helper that returns 'reconnect' (immediate
rebuild, no backoff/park/deregister). It re-raises instead of masking when
shutdown is in progress, when the group carries a real CancelledError
(cancellation must propagate, cf. #9930), or when no live session was
established this attempt (a connect/handshake failure that should fall
through to run()'s backoff rather than hot-loop).
Fixes#66092
Co-authored-by: 雨哥 <hanyu1212@users.noreply.github.com>
Use the direct POSIX parent relationship instead of process creation time and pid_exists checks. Remove the dead create-time argument chain while preserving process-group cleanup and signal forwarding.\n\nRefs #62505
Per the MCP spec the cursor is an opaque string; anything else
(including MagicMock auto-attributes in tests) means no more pages.
Fixes test_mcp_tool_session_expired mock-session runaway.
Port from anomalyco/opencode#35439/#35500: preserve full MCP catalogs
across paginated tools/list responses.
The MCP spec allows servers to paginate tools/list, resources/list, and
prompts/list via an opaque nextCursor token. The Python SDK's
ClientSession.list_* methods fetch exactly one page per call, and hermes
never passed the cursor back — on a paginated server every tool,
resource, and prompt past page 1 was silently invisible to the agent.
Adds _paginate_full_list() (cursor-draining helper with a 50-page
runaway cap and spec-correct opaque-string cursor validation) and
applies it at all three discovery sites: _discover_tools(), the
tools/list_changed refresh handler, and the list_resources/list_prompts
utility handlers. The keepalive probe intentionally keeps its
single-page call (liveness only).
E2E: real stdio MCP server serving 3 pages (2/1/1 tools) — old code
discovered 2 tools, new code discovers all 4.
Follow-ups on top of #64061's salvage:
- ResourceLink markers now point at mcp__<server>__read_resource (the
actual registered tool name via mcp_prefixed_tool_name) instead of a
nonexistent <server>_read_resource the agent could hallucinate-call.
- The isError path now surfaces EmbeddedResource .resource.text blocks
instead of dropping them, so error payloads carried in resources no
longer collapse to a bare 'MCP tool returned an error'. (Same-class
fix flagged in #64061 and independently addressed in #63576 by
@alauer.)
- 3 new error-path tests + updated ResourceLink wire-name assertion.
MCP tool results with non-image binary resources (PDFs, archives, office
docs) were silently dropped: the success path only handled TextContent and
ImageContent, so a PDF-returning MCP tool appeared to return metadata only.
- EmbeddedResource blob contents are decoded (50MB cap), materialized into
the Hermes document cache via cache_document_from_bytes (sanitized
filename, traversal-safe), and surfaced as a local-path marker the agent
can read with file/terminal tools.
- EmbeddedResource text contents are inlined directly.
- ResourceLink blocks preserve the URI and point the agent at the server's
read_resource tool; no arbitrary network fetch outside the MCP session.
- AudioContent blocks are cached via cache_audio_from_bytes as MEDIA: tags.
- read_resource blob contents are materialized the same way instead of
returning '[binary data, N bytes]'.
- Unsupported blocks are logged instead of silently discarded.
- Existing ImageContent MEDIA: behavior unchanged.
Reported by an enterprise customer; reproduced against an HTTP MCP server
returning application/pdf resources.
Two fixes on top of the salvaged parent-death watchdog:
- Apply the watchdog wrap AFTER the OSV malware preflight so the check
inspects the real npx/uvx package instead of the python wrapper
(the wrap previously made the preflight a silent no-op for every
stdio server).
- The real server runs in its own process group under the watchdog, so
the graceful-shutdown killpg no longer reached it; the watchdog now
forwards SIGTERM/SIGINT to the child's group, keeping wedged servers
killable on clean shutdown.
A stdio MCP server (e.g. `npx -y mcp-remote <url>`) is spawned as a direct
child of the Hermes process. Existing teardown (MCPServerTask.shutdown() /
_kill_orphaned_mcp_children()) reaps it correctly on a clean exit, but a
kill -9 / crash / force-quit of the Hermes process skips that path entirely
-- the child (and its own descendants, e.g. mcp-remote's spawned node
process) is orphaned and keeps running. Repeated ungraceful restarts pile up
N orphaned processes racing to hold the same upstream SSE session, producing
errors like 'Invalid request parameters' on legitimate reconnects.
macOS/Linux have no portable equivalent of prctl(PR_SET_PDEATHSIG) at the
Python subprocess level, so this adds a thin supervisor
(tools/mcp_stdio_watchdog.py) that:
- execs the real command as its own child in its own process group
- passes stdin/stdout/stderr through untouched (MCP stdio protocol
talks directly over those streams)
- polls the original spawning PID with the same orphan-detection
algorithm already proven in tui_gateway/slash_worker.py (ppid
comparison + psutil creation-time guard against PID reuse)
- SIGTERM-then-SIGKILL's the child's process group the moment the
original parent is gone
Wired into _run_stdio via a new _wrap_command_with_watchdog() helper,
POSIX-only (matches the existing killpg-based cleanup's platform scope),
fails open (any error resolving pid/create-time falls back to the
unwrapped command) so this can never be the reason a working MCP server
stops starting.
Verified: reproduced the exact orphan scenario standalone (fake parent
process spawns watchdog + fake long-running MCP child, kill -9 the fake
parent, confirm the watchdog reaps the child within its poll window with
zero leaked processes). Updated test_mcp_tool_issue_948.py's resolved-path
assertion to check the watchdog-wrapped command instead of the raw
resolved binary. Full test_mcp_tool.py + test_mcp_stability.py +
test_mcp_tool_issue_948.py suite: 232 passed. Full -k mcp sweep across the
whole test tree: 1003 passed, 2 skipped, 0 failed.
Sibling sites of the same bug class as the salvaged stdio fix:
- SSE, streamable-HTTP (new + deprecated API) initialize() calls are now
bounded by the same connect_timeout, so an endpoint that accepts the
connection but never answers the handshake cannot park the run() task
forever.
- start() now cancels its ensure_future'd run() task when the caller's
connect timeout cancels start() itself — the orphaned-task leak was
the root mechanism behind #59349, and this closes the class for any
future pre-ready hang.
A stdio MCP server that never completes `initialize` (e.g. emits a
non-JSON-RPC frame and then blocks on stdin) leaks a child process plus its
stdio pipes/pidfd on every discovery-retry cycle — unbounded, until the
gateway hits EMFILE and every new open()/spawn fails (#59349).
Root cause (confirmed by instrumenting the live repro, and different from the
issue's own hypothesis): the spawned child IS captured in `new_pids`, so the
report's "new_pids empty at finally" guess is not it. The real cause is that
`session.initialize()` hangs forever on the garbage stream. `connect_timeout`
only bounds the caller's `.result()` wait on the foreground thread — it does
NOT cancel the `_run_stdio` coroutine on the background MCP loop. So the
coroutine is stuck at `await session.initialize()` permanently, its cleanup
`finally` never runs, the child is never reaped, and it stays invisible to the
orphan-reaper (whose `_orphan_stdio_pids` set never gets populated).
Fix: wrap `session.initialize()` in `asyncio.wait_for(..., connect_timeout)`
so a stalled handshake fails instead of hanging. The TimeoutError unwinds
through the SDK context managers (closing the child's stdin -> EOF -> exit)
and lets the existing `finally` reap any straggler. Cross-platform — no
signals/pgid/proc.
Scope: stdio only. The HTTP path has the same `await session.initialize()`
shape but spawns no subprocess (so it can't cause this leak) and already has
httpx transport timeouts.
Verified: the reporter's repro goes from unbounded growth to draining to zero;
added a hermetic regression test (fake transport whose `initialize()` hangs,
asserts the connect is bounded by connect_timeout) that fails on the pre-fix
code and passes on the fix; 566 existing MCP tests pass; ruff clean.
Repro confirmed on macOS (pipe FDs); the Linux-specific pidfd growth in the
report should be equivalent — the reporter offered to validate on Linux.
Closes#59349
Merge the two cherry-picked reap call sites into one unscoped sweep at
the top of _run_stdio (the unscoped sweep is a superset of the
per-server one), and run it via asyncio.to_thread so the 2s
SIGTERM->SIGKILL escalation cannot stall the shared MCP event loop.
When an MCP stdio subprocess fails to connect (token expiry, port
contention, timeout), the run() reconnect loop retries with backoff.
Each retry calls _run_stdio() which spawns a new process pair, but the
previous failed pair was only detected as orphaned (added to
_orphan_stdio_pids) — never actually killed. This caused rapid zombie
accumulation: 5 failed attempts × 2 procs each = 10 orphans competing
for the same port.
Add a _kill_orphaned_mcp_children() call at the top of _run_stdio(),
before the _snapshot_child_pids() baseline, so any orphans from prior
failed attempts are reaped before a new subprocess is spawned.
Fixes#57355
register_mcp_servers now nudges cached entries whose session is None
via _signal_reconnect, so a new agent session recovers a parked server
immediately instead of waiting up to _PARKED_RETRY_INTERVAL for the
next self-probe (#50170). Gate-check idea credit: @izumi0uu (#50184),
@LeonSGP43 (#37772), @Tranquil-Flow (#37899).
_wait_for_server_session_ready used a time.monotonic deadline; the
circuit-breaker tests freeze monotonic, turning the loop into an
infinite spin (300s SIGKILL in CI-parity runs). Bound by iteration
count instead.
Some MCP servers (e.g. Spring Boot apps with a React SPA) serve their
frontend on any unmatched GET route. The MCP endpoint works perfectly
via POST (JSON-RPC), but a GET to /mcp falls through to the SPA
controller and returns text/html. Hermes's preflight content-type probe
sees HTML instead of application/json or text/event-stream and refuses
to connect.
This adds a per-server config option that
bypasses the content-type probe, letting the SDK connect directly via
POST where it works fine.
```yaml
mcp_servers:
stirling-pdf:
url: http://localhost:8090/mcp
headers:
X-API-KEY: <key>
skip_preflight: true
```
Related: #52460 (OAuth redirect preflight), #51600 (skip probe on mcp add),
#40366 (skip probe on reconnect — already merged).
Some MCP servers (e.g. DocuSeal) serve their web UI on HEAD/GET but
speak Streamable HTTP only via POST. The preflight probe now tries a
lightweight JSON-RPC `initialize` POST before rejecting endpoints
whose HEAD/GET returns a non-MCP content type (e.g. `text/html`).
If the POST returns `application/json` or `text/event-stream` with a
2xx status, the endpoint is accepted. Otherwise the original rejection
behaviour is preserved.
Adds 5 new test cases covering the POST probe path:
- POST rescues HTML HEAD with JSON response
- POST rescues HTML HEAD with event-stream response
- POST still rejects when it also returns HTML
- POST still rejects on non-2xx status
- POST not attempted when HEAD already returns valid MCP content type
Parking deregisters the server's tools, which removes the only paths
that could ever set _reconnect_event (circuit-breaker half-open probe
and _signal_reconnect both live inside registered tool handlers). A
parked server was therefore unrevivable short of a manual /mcp reload —
the park comment's promised breaker wake could never fire.
Make the parked wait a timed wait: every _PARKED_RETRY_INTERVAL (300s)
the run task wakes and attempts one revival probe, re-parking on
failure instead of burning the full 5-retry budget each cycle. Explicit
reconnect requests still wake it immediately. Idea credit: @Hellbayne
(PR #38881, earliest never-abandon proposal), reconciled with the
park design from #53599.
_discover_tools only filled self._tools; registry registration happened
only in _discover_and_register_server (initial start) and _refresh_tools.
After parking deregistered a server's tools, a revival rebuilt the
transport but published zero tools — a phantom recovery.
Register freshly discovered tools whenever _ready is set and the
registry entry list is empty. Extracted from PR #54139 by @nicha16
(the remainder of that PR reverses the park design and is not taken).
The local retries variable in MCPServerTask.run() accumulated across
transient disconnections — each transport exception incremented it, but
only clean transport returns (auth recovery / manual refresh) or
park-wake reset it. Five transient blips over a long-uptime gateway
would permanently park the MCP server.
Promote retries to instance attribute _reconnect_retries and reset it
at all 4 session-establishment sites in _run_stdio / _run_http, so only
consecutive failures without successful reconnection count toward the
parking budget.
Fixes#57604
Port from anomalyco/opencode#33533. Native MCP tools now register as
mcp__<server>__<tool> (double-underscore delimiter) instead of
mcp_<server>_<tool>, aligning with the convention used by Claude Code,
Codex, and OpenCode.
The double-underscore delimiter disambiguates the server/tool boundary
even when either component contains underscores (the single-underscore
form was ambiguous, which is why is_mcp_tool_parallel_safe already had to
track provenance in a side-map). It also unifies native registration with
the Anthropic-OAuth wire form (_MCP_TOOL_PREFIX = 'mcp__'), so the
single->double promotion that path performed is now a no-op for native
tools while still handling legacy replayed names.
- tools/mcp_tool.py: add MCP_TOOL_NAME_PREFIX + mcp_prefixed_tool_name()
helper; route _convert_mcp_schema, utility schemas, refresh stale-set,
and the parallel-safe prefix gate through it
- agent/transports/codex_event_projector.py: mirror convention in the
deterministic call_id input for MCP server-executed tool calls
- tests: update produced-name assertions to the new convention
Port from anomalyco/opencode#34529: MCP servers can emit
notifications/message logging notifications (RFC 5424 levels), but the
MCP SDK's default logging_callback silently discards them — server-side
warnings/errors during tool calls were invisible.
- tools/mcp_tool.py: pass a logging_callback to every ClientSession
(stdio, SSE, streamable HTTP old+new API paths via the shared
sampling_kwargs sites), mapping the 8 MCP log levels onto Python
logging levels and tagging entries with [server/logger] origin.
- JSON-serialize non-string payloads, cap at 2000 chars so a chatty
server can't flood agent.log, never raise from the handler.
- Gated on SDK support (_check_logging_callback_support) mirroring the
existing message_handler gate for old SDK versions.
- tests/tools/test_mcp_server_log_notifications.py: 10 tests covering
level mapping, origin tagging, JSON payloads, truncation, and the
never-raise contract.
A Cursor-style MCP manager inside Capabilities, plus the backend it needs.
- Server list with brand/favicon avatars + live status dot and a capability
summary (N tools, M prompts, K resources); Servers | Catalog views.
- Catalog: one-click install of Nous-approved servers with required-env prompts.
- GUI OAuth: Authenticate opens the system browser from the TTY-less backend and
verifies a token actually lands; header/API-key servers are never pushed down
OAuth; a dirty mcp.json can't drop a freshly-persisted auth field.
- Full-width mcp.json editor (ecosystem document format) + pinned stdio/agent
LogTail; probes cached 5m and keyed by (profile, config) so revisiting never
respawns the fleet or shows a stale probe.
- Whole-map persistence (PUT /api/mcp/servers) so deletes/toggles actually stick
(the generic /api/config deep-merge could not remove keys).
- perf: MCP probe/auth no longer hold the global skills lock, so a slow stdio
spawn can't stall every other request into a 15s timeout.
- per-tool include/exclude gating (lib/mcp-tool-filter) mirroring the CLI loader.
Root cause: gateway spawns LSP servers (jdtls/pyright/yaml-ls) and
slash_worker without start_new_session=True, so they inherit the
gateway process group (= TUI parent PID). When mcp_tool
_snapshot_child_pids() races with these spawns during stdio MCP
server startup, non-MCP children leak into _stdio_pgids with the
TUI parent PGID. shutdown_mcp_servers() then killpg(tui_parent_pid,
SIGTERM), killing the TUI itself.
Evidence: tui_gateway_crash.log shows recurring SIGTERM stacks:
shutdown_mcp_servers -> _kill_orphaned_mcp_children ->
_send_signal -> killpg(pgid, sig) -> SIGTERM received
Fix (3 layers):
1. agent/lsp/client.py: add start_new_session=True to LSP server
spawn so each LSP server gets its own process group/session.
2. tui_gateway/server.py: same fix for slash_worker spawn, the
symmetric root-cause patch so no gateway direct child shares
the TUI parent pgid.
3. tools/mcp_tool.py: add _filter_mcp_children() defense-in-depth
that drops non-MCP children (slash_worker, jdtls/eclipse LSP)
from the PID delta before they can poison _stdio_pgids.
The MCP input-schema normalizer in _normalize_mcp_input_schema promotes the
legacy JSON Schema 'definitions' meta-keyword to '$defs' (draft 2019-09+)
so local '$ref' resolution works downstream. The previous walk renamed
*any* key named 'definitions' anywhere in the tree, including inside
'properties' dicts. That turned user-facing parameter names into '$defs',
producing property keys that contain '$', which Anthropic and OpenAI
both reject with HTTP 400 (pattern '^[a-zA-Z0-9_.-]{1,64}$').
Real-world repro: an MCP server that exposes a CI/pipelines tool whose
'definitions' parameter is an array of pipeline-definition IDs. Such a tool
is enough on its own to break every conversation, because the full tools
array is sent on every request.
Fix: when descending into a 'properties' or 'patternProperties' mapping,
iterate property-name -> schema pairs directly, leaving the property names
verbatim. Ordinary JSON Schema semantics resume inside each property's
schema, so a legitimately nested 'definitions' meta-keyword inside a
property's schema is still promoted.
Adds two regression tests:
- test_definitions_as_property_name_is_preserved (the property-name case)
- test_definitions_property_and_meta_keyword_coexist (both forms in one
schema; the property name stays, the meta-keyword promotes)
OAuth-protected MCP servers (e.g. Hospitable) return 200 text/html on an
unauthenticated HEAD probe — a login/landing page the server cannot substitute
for a real MCP response without a Bearer token. The preflight cannot
distinguish this from a misconfigured URL, so it raises NonMcpEndpointError
before the OAuth browser flow has a chance to run.
Add `and self._auth_type != "oauth"` to the preflight condition in
MCPServerTask.run(). The probe is inapplicable to OAuth servers: their URL
legitimacy is established by .well-known/oauth-protected-resource during the
OAuth handshake, not by a GET content-type check.
Concrete repro: Hospitable (https://mcp.hospitable.com/mcp) returns
`200 text/html` to an unauthenticated httpx HEAD. Without the guard:
✗ NonMcpEndpointError at `hermes mcp test`
With the guard:
✓ Connected (1487ms) — 63 tools discovered
Relation to open PRs:
- #37598 adds a POST probe fallback for POST-only non-OAuth servers (e.g.
DocuSeal), but only passes when POST returns 2xx + MCP content-type.
Hospitable returns 401 on the POST probe (Bearer challenge), so #37598
does not cover this case.
- #49463 extends the POST probe to also pass on non-2xx auth challenges
(making it OAuth-aware), but is labeled duplicate of #37598 and may not
land independently.
This fix is complementary: it handles OAuth servers with zero extra
round-trips rather than adding a POST probe step.
Tests:
- test_oauth_server_html_response_raises_without_skip: documents that
_preflight_content_type raises NonMcpEndpointError for 200 text/html
(the underlying issue), with an OAuth-server docstring.
- test_run_skips_preflight_for_oauth: verifies that run() does NOT invoke
_preflight_content_type when auth_type=="oauth", using class-level
monkeypatching so the gate is exercised without a live MCP transport.
23 passed tests/tools/test_mcp_preflight_content_type.py
A long-running gateway session could permanently lose an MCP server: once a
stdio subprocess died (or transient drops accumulated over the session), the
run loop exhausted its reconnect budget and returned, orphaning the task. With
no listener for _reconnect_event, the circuit breaker's half-open probe could
never revive the server — every probe hit a dead/absent session, re-armed the
60s cooldown, and looped forever until a full gateway restart (#16788).
Root cause was split ownership of transport liveness between the run loop and
the tool handler, plus a permanent give-up path. Fixed by one invariant: a
non-shutdown server task is always reconnectable.
- run loop parks (deregisters phantom tools, then awaits _reconnect_event)
instead of returning when the reconnect budget is exhausted, so the task
stays alive as a dormant listener
- retry budget resets on every successful (re)connect, so a healthy
long-lived server can't accumulate lifetime drops into a death sentence
- half-open probe with no live session signals a reconnect (reviving a
parked/dead task and respawning a dead stdio subprocess) and returns a
clean 'reconnecting' error instead of writing into a dead pipe
- breaker resets on successful session init across all transports
(stdio/HTTP/SSE) — fully transport-agnostic, no PID/pipe polling
Builds on the closed-PR cluster for this issue: keeps #49255's deregister-on-
exhaustion insight and #21006's signal-don't-probe insight, discards the racy
os.kill PID machinery.
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
During stdio MCP server startup, _run_stdio (an async method) called the
synchronous check_package_for_malware() inline. That makes a blocking
urllib HTTPS POST to api.osv.dev whose own timeout doesn't reliably cover a
stalled SSL handshake, so an intermittent network issue froze the entire
asyncio event loop for up to ~120s — blowing past the TUI/gateway's 15s
startup budget and showing "gateway startup timeout".
Run the check via asyncio.to_thread (off the loop) AND bound it with
asyncio.wait_for(timeout=_OSV_MALWARE_CHECK_TIMEOUT_S=12s). The malware check
is fail-open, so on timeout we log and proceed rather than blocking startup.
Salvaged from #29190 by @qdaszx (re-applied on current main — the call site
moved since the PR was opened), combining the to_thread approach also proposed
in #29192 by @ygd58. Two load-bearing tests: event-loop-not-blocked-during-
check and timeout-fails-open — both mutation-verified to fail against the old
inline blocking call.
Closes#29184.
Co-authored-by: ygd58 <buraysandro9@gmail.com>
/reload-mcp -> shutdown_mcp_servers -> _kill_orphaned_mcp_children(include_active=True)
-> _send_signal -> killpg(pgid, SIGTERM). When a tracked MCP stdio child shares
the gateway's OWN process group, killpg delivers SIGTERM to the gateway itself,
firing its SIGTERM handler -> os._exit(0): /reload-mcp crashes the gateway.
Pre-compute the gateway's own pgid (os.getpgrp(), None on Windows/restricted)
and, in _send_signal, skip killpg when pgid == own pgid, falling through to the
per-pid os.kill path so the child is still reaped without self-signaling.
Adds a regression test (folded in) that pins the guard: with a tracked pgid
equal to the gateway's own pgid, killpg is never called for that pgid and the
per-pid kill fallback is used. Mutation-checked.
Co-authored-by: kshitijk4poor <82637225+kshitijk4poor@users.noreply.github.com>
A server that doesn't implement the optional 'ping' utility answers a
keepalive ping with JSON-RPC method-not-found. _is_method_not_found_error
latches that condition so the probe falls back to list_tools instead of
reconnect-looping.
The substring fallback only matched 'method not found' / '-32601' /
'not found: ping'. Servers that surface method-not-found as the common
'Unknown method: <name>' phrasing without a structural -32601 code (e.g.
agentmemory's MCP server) slipped through, so the fallback never latched
and the keepalive reconnect-looped every cycle.
Add 'unknown method' to the substring fallback so the ping->list_tools
keepalive fallback latches for these servers too.
Fixes#50028.
