Commit graph

14868 commits

Author SHA1 Message Date
teknium1
48788032da fix(tui): derive gateway-owned sources from the Platform enum, not a hardcoded list
The salvaged guard used a hand-maintained frozenset of 14 platform names —
several of which (line, wechat, facebook, imessage, googlechat) aren't
actual Hermes Platform values, while real ones (whatsapp_cloud, feishu,
wecom, dingtalk, qqbot, yuanbao, plugin platforms like irc) were missing.
Resolve the source through gateway.config.Platform instead (built-ins +
registered plugin platforms via _missing_), with an explicit exclusion set
for self-owned/local sources. Adds tests for the guard and both reap paths.
2026-07-07 22:15:36 -07:00
AIalliAI
f5ef7ee9da fix(tui): prevent ws_orphan_reap from ending gateway-originated sessions
Guard _finalize_session's db.end_session() call against gateway-owned
sessions (telegram, bluebubbles, discord, etc.).  The TUI is a viewer
for these sessions, not the lifecycle owner.  Unconditionally ending
them in state.db creates a Groundhog Day routing loop: the gateway's
#54878 self-heal detects the stale entry, recovers to the parent
session, context compression splits back to the reaped child, and the
cycle repeats on every inbound message — causing complete conversational
context amnesia.

Fixes #60609
2026-07-07 22:15:36 -07:00
teknium1
ecc6725855 fix(gateway,cron): reconcile #60612 + #60631 onto one drain surface
Keep #60631's get_running_job_ids() snapshot + _active_cron_job_count()
(import-guarded for minimal test doubles) as the single read path, and
retarget #60612's drain tests at it. Drops the redundant
cron_jobs_in_flight() helper so there is one surface, not two.
2026-07-07 22:15:04 -07:00
joaomarcos
8a573bb6e7 fix(cron): stop interrupted jobs from delivering their pre-kill output
Follow-up to the previous commit on #60432. The status-write guard
(_consume_interrupted_flag, checked right before mark_job_run) closes
the false-success bookkeeping gap, but run_one_job delivers its result
BEFORE that check: delivery happens right after run_job() returns,
mark_job_run happens at the very end. A job whose tool subprocess was
killed mid-flight can still produce a plausible-looking final_response
from the truncated output, and that response would reach the user via
_deliver_result before the interrupted flag was ever consulted --
correct status in jobs.json, wrong message already sent.

Adds _is_interrupted(), a non-destructive peek at the same
_interrupted_job_ids set (_consume_interrupted_flag stays as the
consuming, authoritative check right before the status write -- this
needed a peek instead since the flag has to still be visible there).
Checked right after save_job_output, before the deliver_content
decision: if the run looked successful but was flagged interrupted,
force success=False with an explicit interruption message. This
routes delivery through the existing _summarize_cron_failure_for_delivery
path (the same one a real failure already uses) instead of the raw
final_response, so the user gets an honest "this run was interrupted"
instead of a truncated/misleading result.

Testing: 4 new tests in tests/cron/test_shutdown_interrupt.py --
_is_interrupted peek semantics (false/true/does-not-clear, as opposed
to the consuming _consume_interrupted_flag), and the delivery-gate
test itself, which mocks run_job to return a normal-looking success
with a "plausible final response" while the job is pre-marked
interrupted, and asserts _deliver_result receives the failure summary
("This run was interrupted.") instead, with the summarizer's error
argument confirmed to mention the interruption.

Fail-then-pass: reverted cron/scheduler.py only, the 4 new tests fail
(3 on the missing _is_interrupted attribute, 1 -- the delivery-gate
test -- on _summarize_cron_failure_for_delivery never being called,
i.e. the raw response would have gone out); restored, all 16 tests in
the file pass.

Regression: tests/cron/ (683 tests) + test_cron_active_work_drain.py +
test_gateway_shutdown.py + test_shutdown_cache_cleanup.py -- 11
pre-existing failures (Unix file-permission-bit and path-tilde
assertions that don't apply on this Windows dev box), matching the
same set already established as pre-existing in the prior commit's
regression check. Zero new failures.

Continues #60432
2026-07-07 22:15:04 -07:00
joaomarcos
24e9ed73c2 fix(gateway,cron): make shutdown drain visible to in-flight cron work
Cron jobs run through cron/scheduler.py's own ThreadPoolExecutor via a
standalone AIAgent (run_job/run_one_job), entirely outside
GatewayRunner._running_agents -- the dict _drain_active_agents() and
every other active-work check on that class reads. A gateway shutdown
(/update, /restart, and SIGUSR1 all funnel through the same stop())
could log active_at_start=0 and immediately kill tool subprocesses
while a cron job's terminal command was still running, with no wait
and no indication anything was interrupted.

Real-world impact (from the issue): a scheduled daily briefing cron
job was in flight during /update, its tool subprocess got killed
by the unconditional shutdown cleanup, and the job was never marked
failed -- it simply never completed or delivered, with no error
surfaced anywhere. A repro with a 30-minute `sleep` cron job in flight
during /update reproduced the same pattern: subprocess killed at
+0.22s of drain (active_at_start=0), the job's agent thread continued
in-process and produced a plausible-looking final response from the
truncated tool output, and the scheduler marked the run successful.

Root cause is layered, not a single line:

1. GatewayRunner._drain_active_agents() only waits on _running_agents.
   Cron work was invisible to it, so drain returned instantly whenever
   the only active work was a cron job.
2. Even with visibility, the shutdown's final tool-subprocess kill
   (process_registry.kill_all()) is a global, unconditional sweep with
   no per-job targeting -- a long-running cron job that outlives the
   drain timeout still gets its subprocess killed.
3. cron/scheduler.py had no way to detect that a job's tool subprocess
   was killed out from under it mid-run; the agent thread kept going
   and its eventual (often degraded but plausible-looking) response
   got reported as a normal successful completion.

Fix, three parts:

- cron/scheduler.py: expose get_running_job_ids() (thread-safe
  snapshot of the existing _running_job_ids set, already used to
  prevent double-dispatch) so the gateway can read cron's in-flight
  state without reaching into private module internals.

- gateway/run.py: GatewayRunner._active_cron_job_count() reads that
  snapshot. _drain_active_agents() now waits on
  (_running_agents OR active cron jobs), so a cron-only workload gets
  the same bounded wait chat sessions already get instead of an
  instant active_at_start=0. Shutdown drain logging gains
  cron_active_at_start/cron_active_now fields alongside the existing
  ones (unchanged, for compat).

- cron/scheduler.py: mark_running_jobs_interrupted(reason), called by
  gateway/run.py's _kill_tool_subprocesses() right after
  process_registry.kill_all(), marks every job still in
  _running_job_ids at that instant as failed/interrupted via the
  existing mark_job_run() -- and records the job IDs in
  _interrupted_job_ids BEFORE writing, so run_one_job()'s own
  eventual completion for the same run (racing in its own thread)
  checks that flag and skips its normal write instead of clobbering
  the interrupted status with a false "ok" produced from the
  now-truncated tool output. This does not attempt to correlate a
  killed PID to a specific job ID (process_registry tracks PIDs, not
  job IDs) -- any job still dispatched at the moment of a forced kill
  is treated as interrupted, matching the existing coarser precedent
  set by _interrupt_running_agents(), which interrupts every entry in
  _running_agents on a drain timeout without per-agent correlation
  either.

Deliberately out of scope (flagged in the issue as a separate,
lower-priority concern): startup-time reconciliation of cron runs that
started but never reached a terminal status.

Testing:

- tests/cron/test_shutdown_interrupt.py (12 tests): get_running_job_ids
  snapshot semantics, mark_running_jobs_interrupted marking/no-op/
  partial-failure behavior, and -- the core race guard -- run_one_job
  skipping its own last_status write (both the success path and the
  exception path) when the shutdown path already marked the run
  interrupted, with a control test proving ordinary un-interrupted
  completions are unaffected.

- tests/gateway/test_cron_active_work_drain.py (9 tests):
  _active_cron_job_count reading cron state and failing closed (0) if
  the cron module is unavailable; _drain_active_agents waiting for an
  in-flight cron job the same way it waits for chat sessions, timing
  out if the job outruns the window, and leaving existing chat-session
  drain behavior unchanged; a full runner.stop() integration test
  (drain-timeout path) proving mark_running_jobs_interrupted actually
  fires with the right job ID when a tool subprocess is force-killed,
  plus a no-op control when nothing cron-related is in flight.

- tests/gateway/test_shutdown_cache_cleanup.py: added
  _active_cron_job_count() to that file's hand-rolled _FakeGateway test
  double, which stop() now calls -- without it those 8 pre-existing
  tests AttributeError (caught by fail-then-pass below, not a
  production bug).

Fail-then-pass: reverted gateway/run.py + cron/scheduler.py, all 21
new tests fail (fixture/attribute errors -- the feature doesn't exist
yet); restored, all 21 pass.

Regression check: ran the full plausibly-affected surface --
tests/gateway/{test_gateway_shutdown,test_restart_drain,
test_restart_notification,test_restart_redelivery_dedup,
test_restart_resume_pending,test_restart_service_detection,
test_shutdown_cache_cleanup,test_stuck_loop,test_clean_shutdown_marker,
test_external_drain_control,test_session_state_cleanup,
test_update_command,test_update_streaming}.py plus tests/cron/ (944
tests) -- against a clean upstream/main checkout and against this
branch. Diffed the two FAILED lists: identical, 20 pre-existing
failures on both sides (Windows-locale/cp1252 file-encoding issues and
Unix-permission-bit assertions that don't apply on this Windows dev
box), zero new failures, zero fixed-by-accident. The 8
test_shutdown_cache_cleanup.py failures found mid-development were
from the _FakeGateway gap above, fixed in the same commit and
confirmed clean on the final rerun (diff against baseline: exit 0).

Fixes #60432
2026-07-07 22:15:04 -07:00
HexLab98
e6077af279 test(gateway): cover cron drain during gateway shutdown (#60432) 2026-07-07 22:15:04 -07:00
HexLab98
862aee4956 fix(gateway): drain in-flight cron jobs before shutdown tool kill
/update and other shutdown paths only waited on gateway session agents,
so active cron tool work was killed immediately in final-cleanup while
the scheduler could still mark the job successful (#60432).
2026-07-07 22:15:04 -07:00
teknium1
a208b7eeb4 chore: add AUTHOR_MAP entry for neoguyverx (PR #60526 salvage) 2026-07-07 22:14:33 -07:00
teknium1
6695640c1d fix(tools): make the YAML write gate syntax-only so multi-doc/tagged YAML isn't refused
safe_load() raises ComposerError on multi-document streams (k8s manifests)
and ConstructorError on application-defined tags (CloudFormation !Sub,
Ansible !vault) — both valid YAML syntax. Now that the linter's verdict is
a fail-closed write gate, those false positives would refuse legitimate
writes outright. Switch to yaml.parse() (scanner+parser only), which still
catches real syntax failures.
2026-07-07 22:14:33 -07:00
Neo Guyver
2e1982f83d Fail closed on invalid JSON/YAML/TOML writes instead of writing then reporting
write_file() previously called _atomic_write() first and only ran the
JSON/YAML/TOML/Python syntax check afterward as an informational lint
delta -- a parse failure never set the top-level `error` key, so a
corrupt structured-data write still landed on disk (and file_tools.py's
files_modified gating, which keys off `error`, silently reported it as
a successful modification).

Move the in-process syntax check for JSON/YAML/TOML ahead of
_atomic_write() and refuse the write outright on a parse failure: no
temp file, no rename, nothing touches disk, and the result carries a
top-level `error` so callers correctly see it as unmodified.

Deliberately scoped to _FAIL_CLOSED_INPROC_EXTS (JSON/YAML/TOML), not
all of LINTERS_INPROC -- .py is excluded because this codebase's own
test fixtures (TestPatchReplacePostWriteVerification et al.) write
arbitrary non-Python text through *.py paths purely to exercise
write-mechanics; a hard block there broke 3 previously-passing tests
during development. Python keeps its pre-existing non-blocking
lint-delta report.

Adds tests/tools/test_write_file_syntax_gate.py: invalid JSON/YAML/YML/
TOML refused with nothing written (new file) and nothing modified
(existing file); valid JSON/YAML still written byte-for-byte; a
non-linted extension with garbage content is unaffected; invalid Python
is confirmed NOT hard-refused (still just reported).
2026-07-07 22:14:33 -07:00
ethernet
4d7f8ade3e
feat(install): warn pip/Homebrew installs are unsupported (CLI, TUI, desktop) (#57225)
* feat(install): warn pip/Homebrew installs are unsupported (CLI, TUI, desktop)

pip and Homebrew are now Unsupported install methods per
website/docs/getting-started/platform-support.md. Surface a
warn-don't-block deprecation notice everywhere the install method is
already shown, pointing at the platform-support docs and noting these
installs will not receive further updates. NixOS (Tier 2) is untouched.

- hermes_cli/config.py: shared is_unsupported_install_method() /
  format_unsupported_install_warning() helpers so the wording and docs
  link stay consistent across every surface.
- hermes_cli/banner.py: generalize the existing pip-only banner
  warning to also cover Homebrew.
- hermes_cli/main.py: hermes update and hermes update --check print
  the warning before proceeding (still update; warn, don't block).
- tui_gateway/server.py: session.info gains install_warning.
- ui-tui: SessionPanel renders install_warning alongside the existing
  'N commits behind' notice.
- apps/desktop: SessionRuntimeInfo/GatewayEventPayload gain
  install_warning; applyRuntimeInfo + the live session.info event fire
  a snoozable warning toast via a new reportInstallMethodWarning(),
  mirroring the existing backend-contract-skew toast pattern. i18n
  strings added for en/zh/zh-hant/ja.
- Tests: updated pip banner assertions for the new wording, added a
  Homebrew banner test, and two tui_gateway session_info tests
  (install_warning present for pip, absent for git).

* fix(nix): make `hermes` in developement environment actually work

install modules as editable overlay with uv

* feat: print install method when running --version

* fix: correct detect install method when running from a subtree
2026-07-07 21:13:19 -07:00
Teknium
9de9c25f62
chore: release v0.18.2 (2026.7.7.2) (#60651) 2026-07-07 20:11:08 -07:00
Teknium
c30c9753b6
fix(whatsapp): unpin Baileys from git commit, use published 7.0.0-rc13 (#60643)
The April 2026 pin to WhiskeySockets/Baileys#01047deb existed only to
pick up the abprops bad-request fix (Baileys PR #2473) before it was
released. That fix shipped in v7.0.0-rc11 (May 2026); our pinned commit
is now 48 commits behind rc13.

The git pin forced npm to clone the repo and compile Baileys from
TypeScript source on every fresh install (~3 min), which blew past the
dashboard pairing flow's timeout. Registry install takes ~3s.

Validation: all 9 bridge.js imports present in rc13, bridge.native.test.mjs
passes (13/13), live bridge boot renders pairing QR against real WA servers.
2026-07-07 19:49:26 -07:00
Teknium
f9eca7e15f
chore: release v0.18.1 (2026.7.7) (#60595) 2026-07-07 18:14:48 -07:00
Shannon Sands
4f620a0bbc Add WhatsApp dashboard pairing flow 2026-07-07 17:45:51 -07:00
teknium1
6015ee5d2a fix: pass profile-scoped SessionDB to _session_latest_descendant in dashboard chat PTY resume
The chat PTY launch path landed on main after PR #50558 and still called
_session_latest_descendant() with the old one-arg signature. Open the
requested profile's state DB (matching the REST endpoint) so profile-scoped
resume resolves descendants in the right database.
2026-07-07 17:44:44 -07:00
Shannon Sands
543f069093 Fix dashboard chat model profile scoping 2026-07-07 17:44:44 -07:00
Ben Barclay
75de0057bc
feat(gateway): GATEWAY_MULTIPLEX_PROFILES env override for multiplex flag (#60589)
The connector now depends on the single multiplexed gateway for per-profile
relay routing, so hosted deployments need to FORCE multiplexing on regardless
of the image's config.yaml. gateway.multiplex_profiles was config.yaml-only,
which a user could leave unset or flip off.

Add GATEWAY_MULTIPLEX_PROFILES as a standard operator override on top of the
existing config key — the same 'config.yaml is canonical, env is the operator
override' pattern the Telegram/Signal require_mention bridges use:

  env (recognized token) > config.yaml (top-level or nested gateway.*) > False

- gateway/config.py: _env_multiplex_profiles_override() resolves the env var
  tri-state — recognized truthy/falsy token → bool; unset/blank/unrecognized
  → None (fall through to config). Blank is deliberately None, not False, so a
  provisioned-but-unpopulated Fly secret ('') can't shadow a config.yaml opt-in
  (the empty-secret trap). Wired into GatewayConfig.from_dict so every consumer
  (run.py, session.py via self.config) sees the resolved value.
- hermes_cli/gateway.py: the named-profile-start guard
  (_guard_named_profile_under_multiplexer) reads config.yaml directly, so it
  gets the SAME env precedence — otherwise env-forced multiplex would leave the
  guard blind and someone could start a conflicting per-profile gateway that
  double-binds a bot token. Env-forced-on trips the guard even with no
  config.yaml key; env-forced-off disables it over a config opt-in.

Tests: full 3-tier precedence in test_config.py (incl. the discriminating
env-overrides-config cases + the empty/whitespace/unrecognized fall-through
trap + resolver tri-state), mutation-verified (flipping precedence fails
exactly the two env-wins tests); guard env cases in test_multiplex_lifecycle.py.

Force-on is safe on a single-profile instance: session keys stay byte-identical
(agent:main) and the _run_agent wrapper installs the per-turn secret scope, so
the fail-closed get_secret() path is satisfied.
2026-07-08 00:34:34 +00:00
teknium1
d9a4b5a5e5 fix: validate memory provider names before filesystem lookup and setup commands
Strict charset allowlist (alnum + - _, max 64) on the {name} path param of
the memory-provider config/setup endpoints. Prevents traversal-shaped names
from reaching find_provider_dir(), and setup now 404s when neither a
loadable provider nor a plugin manifest exists, so the command-running path
is only reachable for discoverable plugins. Adds regression tests.
2026-07-07 17:27:54 -07:00
Shannon Sands
4b184cbe54 Add dashboard memory provider switching 2026-07-07 17:27:54 -07:00
Ben Barclay
4e4a69cbf7
feat(relay): carry routed profile from the connector wire source (#60586)
The multiplex machinery already routes an inbound message to a profile via
SessionSource.profile (build_session_key namespacing + the per-turn
config/credential scope in SessionStore._resolve_profile_for_key). But the
relay path never populated it: _event_from_wire rebuilt the SessionSource
field-by-field and dropped any 'profile' the connector sent, so a
Team-Gateway (connector + relay) message could not be routed to a specific
profile the way the /p/<profile>/ HTTP prefix and per-credential polling
adapters already can.

Stamp source.profile from the wire payload in _event_from_wire. This is the
last missing link for NAS-driven per-profile routing over the relay in
multiplex mode; the connector populating the field ships separately
(gateway-gateway contract adds the optional wire field).

Back-compat: absent 'profile' → None → legacy agent:main namespace,
byte-identical to today for every single-profile gateway.
2026-07-08 00:20:23 +00:00
Ben Barclay
8d66e78844
feat(dashboard): expose profile names + gateway_mode on gated /api/status (#60585)
The profile+gateway topology added in #60537 sits entirely behind the
loopback/--insecure auth gate. But a hosted agent (Hermes Cloud) binds
non-loopback with OAuth, so should_require_auth is True, and NAS reads
/api/status over the network (fly-provider.ts getInstanceRuntimeStatus)
with no session token. On that gated path the whole topology block was
omitted, so the Portal could never render the profile list.

Split the topology readout by sensitivity:
- profile NAMES (profiles) + gateway_mode are low-sensitivity product
  surface and now ride the always-public status body, surviving the auth
  gate so NAS/the Portal can enumerate profiles.
- the per-gateway detail (gateways[], carrying host ports) is deployment
  recon and stays gated alongside hermes_home / config_path / env_path /
  gateway_pid / gateway_health_url.

The collector now runs unconditionally (still in the executor, off the
event loop). No new fields; only the gate placement changes.
2026-07-08 00:20:13 +00:00
Teknium
5633fa19b8
fix(dashboard): advertise truecolor to the embedded chat TUI (#60576)
Headless/hosted deploys run the dashboard server without COLORTERM in
the process environment, so chalk inside the PTY-spawned TUI child
downgraded every skin hex color to the xterm 256 palette — the default
skin's bronze banner border (#CD7F32) snapped to palette 173 (#D7875F,
salmon red) and the gold caduceus rendered red/yellow on fresh cloud
instances. Local launches never reproduced it because the operator's
interactive terminal leaks COLORTERM=truecolor into the server env.

xterm.js always renders 24-bit RGB, so the dashboard PTY child should
always advertise truecolor: backfill COLORTERM=truecolor in
_resolve_chat_argv via setdefault (an explicit operator value wins).

Verified with a clean-env PTY probe of the real TUI binary:
no COLORTERM -> 0 truecolor SGRs / 165 palette-256 (salmon 38;5;173);
with the backfill -> 166 truecolor SGRs, exact bronze 38;2;205;127;50.
2026-07-07 17:17:51 -07:00
Shannon Sands
bf7639138e Use read-only config loader and honor HERMES_IGNORE_USER_CONFIG in delegation config 2026-07-07 17:17:38 -07:00
Shannon Sands
0263f1d12e Fix delegation config precedence 2026-07-07 17:17:38 -07:00
Teknium
6ca3d701fc
fix(gateway): only session-discover channel targets for connected platforms (#60574)
Session-based channel discovery resurrected historical origins for
platforms with no connected adapter, exposing stale send_message
targets that can no longer deliver. Gate both the enum loop and the
plugin-registry loop on the live adapter set.

Surgical reapply of the channel-directory portion of PR #25959 (branch
was 6.5k commits stale; the text-batching delay changes bundled there
were dropped - separate concern, defaults have since been retuned on
main).

Co-authored-by: Marco-Olivier Lavoie <marcolivier@gmail.com>
2026-07-07 17:04:32 -07:00
teknium1
db9e3e4ef9 docs(discord): troubleshoot silent fail-closed denials
Docs portion of PR #57067: 'bot connects but never replies' section
pointing at the gateway.log warning and the allowlist/policy knobs.

Co-authored-by: ooovenenoso <120500656+ooovenenoso@users.noreply.github.com>
2026-07-07 17:01:08 -07:00
yungchentang
3e7ade418d fix(discord): explain fail-closed allowlist default
Log a one-shot structured warning when Discord denies traffic because
no allowlist/policy is configured, and correct the setup wizard's
inverted warning text. The fail-closed default itself is unchanged.

Fixes #58682.
2026-07-07 17:01:08 -07:00
BROCCOLO1D
c3808cfc14 fix(discord): honor pairing grants for message auth 2026-07-07 17:00:58 -07:00
Teknium
551f00109d
docs(sessions): unify export docs under one overview section (#60554)
Restructures the five parallel export sections into a single 'Export
Sessions' section: a format table (jsonl/md/qmd/html/trace + --only
user-prompts), one shared-filters paragraph covering all formats, and
per-format subsections nested beneath. EN + zh-Hans.
2026-07-07 16:29:27 -07:00
Tranquil-Flow
8a726e91ba fix(tools): enable platform-native toolsets when their composite is explicitly configured (#35527)
When a user explicitly configures a platform with its native composite
(e.g. platform_toolsets.discord: [hermes-discord]), the discord and
discord_admin toolsets were silently stripped by _DEFAULT_OFF_TOOLSETS
even though the composite contains those tools. The strip could not tell
an explicit composite opt-in apart from the unconfigured default.

Track whether the platform was explicitly configured and, when it was,
exempt toolsets that are both default-off and platform-restricted to the
current platform from the strip. Only discord/discord_admin are affected
(the sole entries in both _DEFAULT_OFF_TOOLSETS and
_TOOLSET_PLATFORM_RESTRICTIONS). Unconfigured and empty-list platforms
keep the security default-off behaviour.
2026-07-07 16:27:28 -07:00
Teknium
b062083d0a
feat(dashboard): report profile + gateway topology in /api/status (#60537)
/api/status (loopback/insecure binds only) now includes:
- profiles: every profile on the host (default + named)
- gateway_mode: none | single | multiple | multiplex
- gateways: one entry per live gateway with the host ports its
  port-binding platforms listen on, plus served_profiles when the
  default gateway is multiplexing

Ports resolve from each profile's config.yaml (top-level platforms:
wins over gateway.platforms:, matching load_gateway_config precedence)
with adapter defaults as fallback. Topology enumeration runs in an
executor so the profile scan + process-table probes stay off the event
loop, and the whole block is gated behind the same loopback-only split
as hermes_home/gateway_pid so gated binds leak nothing new.
2026-07-07 16:13:03 -07:00
teknium1
838d50495f fix(mcp): guard POSIX-only kill primitives in stdio watchdog for the Windows footgun linter
signal.SIGKILL / os.killpg don't exist on Windows. The watchdog is only
spawned on POSIX (wrap site gates on os.name), but guard via getattr with
a plain terminate/kill fallback so an accidental Windows import can't
AttributeError.
2026-07-07 15:16:00 -07:00
teknium1
2d4fd1d52f test(mcp): unblock recycle-reconnect test from the parked self-probe wait
The salvaged test predates the parked-server self-probe
(_PARKED_RETRY_INTERVAL, landed on main after the PR branched): after the
final failed retry, run() parks in a real asyncio.wait that the patched
asyncio.sleep doesn't cover, stalling the test 300s. Signal shutdown once
the retry budget is exhausted so the park exits immediately.
2026-07-07 15:16:00 -07:00
teknium1
a6203839b4 docs(mcp): document idle_timeout_seconds / max_lifetime_seconds recycle keys + handshake-bound note 2026-07-07 15:16:00 -07:00
teknium1
bae3954f4f chore(release): map rainbowgore + thestudionorth in AUTHOR_MAP for MCP leak salvages 2026-07-07 15:16:00 -07:00
harjoth
ea0b42c43a Handle minimal MCP server fakes 2026-07-07 15:16:00 -07:00
harjoth
6c731fe591 Recycle idle MCP stdio servers 2026-07-07 15:16:00 -07:00
teknium1
86c5febdd1 fix(mcp): watchdog wrap after OSV preflight + forward SIGTERM to child group
Two fixes on top of the salvaged parent-death watchdog:
- Apply the watchdog wrap AFTER the OSV malware preflight so the check
  inspects the real npx/uvx package instead of the python wrapper
  (the wrap previously made the preflight a silent no-op for every
  stdio server).
- The real server runs in its own process group under the watchdog, so
  the graceful-shutdown killpg no longer reached it; the watchdog now
  forwards SIGTERM/SIGINT to the child's group, keeping wedged servers
  killable on clean shutdown.
2026-07-07 15:16:00 -07:00
Sage
5089c84dbf fix(mcp): reap orphaned stdio MCP children on ungraceful parent death
A stdio MCP server (e.g. `npx -y mcp-remote <url>`) is spawned as a direct
child of the Hermes process. Existing teardown (MCPServerTask.shutdown() /
_kill_orphaned_mcp_children()) reaps it correctly on a clean exit, but a
kill -9 / crash / force-quit of the Hermes process skips that path entirely
-- the child (and its own descendants, e.g. mcp-remote's spawned node
process) is orphaned and keeps running. Repeated ungraceful restarts pile up
N orphaned processes racing to hold the same upstream SSE session, producing
errors like 'Invalid request parameters' on legitimate reconnects.

macOS/Linux have no portable equivalent of prctl(PR_SET_PDEATHSIG) at the
Python subprocess level, so this adds a thin supervisor
(tools/mcp_stdio_watchdog.py) that:
  - execs the real command as its own child in its own process group
  - passes stdin/stdout/stderr through untouched (MCP stdio protocol
    talks directly over those streams)
  - polls the original spawning PID with the same orphan-detection
    algorithm already proven in tui_gateway/slash_worker.py (ppid
    comparison + psutil creation-time guard against PID reuse)
  - SIGTERM-then-SIGKILL's the child's process group the moment the
    original parent is gone

Wired into _run_stdio via a new _wrap_command_with_watchdog() helper,
POSIX-only (matches the existing killpg-based cleanup's platform scope),
fails open (any error resolving pid/create-time falls back to the
unwrapped command) so this can never be the reason a working MCP server
stops starting.

Verified: reproduced the exact orphan scenario standalone (fake parent
process spawns watchdog + fake long-running MCP child, kill -9 the fake
parent, confirm the watchdog reaps the child within its poll window with
zero leaked processes). Updated test_mcp_tool_issue_948.py's resolved-path
assertion to check the watchdog-wrapped command instead of the raw
resolved binary. Full test_mcp_tool.py + test_mcp_stability.py +
test_mcp_tool_issue_948.py suite: 232 passed. Full -k mcp sweep across the
whole test tree: 1003 passed, 2 skipped, 0 failed.
2026-07-07 15:16:00 -07:00
teknium1
4638f3b433 fix(mcp): widen #59349 handshake bound to HTTP transports + cancel abandoned start() task
Sibling sites of the same bug class as the salvaged stdio fix:
- SSE, streamable-HTTP (new + deprecated API) initialize() calls are now
  bounded by the same connect_timeout, so an endpoint that accepts the
  connection but never answers the handshake cannot park the run() task
  forever.
- start() now cancels its ensure_future'd run() task when the caller's
  connect timeout cancels start() itself — the orphaned-task leak was
  the root mechanism behind #59349, and this closes the class for any
  future pre-ready hang.
2026-07-07 15:16:00 -07:00
rainbowgits
1f6836cd81 fix(mcp): bound stdio initialize handshake to stop subprocess/FD leak
A stdio MCP server that never completes `initialize` (e.g. emits a
non-JSON-RPC frame and then blocks on stdin) leaks a child process plus its
stdio pipes/pidfd on every discovery-retry cycle — unbounded, until the
gateway hits EMFILE and every new open()/spawn fails (#59349).

Root cause (confirmed by instrumenting the live repro, and different from the
issue's own hypothesis): the spawned child IS captured in `new_pids`, so the
report's "new_pids empty at finally" guess is not it. The real cause is that
`session.initialize()` hangs forever on the garbage stream. `connect_timeout`
only bounds the caller's `.result()` wait on the foreground thread — it does
NOT cancel the `_run_stdio` coroutine on the background MCP loop. So the
coroutine is stuck at `await session.initialize()` permanently, its cleanup
`finally` never runs, the child is never reaped, and it stays invisible to the
orphan-reaper (whose `_orphan_stdio_pids` set never gets populated).

Fix: wrap `session.initialize()` in `asyncio.wait_for(..., connect_timeout)`
so a stalled handshake fails instead of hanging. The TimeoutError unwinds
through the SDK context managers (closing the child's stdin -> EOF -> exit)
and lets the existing `finally` reap any straggler. Cross-platform — no
signals/pgid/proc.

Scope: stdio only. The HTTP path has the same `await session.initialize()`
shape but spawns no subprocess (so it can't cause this leak) and already has
httpx transport timeouts.

Verified: the reporter's repro goes from unbounded growth to draining to zero;
added a hermetic regression test (fake transport whose `initialize()` hangs,
asserts the connect is bounded by connect_timeout) that fails on the pre-fix
code and passes on the fix; 566 existing MCP tests pass; ruff clean.

Repro confirmed on macOS (pipe FDs); the Linux-specific pidfd growth in the
report should be equivalent — the reporter offered to validate on Linux.

Closes #59349
2026-07-07 15:16:00 -07:00
teknium1
743c116fb2 fix(mcp): unify reconnect orphan reaping + move off the event loop
Merge the two cherry-picked reap call sites into one unscoped sweep at
the top of _run_stdio (the unscoped sweep is a superset of the
per-server one), and run it via asyncio.to_thread so the 2s
SIGTERM->SIGKILL escalation cannot stall the shared MCP event loop.
2026-07-07 15:16:00 -07:00
yoma
f99e9f0d27 fix(mcp): reap stdio orphans before reconnect 2026-07-07 15:16:00 -07:00
liuhao1024
086596ca2b fix(mcp): reap orphaned subprocesses before spawning new ones on retry
When an MCP stdio subprocess fails to connect (token expiry, port
contention, timeout), the run() reconnect loop retries with backoff.
Each retry calls _run_stdio() which spawns a new process pair, but the
previous failed pair was only detected as orphaned (added to
_orphan_stdio_pids) — never actually killed.  This caused rapid zombie
accumulation: 5 failed attempts × 2 procs each = 10 orphans competing
for the same port.

Add a _kill_orphaned_mcp_children() call at the top of _run_stdio(),
before the _snapshot_child_pids() baseline, so any orphans from prior
failed attempts are reaped before a new subprocess is spawned.

Fixes #57355
2026-07-07 15:16:00 -07:00
Benn Denton
79f4f78fa4 feat(chat): persist attach token, reconnect on transient close
ChatPage sends ?attach=<localStorage token> so /chat reattaches to its
live PTY across refresh. onclose: 4410=process-exit (session ended),
4409=superseded (quiet), else transient -> auto-reconnect.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 15:15:37 -07:00
Benn Denton
c3d2be073a feat(pty): periodic reaper wired into dashboard lifespan
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 15:15:37 -07:00
Benn Denton
e10e4bca82 feat(chat): reattach /api/pty sessions via ?attach= token
Keep-alive path when ?attach=<token> is present: PTY outlives the socket
via PTY_REGISTRY, reattaches on reconnect. No token = unchanged legacy
pump (_legacy_pump). detach (not close) on disconnect.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 15:15:37 -07:00
Benn Denton
41166bbe0d feat(pty): PtySessionRegistry with reap + capacity
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 15:15:37 -07:00
Benn Denton
e5ac169c28 feat(pty): PtySession drain/attach/detach with EOF close 4410
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 15:15:37 -07:00