Cherry-picked from #39840 by @flyinhigh and rebased cleanly on main.
- Defer config fetch in createGatewayEventHandler until gateway.ready to
avoid render-phase RPC that can mutate transcript state and trigger
React error 301 in embedded dashboard PTYs.
- Use undici WebSocket fallback when globalThis.WebSocket is unavailable
(Node attach mode and sidecar mirror sockets).
- Add regression tests for both fixes.
Co-authored-by: flyinhigh <flyinhigh@users.noreply.github.com>
* fix(tui): persist gateway lifecycle breadcrumbs to crash log
A backend SIGTERM (`=== SIGTERM received ===` in tui_gateway_crash.log) is
always a parent action — `gw.kill()` (graceful-exit on a signal to Node, or an
explicit /quit) or `start()` replacing a live child. #31051 added parent-side
lifecycle breadcrumbs but left them in an in-memory CircularBuffer that dies
with the process, so SIGTERM crash reports arrive with no parent context and no
way to tell a signal-driven kill from a memory-critical `process.exit(137)`
(which closes the child's stdin → clean EOF, not SIGTERM).
Persist the death-explaining breadcrumbs (spawn / transport-exit / child-exit /
replace-live-child / kill-reason / startup-timeout) plus the graceful-exit
signal name and the memory-critical exit into the same crash log the Python
side writes, so they interleave by timestamp next to the child's panic entry —
making these recurring reports diagnosable.
Gated off under VITEST so unit tests stay hermetic.
* feat(tui): auto-recover the session when the gateway dies unexpectedly
When a still-owned gateway child dies while the TUI is alive (a crash, OOM
process.exit, or a SIGTERM/SIGHUP forwarded to it), the app currently nulls the
session and drops to an inert "gateway exited" state — the user loses a long
session and has to restart + re-run everything. That single behavior is most of
the "TUI doesn't survive heavy work" complaint, independent of what does the
killing.
The 'exit' event only reaches this handler on an *unexpected* death: a user
/quit calls process.exit before it fires, and a replaced child is identity-
skipped in GatewayClient. So on exit we now respawn the gateway and resume the
session that was live (history is persisted in SQLite) via a one-shot
recoverSidRef the next gateway.ready consults before forging a new session. The
in-flight reply is lost (it died with the process) but the session survives.
Bounded to GATEWAY_RECOVERY_LIMIT (3) attempts per GATEWAY_RECOVERY_WINDOW_MS
(60s) so a gateway that crash-loops on startup can't spawn-storm; past the
budget we fall back to the inert state.
* fix(tui): sanitize newlines + soften SIGTERM-cause claim in parentLog
Address PR review:
- recordParentLifecycle collapses embedded \r\n so a multi-line value (e.g. an
error message) stays a single breadcrumb and can't masquerade as a separate
entry or as the child's panic output sharing the crash log.
- Reword the header: a backend SIGTERM is *usually* a parent action but can come
straight from an external supervisor (s6, cgroup OOM, stray kill); the
presence/absence of a [tui-parent] line before the child's panic is precisely
what disambiguates the two.
* fix(tui): clear sid during recovery + extract/test the recovery budget
Address PR review:
- Null `sid` immediately in the gateway exit handler. While the gateway is down
(busy=false) the old sid would otherwise let sid-guarded effects (the 1.5s
session.active_list poll, queue drain) fire RPCs at a dead/respawning gateway.
recoverSidRef carries the session forward; resumeById restores sid on ready.
- Extract the respawn budget into a pure evalRecovery() (gatewayRecovery.ts) and
unit-test the bound: allows GATEWAY_RECOVERY_LIMIT within the window, blocks
past it, and prunes attempts older than the window so recovery re-arms.
* fix(tui): cap parent-log breadcrumb length (PR review)
Truncate a single persisted breadcrumb to 4096 chars (matching GatewayClient's
in-memory log-line cap) so a pathological value — e.g. a giant error string —
can't bloat the shared crash log or add noticeable blocking on the synchronous
append during a failure path. Covered by a test.
* fix(tui): keep "recovering session…" status visible during resume (PR review)
resumeById() synchronously sets status to 'resuming…' on entry, so the
recovery branch now applies its 'recovering session…' label *after* calling
resumeById — the distinct label sticks for the duration of the resume RPC
(which later flips to 'ready') instead of being immediately clobbered. Test
updated to assert the ordering.
* fix(tui): keep recovery budget alive across a startup crash-loop (PR review)
deadSid was read from getUiState().sid, which the first exit nulls — so if the
respawned gateway crash-looped before gateway.ready (resumeById never restored
sid), later exits saw null and abandoned the session after a single attempt,
defeating the bounded retry budget.
Lift the whole decision into a pure planGatewayRecovery() that falls back to the
pending recoverSidRef target when the live sid is already cleared, and unit-test
the crash-loop sequence (keeps retrying the same session up to the limit, then
falls back to inert). Supersedes evalRecovery.
* chore(tui): drop non-null assertion + clarify breadcrumb cap comment (PR review)
- Recovery branch guards on `recoverSidRef && recoverSid` so the ref write needs
no `!` assertion (avoids a future unsafe refactor).
- Reword the parentLog cap comment: it slices the value to 4096 chars and
appends a short truncation marker (so the written line is slightly longer),
rather than implying a strict 4096-byte limit.
* chore(tui): soften "absence ⇒ external signal" + "any in-flight reply" (PR review)
- parentLog header: a missing [tui-parent] line only *suggests* an external
signal (the logger is best-effort: VITEST-disabled, failed append swallowed),
not a definitive conclusion.
- Recovery notice says "any in-flight reply was lost" since the gateway can also
exit while idle.
* fix(tui): log parent gateway lifecycle exits
Add parent-side breadcrumbs for TUI gateway shutdown and transport exits so future backend EOF/SIGTERM reports identify the parent action that caused them.
* chore(tui): retrigger lifecycle logging checks
Retry transient GitHub checkout failures on the lifecycle logging PR.
* feat(tui): support attaching to an existing gateway
Allow the TUI gateway client to connect via HERMES_TUI_GATEWAY_URL while preserving spawned gateway fallback, and mirror event frames to sidecar feeds so dashboard tool activity remains visible.
* review(copilot): redact attach URLs and gate stale transport exits
Strip query strings (and any user info) from gateway / sidecar URLs before logging or surfacing them in `gateway.start_timeout`, so attach tokens never leak into the TUI log tail or activity feed. Also gate the spawned-proc and websocket close handlers on transport identity so a stale child or socket cannot clear a freshly-started ready timer or reject newly-issued pending requests during reconnect.
* review(copilot): tighten transport restart and shutdown lifecycle
Reject any in-flight RPCs in resetStartupState so callers do not hang on promises issued to the previous transport when start() swaps a child or socket. Have kill() explicitly reject pending so attach-mode promises drain after an intentional shutdown, and reattach when HERMES_TUI_GATEWAY_URL rotates between requests instead of silently keeping the old session. Fold the spawned child error path through handleTransportExit so a failed spawn clears the startup timer and emits a single exit event. Also null the websocket reference before calling close so the identity guard correctly tags stale close events on real WebSocket timing. Locks the new behaviors in with regression tests for kill, URL rotation, and stale-pending cleanup.
* review(copilot): swallow stray ws connect rejection and isolate test env
Attach a no-op catch handler on the websocket connect promise so an unobserved connect-error / early-close rejection cannot surface as an unhandled promise rejection in Node when no request is currently racing the open. Snapshot HERMES_TUI_GATEWAY_URL / HERMES_TUI_SIDECAR_URL in beforeEach and restore them in afterEach so vitest runs that set those env vars beforehand do not get permanently cleared.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* review(copilot): hoist wire decoder and harden redact fallback
Reuse a single module-level TextDecoder for binary websocket frames so high-frequency attach-mode traffic does not allocate one per message. Strengthen the redactUrl fallback so embedded user:pass@ credentials are also masked when the WHATWG URL parser rejects the input, and pin the new behavior with a regression test that drives a malformed bearer URL through the gateway-stderr publish path.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* review(copilot): force redact fallback path with deterministic fixture
Replace the "%zz" user-info fixture, which WHATWG URL actually accepts in recent Node and silently routed the test back through the structured-URL branch, with a port-99999 fixture that the parser rejects across Node versions. Add a pre-flight `expect(() => new URL(fixture)).toThrow()` assertion so a future URL-parser change can never silently bypass `redactUrl()`'s fallback again.
* review(copilot): sanitize websocket constructor failures
Avoid logging raw WebSocket constructor error messages because some implementations include the full input URL, including token-bearing query strings. Log the redacted gateway or sidecar URL with the error class instead, and add regression coverage for constructor-throw paths on both attach and sidecar sockets.
* review(self): restart transport on attach-mode transition
Route runtime HERMES_TUI_GATEWAY_URL changes through start() so switching from spawned-gateway mode to attach mode also tears down the previously spawned Python child instead of leaving it alive. Keep the existing fast-fail behavior for pending RPCs. Also make constructor-failure logging fully generic after the redacted URL, avoiding even implementation-specific error class text in the log tail.
* review(copilot): use websocket wording for attach close errors
When the attached websocket closes, reject pending RPCs with an explicit websocket-closed reason instead of the spawned-process oriented `gateway exited` wording. Add coverage to ensure close code 1011 surfaces as `gateway websocket closed (1011)`.
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(tui): append gateway stderr tail to start_timeout activity
`gateway.start_timeout` previously published only `cwd` + `python`,
which made TUI startup failures hard to disambiguate. The user saw
`gateway startup timed out · /path/to/python /repo · /logs to inspect`
with no signal whether the actual cause was a wrong python interpreter,
a missing dependency, or a config parse failure.
Plumb a 20-line stderr tail through the event so the most useful lines
land directly in the TUI activity feed, capped to the last 8 non-empty
lines for readability:
* `gatewayClient.ts` — collect `getLogTail(20)` when the readyTimer
fires and attach it as `payload.stderr_tail`.
* `gatewayTypes.ts` — extend the `gateway.start_timeout` event union
with the new optional field.
* `createGatewayEventHandler.ts` — emit the trimmed lines after the
existing `gateway startup timed out` activity entry, classified
`error`.
Tests: regression test in `createGatewayEventHandler.test.ts` checks
that `ModuleNotFoundError` / `FileNotFoundError` lines from the tail
land in `getTurnState().activity` so they show up in the UI immediately.
Validation: `npm run type-check` clean, `npm test --run` 390/390.
* review(copilot): filter blanks before slice and cap stderr tail at 120 chars
Six small fixes, all valid review feedback:
- gatewayClient: onTimeout is now a class-field arrow so setTimeout gets a
stable reference — no per-request bind allocation (the whole point of
the original refactor).
- memory: growth rate was lifetime average of rss/uptime, which reports
phantom growth for stable processes. Now computed as delta since a
module-load baseline (STARTED_AT). Sanity-checked: 0.00 MB/hr at
steady-state, non-zero after an allocation.
- hermes_cli: NODE_OPTIONS merge is now token-aware — respects a
user-supplied --max-old-space-size (don't downgrade a deliberate 16GB
setting) and avoids duplicating --expose-gc.
- useVirtualHistory: if items shrink past the frozen range's start
mid-freeze (/clear, compaction), drop the freeze and fall through to
the normal range calc instead of collapsing to an empty mount.
- circularBuffer: throw on non-positive capacity instead of silently
producing NaN indices.
- debug slash help: /heapdump mentions HERMES_HEAPDUMP_DIR override
instead of hardcoding the default path.
Validation: tsc clean, eslint clean, vitest 102/102, growth-rate smoke
test confirms baseline=0 → post-alloc>0.
Long TUI sessions were crashing Node via V8 fatal-OOM once transcripts +
reasoning blobs crossed the default 1.5–4GB heap cap. This adds defense
in depth: a bigger heap, leak-proofing the RPC hot path, bounded
diagnostic buffers, automatic heap dumps at high-water marks, and
graceful signal / uncaught handlers.
## Changes
### Heap budget
- hermes_cli/main.py: `_launch_tui` now injects `NODE_OPTIONS=
--max-old-space-size=8192 --expose-gc` (appended — does not clobber
user-supplied NODE_OPTIONS). Covers both `node dist/entry.js` and
`tsx src/entry.tsx` launch paths.
- ui-tui/src/entry.tsx: shebang rewritten to
`#!/usr/bin/env -S node --max-old-space-size=8192 --expose-gc` as a
fallback when the binary is invoked directly.
### GatewayClient (ui-tui/src/gatewayClient.ts)
- `setMaxListeners(0)` — silences spurious warnings from React hook
subscribers.
- `logs` and `bufferedEvents` replaced with fixed-capacity
CircularBuffer — O(1) push, no splice(0, …) copies under load.
- RPC timeout refactor: `setTimeout(this.onTimeout.bind(this), …, id)`
replaces the inline arrow closure that captured `method`/`params`/
`resolve`/`reject` for the full 120 s request timeout. Each Pending
record now stores its own timeout handle, `.unref()`'d so stuck
timers never keep the event loop alive, and `rejectPending()` clears
them (previously leaked the timer itself).
### Memory diagnostics (new)
- ui-tui/src/lib/memory.ts: `performHeapDump()` +
`captureMemoryDiagnostics()`. Writes heap snapshot + JSON diag
sidecar to `~/.hermes/heapdumps/` (override via
`HERMES_HEAPDUMP_DIR`). Diagnostics are written first so we still get
useful data if the snapshot crashes on very large heaps.
Captures: detached V8 contexts (closure-leak signal), active
handles/requests (`process._getActiveHandles/_getActiveRequests`),
Linux `/proc/self/fd` count + `/proc/self/smaps_rollup`, heap growth
rate (MB/hr), and auto-classifies likely leak sources.
- ui-tui/src/lib/memoryMonitor.ts: 10 s interval polling heapUsed. At
1.5 GB writes an auto heap dump (trigger=`auto-high`); at 2.5 GB
writes a final dump and exits 137 before V8 fatal-OOMs so the user
can restart cleanly. Handle is `.unref()`'d so it never holds the
process open.
### Graceful exit (new)
- ui-tui/src/lib/gracefulExit.ts: SIGINT/SIGTERM/SIGHUP run registered
cleanups through a 4 s failsafe `setTimeout` that hard-exits if
cleanup hangs.
`uncaughtException` / `unhandledRejection` are logged to stderr
instead of crashing — a transient TUI render error should not kill
an in-flight agent turn.
### Slash commands (new)
- ui-tui/src/app/slash/commands/debug.ts:
- `/heapdump` — manual snapshot + diagnostics.
- `/mem` — live heap / rss / external / array-buffer / uptime panel.
- Registered in `ui-tui/src/app/slash/registry.ts`.
### Utility (new)
- ui-tui/src/lib/circularBuffer.ts: small fixed-capacity ring buffer
with `push` / `tail(n)` / `drain()` / `clear()`. Replaces the ad-hoc
`array.splice(0, len - MAX)` pattern.
## Validation
- tsc `--noEmit` clean
- `vitest run`: 15 files, 102 tests passing
- eslint clean on all touched/new files
- build produces executable `dist/entry.js` with preserved shebang
- smoke-tested: `HERMES_HEAPDUMP_DIR=… performHeapDump('manual')`
writes both a valid `.heapsnapshot` and a `.diagnostics.json`
containing detached-contexts, active-handles, smaps_rollup.
## Env knobs
- `HERMES_HEAPDUMP_DIR` — override snapshot output dir
- `HERMES_HEAPDUMP_ON_START=1` — dump once at boot
- existing `NODE_OPTIONS` is respected and appended, not replaced
Guards four unbounded growth paths reachable at idle — the shape matches
reports of the TUI hitting V8's 2GB heap limit after ~1m of idle with 0
tokens used (Mark-Compact freed ~6MB of 2045MB → pure retention).
- `GatewayClient.logs` + `gateway.stderr` events: 200-line cap is bytes-
uncapped; a chatty Python child emitting multi-MB lines (traceback,
dumped config, unsplit JSON) retains everything. Truncate at 4KB/line.
- `GatewayClient.bufferedEvents`: unbounded until `drain()` fires. Cap
at 2000 so a pre-mount event storm can't pin memory indefinitely.
- `useMainApp` gateway `exit` handler: didn't reset `turnController`, so
a mid-stream crash left `bufRef`/`reasoningText` alive forever.
- `pasteSnips` count-capped (32) but byte-uncapped. Add a 4MB total cap
and clear snips in `clearIn` so submitted pastes don't linger.
- `StylePool.transitionCache`: uncapped `Map<number,string>`. Full-clear
at 32k entries (mirrors `charCache` pattern).
- Switch tsconfig to nodenext module resolution for Node 22 (used by
installer script)
- Add shebang to entry.tsx, preserved into index.js
- Add HERMES_ROOT env var fallback for repo root resolution
