Chaos (5 scenarios x 2 UIs): every gateway-death/hang scenario fully
recovers on BOTH UIs — Ink respawns immediately (~80ms, no backoff),
OpenTUI after its 1s backoff; transcripts converge byte-identical to
the never-killed digest; zero orphans. PTY EOF: both exit and reap the
gateway in ~100ms (Ink takes 4.1s to die vs OpenTUI 0.2s).
Pipeline (800 msgs @30ev/s inside a dedicated tmux server): UI CPU
82.4s (ink) vs 79.1s (otui); tmux-server leg ~0.4s BOTH — the
'Ink costs more in the emulator' hypothesis is not supported at this
workload. Frame pacing: otui 22.3fps vs ink 15.8fps, interframe p95
103ms vs 209ms. Echo latency: both excellent (p50 1-2ms); submit to
first-token-paint 44ms (ink) vs 107ms (otui).
Reconstructs what killed gateways and TUI sessions: merges
~/.hermes/logs, journalctl/dmesg OOM kills, sessions-DB abnormal ends,
and git worktree state into one timestamp-sorted timeline plus a
summary (OOM victims, tui_gateway exit-reason histogram, orphaned
sessions). Findings from the 2026-06-04..11 window: gateway deaths were
kernel OOM kills selecting hermes via OOMScoreAdjust=200 under pressure
from OTHER tools' multi-GB leaks; TUI deaths were top-down SIGHUP/EIO
cascades from unit teardown; tui_gateway itself crashed in 0/152 exits.
Real sessions DB (~/.hermes/state.db, 444 tui+cli sessions): p50=20,
p75=53, p90=182, p95=340, p99=1941 msgs. The assumed 200-300 'realistic
band' is actually the p90-p95 region; the typical session is ~20 msgs
and the p99 tail reaches ~1940 (real ~2k-msg sessions exist).
New cells at those anchors (2 reps, ink vs otui-capped, 2G scope), VmHWM
medians: 100 msgs 163 vs 222MB; 300 msgs 180 vs 268MB; 2000 msgs 234 vs
671MB (2.9x). session-distribution.mjs regenerates the JSON; run.mjs
gains --configs to skip redundant configs (otui-uncapped == capped below
the 3000-row cap).
Three behavior changes to the hermes -w worktree lifecycle:
1. Git-native locks. _setup_worktree now locks its worktree
(git worktree lock --reason "hermes session pid=<pid>"), and
_prune_stale_worktrees skips locked worktrees at ANY age — a lock
from a live or crashed session means "do not touch". New helpers
_lock_worktree / _unlock_worktree / _worktree_is_locked (fail-safe:
any error reads as locked) / _worktree_is_dirty (fail-safe: any
error reads as dirty).
2. Dirty trees are preserved. _cleanup_worktree previously destroyed
worktrees with uncommitted changes if there were no unpushed
commits; it now keeps the worktree, branch, and lock when the tree
is dirty OR has unpushed commits, and prints manual cleanup hints
(git worktree unlock + remove --force). The >72h "force remove
regardless" prune tier is removed: pruning may only ever delete
clean, unlocked, fully-pushed worktrees.
3. Branch deletion is gated on removal success. Both cleanup and
prune previously deleted the branch without checking the
git worktree remove returncode, dropping easy reachability of the
commits even when removal failed; the branch is now only deleted
after a successful remove.
Re-ran the cells that crashed, against the fixed binary (a939c9a):
- mem3000 otui-capped: crashed_after_stream exit 7 @ ~900MB
→ completed exit 0, vmhwm 859MB
- mem3000 otui-uncapped: crashed_after_stream exit 7
→ completed exit 0, vmhwm 834MB
- slope10000 otui-uncapped: died exit 7 at 3,000 msgs (197d499 result)
→ completed ALL 10,000 msgs, exit 0, vmhwm 1.57GB —
under the 2GB cgroup cap, no cap-hit, no crash
- fresh ink baselines for both cells (mem3000 257MB / slope10k 328MB).
No "Failed to create SyntaxStyle" anywhere; the store cap now binds at the
handle-safe ceiling (1000 rows) long before the native 65,534-slot handle
table exhausts. report.html + report-assets regenerated via render.mjs
(results are append-only; pre-fix runs remain as the baseline).
Root cause of the bench-suite crash (every otui mem3000/slope cell died at
~3000 lumpy fixture msgs, exit 7, ~880MB RSS — not a cgroup kill):
- @opentui/core 0.4.0 routes EVERY native object through ONE global handle
registry with 16-bit slot indices (core src/zig/handles.zig: INDEX_BITS=16,
MAX_SLOTS=65535, slot 0 reserved). Measured on this install: exactly 65,534
live handles; the next createSyntaxStyle() fails. destroy() DOES recycle
slots — exhaustion means LIVE objects.
- Every TextBufferRenderable burns THREE slots in its constructor
(TextBufferRenderable.ts:77-80: TextBuffer + TextBufferView + SyntaxStyle),
so the mount-everything transcript hits the wall at ~1,400 store rows
(~16 text renderables/row x 3 ~ 47 handles/row): "Failed to create
SyntaxStyle" (zig.ts:4554) throws out of a Solid mount effect.
- The crash was MASKED: CliRenderer's own uncaughtException handler
(handleError -> console.show()) allocates the console-overlay
OptimizedBuffer — another handle — so the handler itself threw "Failed to
create optimized buffer: WxH" and Node died with exit 7 (fatal error in
the uncaughtException handler), hiding the real error.
Why not share one SyntaxStyle (the obvious 3->2): the per-buffer style is
load-bearing — native setStyledText (text-buffer.zig) registers each chunk's
color by NAME ("chunk{i}") into the buffer's OWN style, and registration is
name-keyed-overwrite (syntax-style.zig putStyle), so a shared style would
cross-corrupt chunk colors between every styled <text>. Pooling is unsound
at our layer in core 0.4.0.
The fix, at the seams that are ours:
- boundary/nativeHandles.ts (ffiSafe.ts sibling): SyntaxStyle.create() on a
full table DEGRADES to a detached style (native handle 0) instead of
throwing — JS-side styleDefs/mergeStyles (what markdown/code chunk colors
actually use) keep working; all native calls on handle 0 are inert no-ops.
- boundary/renderer.ts: guard the process error listeners createCliRenderer
installs so an exception INSIDE the handler can never exit-7-mask the
original error again (logged honestly; original error stays the story).
- logic/store.ts: HERMES_TUI_MAX_MESSAGES clamped to a handle-safe ceiling
(1000 rows ~ 47k handles ~ 72% of the table on the realistic fixture).
The old default of 3000 was unreachable — the TUI crashed at ~1,400 rows,
before the cap ever bound. Renderable-weight-aware capping is #27's
(virtualization) to do properly; until then the degrade shim backstops
pathological rows.
TODO(upstream) — issue-shaped, for the OpenTUI repo:
(a) a global 64k handle table with a 3-slot cost per text renderable is
too small for transcript-style TUIs (61k renderables ~ 3k messages);
(b) native allocation failures throw out of the render loop with no
degrade path;
(c) handleError allocates (console overlay buffer) and so crashes on the
very condition it is reporting, masking the root cause with exit 7.
Also: eslint now ignores ui-opentui/.bench/** (bench `nodes`-cell build
artifact broke the lint gate) and .gitignore covers it.
Gate: npm run check green, 599 tests (595 baseline + 3 degrade-path tests
+ 1 cap-clamp test).
The OpenTUI /usage went through the slash-worker subprocess, which
resumes the session WITHOUT a live agent — so it could never show
current-session tokens or costs, and what it did show landed as a
full-screen page.
- slash.exec now answers /usage in-process from the live agent:
per-model rows (requests, tokens in/out, cache, provider-reported
cost when present), session totals/context, a one-line 30-day
summary (SessionDB.usage_totals, real costs only) and a one-line
Nous credits gauge (nous_credits_compact_line, refactored out of
nous_credits_lines). ~8 lines instead of a page.
- Unreported costs render as 'not reported by provider' — never
$0.00 — and the 30d summary omits cost when no session in the
window has a provider-reported figure.
- /usage full keeps the detailed legacy CLI page via the worker.
Cost displays were estimates from a pricing table; on OpenRouter the
status bar never reflected what was actually charged. Now cost is
provider-REPORTED only, end to end:
- OpenRouter requests carry usage:{include:true} (profile + legacy
transport paths); the response usage.cost field (credits, 1:1 USD)
is captured per call into agent.session_actual_cost_usd and
persisted to the sessions DB actual_cost_usd column (NULL-safe:
unreported calls never touch the stored value).
- Nous keeps its x-nous-credits-* header capture; the header delta
now surfaces as the session's real cost via real_session_cost_usd.
- Providers that report nothing accumulate NOTHING: cost fields stay
absent/None (the TUI hides its cost segment), never a fabricated
$0.00 and never an estimate. _get_usage, gateway /usage and the
CLI usage page all switched off estimate_usage_cost for display.
- Per-model session accumulator (session_model_usage) records real
per-call counts and provider-reported cost per model.
User feedback: tool/thinking rows did a "v small quick lil jump up and
down" when toggled, worst on the bottom rows.
Root cause (verified live with 10ms tmux capture sampling): the
transcript scrollbox's sticky-bottom re-pin and the scroll anchor fought
AFTER paint. On a toggle near the bottom, the content-height change runs
ScrollBox.recalculateBarProps -> applyStickyStart("bottom") (the user is
at the sticky position, so _hasManualScroll is false), which paints a
fully bottom-pinned frame; the anchor's 4x16ms scrollTo re-asserts then
yanked the viewport back up. The capture burst shows the transient
pinned frame between two anchored ones on every expand — the visible
down-up flick.
Fix at the cause instead of correcting after the effect: suspend
stickyScroll (a runtime get/set property on ScrollBoxRenderable) BEFORE
running the toggle and restore it ~100ms later, once the content height
has settled. With sticky off, the toggle's layout pass leaves scrollTop
untouched — the clicked header's document position is unchanged (content
grows/shrinks below it), so nothing moves and there is nothing left to
flicker; a collapse past the new bottom clamps naturally via the
ScrollBar scrollSize setter. Restoring recomputes the manual-scroll
state from the actual position: still at the bottom -> keeps pinning for
new content; mid-content -> manual-scroll semantics until the user
returns (the same end state the old anchor produced). Rapid re-toggles
inside the window keep the ORIGINAL saved value.
The far-from-bottom anchor guarantee is unchanged (scrollTop is simply
never touched), pinned headlessly in scrollAnchor.test.tsx along with
the suspension sequencing, the clamp-then-re-pin collapse path, and the
double-toggle restore. ffiSafe's tall-diff scroll-cut regression now
drives the negative-y condition explicitly via wheel scrolls (the old
anchor exercised it through the very transient sticky-bottom frames this
fix removes).
Verified live (tmux, real gateway): before — toggling the bottom rows
painted a transient bottom-pinned frame (f141 of a 10ms burst); after —
three toggle bursts produce ONLY the clean before/after states (4
distinct frames in 458 samples), headers hold their row, including the
bottom-most rows.
User feedback: "for all tools, i'd want all their output viewing enabled
to be infinite by default."
Flip envOutputLines (HERMES_TUI_TOOL_OUTPUT_LINES): unset -> Infinity
(was 200); a positive integer RESTORES a cap (e.g. =200); 0 stays
Infinity for back-compat with the old opt-in-unlimited value; garbage ->
Infinity (unrecognized = no cap asked for). The semantic is now "cap
only when the user asked for one".
The store's raw-result preference follows the same rule: envOutputLinesSet
becomes envOutputUnlimited — whenever the cap is unlimited (the default
now) and a gateway tail-capped result_text (omittedNote) arrives with the
always-full raw result on the wire, the raw result wins, since an
uncapped view of a tail would silently miss the head. With an explicit
finite cap the gateway tail + honest omitted note are kept.
Memory safety is unchanged: tool bodies mount only while EXPANDED (rows
default collapsed and free their Yoga nodes on collapse/unmount), and the
rolling HERMES_TUI_MAX_MESSAGES cap bounds the transcript's high-water
mark.
Tests: env.test.ts expectations flipped (unset/garbage -> Infinity, 0
documented as back-compat); tools.test.tsx "flag unset caps at 200"
becomes "unset renders all 250 lines", plus an explicit =50 cap (+note)
test and =200 restored-cap test; the store preference matrix covers
unset/0 (raw wins), =50 (tail+note kept), and no-raw (tail+note, no
crash). Verified live: seq 1 220 expanded renders rows 201-220 with no
"+N more lines" note.
The native TUI prefetches model.options right after session.create (91df32545,
picker instant-open). The handler is network-bound (~3.7s: pricing fetch + Nous
tier check in build_models_payload) and ran on the gateway's main dispatcher
thread, so every fast-path RPC issued in the first seconds after launch —
complete.slash for the '/' dropdown, session.list, config.get — sat unread
behind it. Measured: first '/' dropdown 1718ms at HEAD vs 53ms at 394f45a3d
(pre-prefetch baseline); 52ms after routing model.options onto the existing
RPC thread pool (_LONG_HANDLERS). The /model picker keeps its 29ms cached open.
