Three behavior changes to the hermes -w worktree lifecycle:
1. Git-native locks. _setup_worktree now locks its worktree
(git worktree lock --reason "hermes session pid=<pid>"), and
_prune_stale_worktrees skips locked worktrees at ANY age — a lock
from a live or crashed session means "do not touch". New helpers
_lock_worktree / _unlock_worktree / _worktree_is_locked (fail-safe:
any error reads as locked) / _worktree_is_dirty (fail-safe: any
error reads as dirty).
2. Dirty trees are preserved. _cleanup_worktree previously destroyed
worktrees with uncommitted changes if there were no unpushed
commits; it now keeps the worktree, branch, and lock when the tree
is dirty OR has unpushed commits, and prints manual cleanup hints
(git worktree unlock + remove --force). The >72h "force remove
regardless" prune tier is removed: pruning may only ever delete
clean, unlocked, fully-pushed worktrees.
3. Branch deletion is gated on removal success. Both cleanup and
prune previously deleted the branch without checking the
git worktree remove returncode, dropping easy reachability of the
commits even when removal failed; the branch is now only deleted
after a successful remove.
Re-ran the cells that crashed, against the fixed binary (a939c9a):
- mem3000 otui-capped: crashed_after_stream exit 7 @ ~900MB
→ completed exit 0, vmhwm 859MB
- mem3000 otui-uncapped: crashed_after_stream exit 7
→ completed exit 0, vmhwm 834MB
- slope10000 otui-uncapped: died exit 7 at 3,000 msgs (197d499 result)
→ completed ALL 10,000 msgs, exit 0, vmhwm 1.57GB —
under the 2GB cgroup cap, no cap-hit, no crash
- fresh ink baselines for both cells (mem3000 257MB / slope10k 328MB).
No "Failed to create SyntaxStyle" anywhere; the store cap now binds at the
handle-safe ceiling (1000 rows) long before the native 65,534-slot handle
table exhausts. report.html + report-assets regenerated via render.mjs
(results are append-only; pre-fix runs remain as the baseline).
Root cause of the bench-suite crash (every otui mem3000/slope cell died at
~3000 lumpy fixture msgs, exit 7, ~880MB RSS — not a cgroup kill):
- @opentui/core 0.4.0 routes EVERY native object through ONE global handle
registry with 16-bit slot indices (core src/zig/handles.zig: INDEX_BITS=16,
MAX_SLOTS=65535, slot 0 reserved). Measured on this install: exactly 65,534
live handles; the next createSyntaxStyle() fails. destroy() DOES recycle
slots — exhaustion means LIVE objects.
- Every TextBufferRenderable burns THREE slots in its constructor
(TextBufferRenderable.ts:77-80: TextBuffer + TextBufferView + SyntaxStyle),
so the mount-everything transcript hits the wall at ~1,400 store rows
(~16 text renderables/row x 3 ~ 47 handles/row): "Failed to create
SyntaxStyle" (zig.ts:4554) throws out of a Solid mount effect.
- The crash was MASKED: CliRenderer's own uncaughtException handler
(handleError -> console.show()) allocates the console-overlay
OptimizedBuffer — another handle — so the handler itself threw "Failed to
create optimized buffer: WxH" and Node died with exit 7 (fatal error in
the uncaughtException handler), hiding the real error.
Why not share one SyntaxStyle (the obvious 3->2): the per-buffer style is
load-bearing — native setStyledText (text-buffer.zig) registers each chunk's
color by NAME ("chunk{i}") into the buffer's OWN style, and registration is
name-keyed-overwrite (syntax-style.zig putStyle), so a shared style would
cross-corrupt chunk colors between every styled <text>. Pooling is unsound
at our layer in core 0.4.0.
The fix, at the seams that are ours:
- boundary/nativeHandles.ts (ffiSafe.ts sibling): SyntaxStyle.create() on a
full table DEGRADES to a detached style (native handle 0) instead of
throwing — JS-side styleDefs/mergeStyles (what markdown/code chunk colors
actually use) keep working; all native calls on handle 0 are inert no-ops.
- boundary/renderer.ts: guard the process error listeners createCliRenderer
installs so an exception INSIDE the handler can never exit-7-mask the
original error again (logged honestly; original error stays the story).
- logic/store.ts: HERMES_TUI_MAX_MESSAGES clamped to a handle-safe ceiling
(1000 rows ~ 47k handles ~ 72% of the table on the realistic fixture).
The old default of 3000 was unreachable — the TUI crashed at ~1,400 rows,
before the cap ever bound. Renderable-weight-aware capping is #27's
(virtualization) to do properly; until then the degrade shim backstops
pathological rows.
TODO(upstream) — issue-shaped, for the OpenTUI repo:
(a) a global 64k handle table with a 3-slot cost per text renderable is
too small for transcript-style TUIs (61k renderables ~ 3k messages);
(b) native allocation failures throw out of the render loop with no
degrade path;
(c) handleError allocates (console overlay buffer) and so crashes on the
very condition it is reporting, masking the root cause with exit 7.
Also: eslint now ignores ui-opentui/.bench/** (bench `nodes`-cell build
artifact broke the lint gate) and .gitignore covers it.
Gate: npm run check green, 599 tests (595 baseline + 3 degrade-path tests
+ 1 cap-clamp test).
The OpenTUI /usage went through the slash-worker subprocess, which
resumes the session WITHOUT a live agent — so it could never show
current-session tokens or costs, and what it did show landed as a
full-screen page.
- slash.exec now answers /usage in-process from the live agent:
per-model rows (requests, tokens in/out, cache, provider-reported
cost when present), session totals/context, a one-line 30-day
summary (SessionDB.usage_totals, real costs only) and a one-line
Nous credits gauge (nous_credits_compact_line, refactored out of
nous_credits_lines). ~8 lines instead of a page.
- Unreported costs render as 'not reported by provider' — never
$0.00 — and the 30d summary omits cost when no session in the
window has a provider-reported figure.
- /usage full keeps the detailed legacy CLI page via the worker.
Cost displays were estimates from a pricing table; on OpenRouter the
status bar never reflected what was actually charged. Now cost is
provider-REPORTED only, end to end:
- OpenRouter requests carry usage:{include:true} (profile + legacy
transport paths); the response usage.cost field (credits, 1:1 USD)
is captured per call into agent.session_actual_cost_usd and
persisted to the sessions DB actual_cost_usd column (NULL-safe:
unreported calls never touch the stored value).
- Nous keeps its x-nous-credits-* header capture; the header delta
now surfaces as the session's real cost via real_session_cost_usd.
- Providers that report nothing accumulate NOTHING: cost fields stay
absent/None (the TUI hides its cost segment), never a fabricated
$0.00 and never an estimate. _get_usage, gateway /usage and the
CLI usage page all switched off estimate_usage_cost for display.
- Per-model session accumulator (session_model_usage) records real
per-call counts and provider-reported cost per model.
User feedback: tool/thinking rows did a "v small quick lil jump up and
down" when toggled, worst on the bottom rows.
Root cause (verified live with 10ms tmux capture sampling): the
transcript scrollbox's sticky-bottom re-pin and the scroll anchor fought
AFTER paint. On a toggle near the bottom, the content-height change runs
ScrollBox.recalculateBarProps -> applyStickyStart("bottom") (the user is
at the sticky position, so _hasManualScroll is false), which paints a
fully bottom-pinned frame; the anchor's 4x16ms scrollTo re-asserts then
yanked the viewport back up. The capture burst shows the transient
pinned frame between two anchored ones on every expand — the visible
down-up flick.
Fix at the cause instead of correcting after the effect: suspend
stickyScroll (a runtime get/set property on ScrollBoxRenderable) BEFORE
running the toggle and restore it ~100ms later, once the content height
has settled. With sticky off, the toggle's layout pass leaves scrollTop
untouched — the clicked header's document position is unchanged (content
grows/shrinks below it), so nothing moves and there is nothing left to
flicker; a collapse past the new bottom clamps naturally via the
ScrollBar scrollSize setter. Restoring recomputes the manual-scroll
state from the actual position: still at the bottom -> keeps pinning for
new content; mid-content -> manual-scroll semantics until the user
returns (the same end state the old anchor produced). Rapid re-toggles
inside the window keep the ORIGINAL saved value.
The far-from-bottom anchor guarantee is unchanged (scrollTop is simply
never touched), pinned headlessly in scrollAnchor.test.tsx along with
the suspension sequencing, the clamp-then-re-pin collapse path, and the
double-toggle restore. ffiSafe's tall-diff scroll-cut regression now
drives the negative-y condition explicitly via wheel scrolls (the old
anchor exercised it through the very transient sticky-bottom frames this
fix removes).
Verified live (tmux, real gateway): before — toggling the bottom rows
painted a transient bottom-pinned frame (f141 of a 10ms burst); after —
three toggle bursts produce ONLY the clean before/after states (4
distinct frames in 458 samples), headers hold their row, including the
bottom-most rows.
User feedback: "for all tools, i'd want all their output viewing enabled
to be infinite by default."
Flip envOutputLines (HERMES_TUI_TOOL_OUTPUT_LINES): unset -> Infinity
(was 200); a positive integer RESTORES a cap (e.g. =200); 0 stays
Infinity for back-compat with the old opt-in-unlimited value; garbage ->
Infinity (unrecognized = no cap asked for). The semantic is now "cap
only when the user asked for one".
The store's raw-result preference follows the same rule: envOutputLinesSet
becomes envOutputUnlimited — whenever the cap is unlimited (the default
now) and a gateway tail-capped result_text (omittedNote) arrives with the
always-full raw result on the wire, the raw result wins, since an
uncapped view of a tail would silently miss the head. With an explicit
finite cap the gateway tail + honest omitted note are kept.
Memory safety is unchanged: tool bodies mount only while EXPANDED (rows
default collapsed and free their Yoga nodes on collapse/unmount), and the
rolling HERMES_TUI_MAX_MESSAGES cap bounds the transcript's high-water
mark.
Tests: env.test.ts expectations flipped (unset/garbage -> Infinity, 0
documented as back-compat); tools.test.tsx "flag unset caps at 200"
becomes "unset renders all 250 lines", plus an explicit =50 cap (+note)
test and =200 restored-cap test; the store preference matrix covers
unset/0 (raw wins), =50 (tail+note kept), and no-raw (tail+note, no
crash). Verified live: seq 1 220 expanded renders rows 201-220 with no
"+N more lines" note.
The native TUI prefetches model.options right after session.create (91df32545,
picker instant-open). The handler is network-bound (~3.7s: pricing fetch + Nous
tier check in build_models_payload) and ran on the gateway's main dispatcher
thread, so every fast-path RPC issued in the first seconds after launch —
complete.slash for the '/' dropdown, session.list, config.get — sat unread
behind it. Measured: first '/' dropdown 1718ms at HEAD vs 53ms at 394f45a3d
(pre-prefetch baseline); 52ms after routing model.options onto the existing
RPC thread pool (_LONG_HANDLERS). The /model picker keeps its 29ms cached open.
A patch tool's result is a JSON record whose payload IS the diff. In a verbose
session the gateway redacts + TAIL-caps result_text (_cap_tui_verbose_text),
so the echo arrived under the native diff in two broken shapes: truncated
mid-JSON (unparseable, so the old JSON.parse check failed open), or — for tall
edits — capped PAST the JSON head, which the store's normalizeOutput then
un-escapes into plain lines that duplicate the diff. North star: no raw JSON
in the transcript, ever.
Three layers:
- gateway: when diff_unified ships, result_text drops the in-JSON diff echo
(_result_sans_diff_echo) — small, parseable, carries only the non-diff
signal (success/files_modified/warnings/lsp_diagnostics).
- fileTool diffOutputPlan: anything starting with '{' under a rendered diff is
suppressed regardless of parseability; parseable JSON with real non-diff
signal (error/warning/lsp_diagnostics) renders JUST those as labeled notes;
a non-JSON fragment whose lines echo the rendered diff is suppressed too
(guards older emitters). Plain-text results (lint tails) still render.
Expanding a tall <diff showLineNumbers> pinned to the scrollbox bottom froze
the TUI with ERR_INVALID_ARG_VALUE looping out of CliRenderer.loop every
frame. Root cause: @opentui/core 0.4.0 marshals OptimizedBuffer
fillRect/drawText/setCell* coordinates as u32 in the FFI table while
LineNumberRenderable.renderSelf passes raw screen coordinates — NEGATIVE when
the diff is partially scrolled above the viewport. Bun's FFI silently wraps
negatives (native side bounds-checks them into a no-op); Node's experimental
node:ffi rejects them. bufferDrawBox already uses i32, which is why ordinary
boxes/text scroll fine and only the diff line-background path crashed.
Fix at the seam we own: boundary/ffiSafe.ts patches OptimizedBuffer to clip
fillRect to the non-negative quadrant and skip negative-origin
drawText/setCell*/drawChar before the FFI call (Bun parity). Installed from
boundary/renderer.ts (live) and test/lib/render.ts (headless). TODO(upstream):
widen those FFI params to i32 so this shim can be deleted.
