docs: the OpenTUI memory story — ELI5 walkthrough of the 686MB→300MB campaign

Shareable explainer: every primitive at play (native handle table, Yoga
WASM grow-only memory, renderables, Solid surgical unmount, V8 GC
laziness, scrollbox draw-only culling) and every decision (windowing vs
store-cull, exact-heights-at-unmount vs Ink's estimate-correct, the
correctionIsLegal zero-jank law, append-time adjudication, never-window
rules, windowing-aware cap restore, heap right-sizing), with the
measured scoreboard and honest open items.

docs/opentui-memory-story.md

@@ -0,0 +1,207 @@
+# How the OpenTUI transcript got from 686MB to ~300MB — the full story
+
+*For: glitch. Branch: `feat/opentui-memory-window`. Everything here is measured,
+not vibes; every number has a result JSON in `bench/results/`.*
+
+---
+
+## 1. The cast of characters (the primitives, bottom-up)
+
+To understand where the memory went, you need to know who's holding it. Six
+layers, from the screen up:
+
+**The terminal grid.** Your terminal is a spreadsheet of character cells.
+Nobody pays per-message here — tmux holds ~5MB flat no matter how long the
+session is (we measured). The terminal is never the problem.
+
+**The OpenTUI native renderer (Zig).** A compiled library that owns the
+"frame buffer" — the grid of cells about to be painted. Every piece of text the
+TUI shows lives in a native **TextBuffer** (the characters + their colors),
+viewed through a **TextBufferView**, styled by a **SyntaxStyle**. Each of those
+is a **native handle** — a ticket into one global table that has only **65,535
+slots, total, ever** (16-bit indices — like a coat check with 65k hooks).
+Destroying a renderable returns its tickets, so the constraint is not "how much
+have you ever created" but **"how much is alive right now."**
+
+**Renderables.** OpenTUI's UI objects — `<text>`, `<box>`, `<markdown>`,
+`<code>`, `<scrollbox>`. One transcript row (a message with its tool calls,
+markdown, code blocks, copy chips) is a *tree* of these: **~16 text renderables
+≈ 47 native handles ≈ ~250–340KB of RSS, per row.** This is the number that
+drives everything. 1,400 mounted rows × 47 handles = table full = the crash we
+root-caused last week.
+
+**Yoga (the layout engine, WASM).** Every renderable also has a Yoga node —
+Yoga is the flexbox calculator that decides where boxes go. OpenTUI ships it
+compiled to **WebAssembly**, and WASM has a brutal property: its memory can
+**grow but never shrink** back to the OS. So the peak number of
+*simultaneously-mounted* renderables sets a high-water mark you pay **forever**,
+even after everything is destroyed. (Fun fact from this week's forensics: we
+spent two days believing Ink had this disease. It doesn't — our Ink fork swapped
+Yoga-WASM for a plain TypeScript port at fork creation. **We** are the ones
+running layout in WASM. The accusation was true; we just had the defendant
+wrong.)
+
+**Solid (the view framework).** Renders each store message into a row via
+`<For>`. The property we exploit: Solid mounts/unmounts *surgically* — remove a
+row from what the component returns and Solid destroys exactly that row's
+renderables (returning its handles and freeing its Yoga nodes), touching
+nothing else. No virtual-DOM diffing, no collateral re-renders.
+
+**V8 (the JavaScript engine) + the store.** The store keeps every message as JS
+strings/objects. V8's garbage collector is *lazy by design*: with the default
+8GB ceiling we launch with, it sees no reason to clean up aggressively, so RSS
+includes a lot of "collectible but not yet collected" garbage. Cheap to fix,
+worth real MB (measured below).
+
+**The scrollbox.** One detail that fooled everyone at some point:
+`viewportCulling` (on by default) skips *drawing* offscreen rows — but they stay
+fully **mounted**: handles held, Yoga nodes alive, memory paid. Culling saves
+paint time, not memory. That misunderstanding is half the reason the "rolling
+store cap" was expected to be enough, and wasn't.
+
+## 2. Why it was 686MB
+
+Simple arithmetic. The old TUI mounted **every message in the store** as a full
+renderable tree. 2,000 messages × ~16 renderables × (handles + Yoga nodes +
+text buffers + V8 objects) ≈ 670–690MB, growing ~300MB per 1,000 messages. And
+at ~1,400 rows the handle table filled: first a hard crash (exit 7), then —
+after our containment fix — survival with **unstyled text** past that point,
+plus a cap clamped from 3,000 rows down to 1,000 as the price of not crashing.
+
+Ink, meanwhile, sat at ~234MB at the same workload, because Ink only ever
+mounts the rows near your viewport (~84–400 live nodes). Its memory is the
+*data* plus some caches — not the *view*.
+
+## 3. The decisions, in order
+
+### Decision 1: virtualize the view, don't starve the store
+
+Two ways to cut view memory: keep fewer messages (opencode's answer — they keep
+100 and delete the rest from memory; transcript truth lives on their server), or
+keep all messages but only *materialize* the ones near the viewport. You vetoed
+the first (your p90 session is 182 messages — a 100-row store truncates normal
+sessions), so: **windowing**. Notably the OpenTUI devs confirmed this week that
+framework-level virtualization is the intended path — the engine doesn't ship
+it out of the box, and opencode never built it. We did.
+
+### Decision 2: exact heights, recorded at unmount — never estimates in your face
+
+This is the load-bearing idea, and it's where we beat Ink at its own game.
+
+The hard problem of any virtualized list: an unmounted row still needs to
+occupy its correct *height*, or the scrollbar lies and content jumps. Ink
+solves it by **guessing** heights and correcting after measurement — those
+corrections are precisely the 83–101ms scroll stutters you hate. You explicitly
+vetoed "estimate-correction jank" as a model.
+
+Our advantage: OpenTUI lays out with real, queryable heights. So when a row
+scrolls out of the window, we record its **exact laid-out height** (an
+`onSizeChange` hook fires inside layout, pre-paint) and replace the row with an
+empty `<box height={exactly-that}/>` — a **spacer**: one Yoga node, zero text
+buffers, zero native handles. Think of a bookshelf where books you're not
+reading are swapped for cardboard sleeves cut to *exactly* the book's
+thickness: the shelf never shifts, and you can't tell from across the room.
+
+The window is your viewport ± one viewport of margin (plus hysteresis so it
+doesn't thrash at the edges). Scroll near a spacer and the real row remounts —
+at the recorded height, so nothing moves.
+
+And one **law**, written into the code as `correctionIsLegal`: a spacer's
+height may only ever be corrected where you *cannot see it* — fully above the
+viewport (with the scroll position compensated in the same frame, so the world
+doesn't move) or fully below it. A correction that would shift visible content
+is forbidden, structurally. Jank isn't tuned down; it's outlawed.
+
+### Decision 3 (the S2 insight): adjudicate on *append*, not just on scroll
+
+S1 alone got 686 → 518MB. Why not more? Because of *when* windowing decided.
+S1 re-decided the window when you **scrolled**. But during a streaming burst —
+an agent turn dumping hundreds of rows — you don't scroll; rows arrive, each
+mounting fully, and only get demoted later. That transient pile-up is mostly
+invisible in steady-state numbers… except for Yoga-WASM, where **the transient
+peak is permanent** (memory never shrinks). The burst was quietly ratcheting
+the floor.
+
+S2 makes the window recompute on **transcript growth**: while you're pinned at
+the bottom, the window anchors to the content *bottom*, so a row that falls
+more than a margin behind the live edge becomes a spacer the moment it's
+measured — not whenever you next scroll. Measured result: across a 1,500-row
+burst, the peak number of simultaneously-mounted rows is **31**.
+
+Same trick for **resume**: opening a 2,000-message session used to mount all of
+it (transient peak again — paid forever). Now resume mounts only the bottom
+window; everything above starts as spacers using a line-count estimate, and an
+idle-time "measure march" quietly mounts ten rows at a time near the window
+edge, records their true heights, and swaps them back — all outside the
+viewport, all invisible by the law above.
+
+### Decision 4: rows that must never be windowed
+
+Windowing has to know what it's not allowed to touch:
+- **Streaming rows** — the native markdown renderer streams incrementally;
+  unmounting mid-stream would restart it visibly.
+- **The bottom 30 rows** — the region you actually live in.
+- **Rows under a mouse selection** — the review caught that a lingering
+  highlight originally froze windowing *forever* (memory regrowing silently).
+  Fixed: only an active drag pauses swaps, and selected rows get pinned, so
+  copy is byte-exact while everything else keeps windowing.
+
+### Decision 5: give back the scrollback (cap 1,000 → 3,000)
+
+The 1,000-row clamp existed only because mounted-rows == stored-rows and the
+handle table dies at ~1,400. With windowing, mounted ≈ 31 regardless of store
+size — so the cap went back to the originally-shipped 3,000. It's
+windowing-aware: the `HERMES_TUI_WINDOWING=0` escape hatch (which mounts
+everything again) keeps the safe 1,000.
+
+### Decision 6 (measured, not yet shipped as default): right-size the V8 heap
+
+Running the windowed TUI with a 512MB heap ceiling instead of 8GB forced V8 to
+actually collect: another −90MB with zero latency cost. That's queued as a
+launcher default change (~1GB), for both engines.
+
+## 4. The scoreboard
+
+At 2,000 messages (your real p99 session size — yes, we checked your DB:
+median session is 20 messages, p99 is 1,941):
+
+| | peak memory | scroll p99 (slowest 1-in-100) |
+|---|---|---|
+| OpenTUI before | 686MB | 16ms |
+| + S1 windowing | 518MB | 16ms |
+| + S2 append/resume windowing | **300–375MB** | **6ms** |
+| Ink (reference) | 229–246MB | ~100ms |
+
+At the **3,000-message stress** with the restored triple-size scrollback:
+**360MB, fully styled, scroll p99 8ms** — a workload that six days ago crashed
+the process, and three days ago survived only by dropping syntax colors.
+
+Scroll got *faster* because there are simply fewer live renderables to walk.
+The determinism gate stayed **byte-identical** — the windowed TUI's settled
+frame is provably the same pixels as before. And the live smoke (2,000-message
+session: full sweep to the top, resize storm, back to bottom) returned a frame
+pixel-identical to boot, with deep history fully syntax-highlighted — something
+the pre-windowing TUI literally could not do.
+
+## 5. What's honestly still open
+
+- The remaining ~60–120MB over Ink is mostly the **store's JS strings** and
+  process baseline — the view is no longer the problem. The structural fix is
+  the **thin renderer** (W1): bodies live in the Python gateway (which already
+  has them in SQLite); the TUI keeps ~300-byte stubs and fetches bodies only
+  for the window. That also fixes the class of problem neither engine handles
+  today: a single 10MB tool output.
+- Two accepted, documented limits: scrollbar-*jumping* deep into a freshly
+  resumed session can land on estimate-height rows that snap to true height as
+  they enter view (normal scrolling doesn't — the margin pre-measures; the idle
+  march erodes the exposure over time), and a tool you expanded, scrolled far
+  away from, then returned to will have re-collapsed (state is component-local;
+  hoisting it to the store is queued).
+- Everything is behind `HERMES_TUI_WINDOWING` (default on, `0` = bit-exact old
+  behavior) — a one-env escape hatch if anything feels off in real use.
+
+*Where to verify: `bench/results/` (every number above), the design+gates doc
+`docs/plans/opentui-transcript-windowing.md`, tests in
+`ui-opentui/src/test/window.test.ts` and `transcriptWindow.test.tsx` (the
+zero-jank invariants are literal assertions: identical scrollHeight windowed
+vs not, byte-stable frames across corrections).*