diff --git a/docs/opentui-native-engine.md b/docs/opentui-native-engine.md new file mode 100644 index 00000000000..783b2a8b35e --- /dev/null +++ b/docs/opentui-native-engine.md @@ -0,0 +1,432 @@ +# OpenTUI native engine — PR documentation + +**Branch:** `feat/opentui-native-engine` · **Base:** `origin/main` (merged in; HEAD is at `~main`) +**New engine root:** `ui-opentui/` (Node 26 + `@opentui/core` 0.4.1 + `@opentui/solid`, Effect at the boundary) +**Legacy engine root:** `ui-tui/` (React + the `@hermes/ink` fork at `ui-tui/packages/hermes-ink/`) + +> This is the canonical in-repo doc for the PR. The companion interactive HTML +> write-up (`~/projects/opentui-perf-writeup/index.html`) is the case/benchmark +> deep-dive; this doc is the reviewable text version + the four things review +> actually needs: **(1) the LoC reduction math, (2) the measured perf deltas, +> (3) the real UI divergence (with screenshots), (4) the non-core / kitchen-sink +> change audit.** + +This PR adds a from-scratch native terminal UI built on OpenTUI, intended to +replace the React/Ink TUI **and the Ink fork we maintain alone**. It currently +ships as a parallel engine (Ink untouched, auto-fallback), selected by +`HERMES_TUI_ENGINE` env > `display.tui_engine` config > auto (OpenTUI when the +host is Node ≥ 26.3 with the built bundle, else Ink). **100% parity with the Ink +TUI is the bar.** + +--- + +## 1. Line-of-code reduction (the headline maintenance win) + +All counts are **git-tracked files only** (respects `.gitignore`; `dist/` and +`node_modules/` are untracked and excluded). Measured live on this branch at +`~HEAD`. "Code" = `.ts/.tsx/.js/.jsx` only; "total" includes config/json/md. + +### What gets *removed* when Ink is retired + +| Area | Files | Total lines | Code lines (ts/tsx/js) | Non-blank code | +|---|---:|---:|---:|---:| +| `ui-tui/src/` — Ink **consumer app** (our React/Ink view code) | 204 | 40,422 | 40,422 | 33,550 | +| `ui-tui/packages/hermes-ink/` — **the fork** (`@hermes/ink`) | 148 | 28,167 | 28,113 | 23,718 | +| **`ui-tui/` whole tree (tracked)** | **362** | **69,320** | **68,831** | **57,545** | + +The `ui-tui/` whole-tree number (69,320) also folds in a handful of build +scripts, `.prettierrc`, `package.json`, etc. The two rows above it are the +load-bearing split: + +- **The fork alone is 28,167 LOC across 148 files** — code we own and can never + sync from upstream. Upstream Ink v6.8.0 `src/` is ~7,259 LOC, so the fork's + renderer core is **~3.2× the size of stock Ink**. (Cross-checked against the + HTML write-up's `ink-fork-analysis.json`: 28,111 LOC / 148 files — the 56-line + delta is a single tracked JSON the file-level count includes.) +- **The consumer app is another 40,422 LOC** — React components/hooks that only + exist to drive Ink. + +### What gets *added* + +| Area | Files | Total lines | Code lines | Non-blank code | +|---|---:|---:|---:|---:| +| `ui-opentui/src/` — new engine (app code **+ its own tests**) | 153 | 28,763 | 28,763 | 26,495 | +|   ↳ non-test (app code only) | 97 | 16,628 | 16,628 | 15,450 | +|   ↳ tests (`src/test/`) | 56 | 12,135 | 12,135 | 11,045 | +| Tree-sitter grammars (`python`…`toml`) | 0 | 0 | 0 | 0 | +| **`ui-opentui/` whole tree (tracked)** | **~170** | **~34,800** | **29,614** | **27,283** | + +> Tree-sitter grammars carry **zero repo lines**: the engine declares the 10 +> extra grammars as remote URLs (`src/boundary/parsers.manifest.json`) and +> OpenTUI fetches+caches each `.wasm`/`.scm` on first use into +> `~/.hermes/cache/opentui-parsers/` (à la opencode, which vendors none). An +> earlier revision vendored them as 37,302 checked-in binary lines (10 `.wasm` + +> 10 `.scm`); that's gone — code lines and total lines now move together. + +### The net reduction (code lines, the honest comparison) + +| Comparison | Removed (ts/tsx/js) | Added (ts/tsx/js) | Net change | +|---|---:|---:|---:| +| **Incl. fork** — retire all of `ui-tui/` vs add `ui-opentui/src` | −68,831 | +28,763 | **−40,068 LOC (−58%)** | +| **Incl. fork, app-vs-app** (exclude both test suites) | −56,463¹ | +16,628 | **−39,835 LOC (−71%)** | +| **Excl. fork** — only the Ink *consumer app* vs new engine | −40,422 | +28,763 | **−11,659 LOC (−29%)** | +| **The fork in isolation** (the unsyncable liability we shed) | −28,113 | — | **−28,113 code lines deleted outright (28,167 incl. its 1 config file)** | + +¹ `ui-tui/src` non-test = 28,350 LOC + fork (≈ all 28,113 code lines are non-test; +it carries only ~54 config lines) = 56,463. (`ui-tui/src` carries 80 test files / +12,072 LOC; the new engine carries 56 test files / 12,135 LOC.) + +**Read it this way:** + +- **The cleanest single number: ~−40k code lines net** (retire all of `ui-tui/`, + add `ui-opentui/src`). That is a **~58% reduction in the TUI's + hand-maintained surface**, and it *includes* the new engine's full 56-file test + suite. +- **The most important number is the fork: −28,167 LOC of unsyncable engine + code** disappears. That is the load-bearing maintenance win — it's not just + fewer lines, it's lines we are the *sole* maintainer of (own reconciler, ANSI + parser, scrollbox, selection/OSC52, hand-rolled memory eviction, Yoga binding). +- **Even excluding the fork** — i.e. if you imagine upstream Ink were free — the + app rewrite is still a net reduction (−11,659 LOC) because the new engine + mounts OpenTUI built-ins instead of hand-building components. + +### Caveat on the comparison (keep it honest for review) + +- These are **whole-tree retirements vs a single source dir add.** If/when Ink is + deleted, the `ui-tui/` `package.json`, lockfile, and build scripts go too; the + table counts `ui-tui/src` + the fork as the apples-to-apples "hand-maintained + TS" figure. +- **Tree-sitter grammars are NOT vendored.** The 10 extra grammars are declared + as remote URLs (`src/boundary/parsers.manifest.json`); OpenTUI fetches each + `.wasm`/`.scm` on first use of a language and caches it under + `~/.hermes/cache/opentui-parsers/` (profile-aware, set via + `HERMES_TUI_PARSER_CACHE` by the launcher). Registration does **zero** network; + the fetch is lazy and off the boot critical path, and an unreachable + GitHub/air-gapped env degrades that language to plain text — never a throw. This + replaces an earlier revision that vendored 37k binary lines, so the repo no + longer grows on disk for syntax highlighting. (Trade-off: first-use-per-language + needs network to `github.com`/`raw.githubusercontent.com`; pre-seed the cache in + a Docker build if you need offline highlighting.) +- Python/backend LoC is **not** part of this reduction: `tui_gateway/` (~12k LOC) + is **shared by both engines** and stays. See §4. + +--- + +## 2. Performance (CPU / latency / memory) + +Measured with the `tui-bench` harness driving **both engines on a real PTY +120×40**, fake gateway feeding deterministic events, `/proc`-sampled identically, +each SUT under `systemd-run --scope -p MemoryMax=2G -p MemorySwapMax=0`, +sequential with a load-gate + 10s cooldown. Determinism gate **GREEN**, 71 result +files, 0 cell errors, 3 reps/cell, `@opentui/core` 0.4.1 native-yoga +(`libopentui.so`, no `yoga.wasm`). Every number traces to a `summary.` in +a result dir. Source: `~/projects/opentui-html/bench-numbers.json` (frozen +2026-06-14, build under test `1ddf7a102` + WIP). + +### Scorecard + +| Dimension | Winner | Margin | Source cell | +|---|---|---|---| +| Streaming frame rate | **OpenTUI** | **~3×** (43 vs 14 fps) | `cpu800.frame_pacing` | +| Streaming smoothness (interframe p95) | **OpenTUI** | **40ms vs ~220ms** (no ¼-second stalls) | `cpu800.frame_pacing` | +| Scroll CPU | **OpenTUI** | **~2.7× cheaper** (134–155 vs 403–416 ticks) | `scroll3000.scroll.cpu_ticks` | +| Cold-start floor | **OpenTUI** | ~97–103 vs ~107–109 MB | `startup.vmhwm_kb` | +| Session-create latency | **OpenTUI** | ~151–177 vs ~204–229 ms | `startup.session_create_ms` | +| First-byte paint | Ink | ~93 vs ~122 ms | `startup.first_byte_ms` | +| Memory @ small/typical | Ink | OpenTUI +30–50 MB | `mem50/100/300.vmhwm` | +| Memory @ heavy tool output | **OpenTUI** | **crossover** (258–265 vs 280–290 MB) | `results-fat-mem-*` | +| Layout reflow latency | **Ink** | **~0ms vs ~13ms** (OpenTUI's one honest loss) | `resize3000.resize.reflow_ms` | + +### The honest reading + +- **OpenTUI wins everything you feel continuously** — frame rate (~3×), scroll + CPU (~2.7×), and smoothness (no 200ms hitches; p95 40ms vs ~220ms). This is the + lead. The single most user-perceptible difference is the stall-free stream. +- **Memory: lead with smoothness, not raw RSS.** Ink is lighter at small/typical + sizes (OpenTUI carries a ~102 MB irreducible Node+V8+`libopentui.so` floor, so + it sits +30–50 MB above Ink there). But it **crosses over** under heavy tool + output (mem300: 258–265 MB OpenTUI vs 280–290 MB Ink) because windowing beats + Ink's mount-every-row. Real-world: 20 memwatch sessions show a flat ~108 MB + floor and ~0 MB/h on long sessions (one 15h session, 0 MB/h; one 4.4h session + plateaus flat at ~237 MB with mounted rows pinned at 33). +- **The one outright loss is layout reflow** (~13ms p50 vs Ink's ~0ms; under a + resize storm OpenTUI degrades to ~14fps/~197ms vs Ink ~26fps/~100ms). Heavier + native renderables vs Ink's string nodes. This is a real, quantified + optimization target — **not** a regression vs current behavior, and **not** the + "halved 0.4.0→0.4.1" delta (we measured the absolute 12–15ms only; do not quote + "halved" from this run). +- **The memory fix is engine-agnostic** — a rolling display cap + (`HERMES_TUI_MAX_MESSAGES=3000` default) that is display-only and never touches + the model's context. Uncapped is a stress config, not real usage (10k msgs + uncapped: 793 MB; capped sessions are flat MB/h). +- **Gut-check vs upstream/opencode: no bugs.** Exactly one frame callback + (early-exits cheaply), zero `writeToScrollback` for the transcript (one sticky + `` + reactive ``), native `` byte-for-byte + parity with live opencode, no reactive-read-outside-tracking-scope (the #1 Solid + trap). Source: `docs/plans/opentui-gutcheck-verification.md`. + +Full methodology + every cell: see the HTML write-up's benchmark sections and +`docs/plans/opentui-endgame-benchmark-report.md`. + +--- + +## 3. UI parity — and where the two engines genuinely diverge visually + +100% *feature* parity is the bar (matrix in §6), but the two engines are **not** +visually identical. The Ink TUI renders the transcript as a **box-drawing tree**; +OpenTUI renders it **flat and marker-based**. This is a deliberate design +divergence, captured in `ui-opentui/src/view/messageLine.tsx`: + +> *"the view is a dark room and gold is the single lamp — it sits on the NEWEST +> answer's `⚕` and the user's `❯`, nowhere else (older assistant glyphs demote to +> grey: they merely happened)."* + +Real screenshots (saved under `docs/research/opentui-screenshots/`), captured live +on a real PTY 120×40 via the `tmux-pane-screenshot` workflow — **same session +resumed in both engines** where possible. + +### Legacy Ink — `docs/research/opentui-screenshots/ink-transcript.png` + +![Ink transcript](research/opentui-screenshots/ink-transcript.png) + +- **Box-drawing tree layout.** Each turn is a nested structure: `└─ Response`, + `└─ ▾ Tool calls (1)`, ` └─ ● Terminal("…")` — explicit corner rails and + disclosure triangles. +- **`┊` dotted quote-bar** prefixes assistant prose. +- **Tool calls collapse by default** behind a `▾ Tool calls (N)` disclosure, + nested one rail deeper. +- **Whole assistant message tinted gold/amber** (body text is colored, not just + the marker). +- Right-edge scrollbar: thin `│` track + `┃`/orange thumb. +- Status bar: `─ ready │ opus 4.8 fast high │ 0/1m │ [░░░░░░] 0% │ 25s │ voice off │ 1 session ─ ~` + — leading dash, pipe-delimited fields, trailing `~`. +- **No top header bar.** + +### New OpenTUI — `docs/research/opentui-screenshots/opentui-transcript.png` (+ `opentui-toolcall.png`) + +![OpenTUI transcript](research/opentui-screenshots/opentui-transcript.png) + +![OpenTUI tool call](research/opentui-screenshots/opentui-toolcall.png) + +- **Flat, marker-based layout.** No tree rails. Assistant = `⚕` (caduceus, gold + only on the newest answer), user = `❯` (gold chevron + gold text). Older + assistant glyphs demote to grey. +- **Neutral body text.** Gold is reserved for markers and inline-code accents; + prose is grey/white (the "single lamp" rule), so the screen reads calmer than + Ink's all-amber blocks. +- **Tool calls render inline, expanded, on one header line:** + `⚕ ▶ delegate_task Run the shell command `…` (/agents to monitor) · 41s (11 lines)` + — marker, `▶` collapse triangle, bold tool name, grey arg preview, hint, + `· duration`, `(N lines)` — and the result flows flat directly below (no nesting + rail). Per-tool renderers exist (`view/tools/registry.tsx`) — bash/file+diff/ + read/search/skill/clarify/todo each render differently, not a uniform dump. +- **Per-block `⧉ copy` affordance** on a quiet footer line under every settled + assistant block and user prompt (click → copies that block's source). +- **Top header bar:** `⚕ Hermes Agent · opentui · ready` + a gold horizontal rule + (Ink has none). +- Status bar (real backend): `● claude-fable-5 │ [▒▒▒] 4% │ …/lively-thrush/hermes-agent (feat/opentui-native-engine)` + — green status dot, model, context/token bar, **right-pinned cwd + branch**. + +### Divergence summary table + +| Aspect | Ink (legacy) | OpenTUI (new) | +|---|---|---| +| Transcript structure | Box-drawing **tree** (`└─`, rails) | **Flat**, indented, marker-based | +| Assistant marker | `└─ Response` rail + `┊` quote-bar | `⚕` caduceus glyph | +| User marker | (rail) | `❯` gold chevron | +| Assistant body color | Tinted gold/amber | Neutral grey/white (gold = accents only) | +| Tool calls | Collapsed `▾ Tool calls (N)`, nested | Inline expanded header + flat result | +| Per-tool rendering | Largely uniform | Dedicated renderers per tool | +| Copy affordance | `/copy` command | `/copy` **+ per-block `⧉ copy`** | +| Header bar | None | `⚕ Hermes Agent · opentui · ready` + rule | +| Status bar | `─`/`│`-delimited, trailing `~` | dot + bars + right-pinned cwd/branch | + +**For review:** the divergence is intentional (a design pass, not an accident), +but it means "drop-in replacement" is true at the *feature* level, not the +*pixel* level. A user switching engines will immediately notice the flatter, +calmer transcript. Worth calling out explicitly so the swap isn't sold as +visually invisible. + +--- + +## 4. Non-core / kitchen-sink change audit (what review should scrutinize) + +Full report: **`docs/research/opentui-noncore-change-audit.md`** (file-by-file, +commit-by-commit, with `file:line` evidence). Summary below. + +This PR's net footprint vs `origin/main` (two-dot diff = exactly this PR's adds, +no main work re-included): + +| Bucket | Files | Net diff | +|---|---:|---:| +| UI (`ui-opentui/`, the engine + tests) | 197 | +36,001 / −1 | +| Docs | 8 | +1,164 / −0 | +| **Other (the review-flag surface)** | **28** | **+3,218 / −204** | + +The 28 "other" files are the only place this PR touches shared Hermes core. They +classify as: + +### ✅ CORE-OPENTUI-NECESSARY (the engine can't work without these; Ink path provably untouched) + +- **`hermes_cli/main.py`** (+382/−5) — dual-engine launcher (engine resolution, + Node 26 / fnm detection, `_make_opentui_argv`, heap override). Default falls + back to Ink unless the host is OpenTUI-ready (`main.py:1685`); OpenTUI is + dispatched *around* the Ink bootstrap, never through it (`main.py:1914-1922`). +- **`scripts/install.sh`** (+78/−1) — `install_opentui` stage, **strictly + best-effort** (every failure returns 0; falls back to Ink; Windows/Termux + skipped). Ink install path unchanged. +- **`Dockerfile`** (+21/−11) — Node 22→**26** bump (required by the `node:ffi` + renderer) + `ui-opentui` build step. Opt-in; Ink build line preserved. **Caveat: + the Node major bump affects the whole image (Ink + web + Playwright)** — the + diff self-flags "verify the full image build on Node 26 in CI." +- **`hermes_cli/_parser.py`** (+16/−2) — bare `--resume` → OpenTUI session picker; + `--resume ` unchanged. +- **`tui_gateway/server.py`** (+612/−40) — predominantly opt-in RPCs/fields the + new engine calls (`session.peek`, `session.list` filters, `startup.catalog`, + `diff_unified`, window-title, skin keys). Each is gated so **the Ink path is + byte-for-byte unchanged** (`server.py:3930`, `:4254`, `:10447`). *Note:* this + file also carries some of the cost-accounting code (below) — separable. + +> `tui_gateway/` (~12k LOC Python) is **shared by both engines** and is **not** +> removed when Ink is retired. Only the `ui-tui/` frontend tree goes. + +### 🚩 FLAG FOR REVIEW — Category C, separable from an OpenTUI PR + +These do **not** need to ship with the engine and a reviewer should ask to split +them out: + +1. **Provider-reported-cost accounting** (commits `85546bb9e` + `364b93a4b` + + `e01b04de4`) — a coherent feature spanning **11 files**: `agent/usage_pricing.py`, + `plugins/model-providers/openrouter/__init__.py`, + `agent/transports/chat_completions.py`, `agent/agent_init.py`, `run_agent.py`, + `agent/conversation_loop.py`, `agent/account_usage.py`, `hermes_state.py`, + `gateway/slash_commands.py`, the cost half of `cli.py`, and the + `_get_usage`/`_compact_usage_text` blocks of `tui_gateway/server.py` (+ 5 test + files). Strongest evidence: commit `85546bb9e` *"gateway: capture real + provider-reported cost (openrouter usage accounting)"* — a provider-accounting + rework, not a renderer. +2. **`plugins/model-providers/openrouter/__init__.py`** — sends + `usage:{include:true}`, a provider request-shape change affecting *all* + interfaces, not just the TUI (`openrouter/__init__.py:85-90` cites the + OpenRouter usage-accounting docs). +3. **Worktree lock / dirty-tree preservation** (commit `94765e48f`, + `cli.py` + `tests/cli/test_worktree.py`, ~145 lines) — git-worktree lifecycle + safety plumbing with **zero TUI references** (`cli.py:1391-1545`, `:1635-1713`). +4. **`tools/clarify_tool.py`** (+16/−4) — docstring/schema-description-only fix + (commit `16e408f3f`); applies to every interface, trivially separable. + +### ✅ Conversation-loop / role-alternation / prompt-cache correctness verdict: **NO RISK** + +Verified: none of `run_agent.py`, `agent/conversation_loop.py`, +`agent/agent_init.py`, `agent/transports/chat_completions.py` touch +message-role alternation or the prompt-cache prefix. The +`conversation_loop.py` added lines grep clean for +`cache_control|alternation|prompt_cach|api_messages`; the cache/alternation +machinery (`:57`, `:660-674`, `:759`) is untouched; the PR's insertion at +`:1809-1879` is purely additive cost bookkeeping after `cost_result`. **Prompt +caching and strict role alternation are preserved.** + +--- + +## 5. What this does and does NOT fix + +**Fixes (structurally, by replacing the rendering substrate):** the renderer bug +class — layout/scroll/input/copy/mouse/markdown/resize — plus the +hand-maintained memory-eviction problem (windowing + Solid keyed `` +unmount→`destroy()`→`free()`), and several long-open feature requests (mouse, +collapsible tool calls, session title/status bar, double-ESC, chronological +thinking/tool ordering). + +**Does NOT fix:** the gateway is unchanged — the biggest single hotspot file in +triage is `tui_gateway/server.py`, and whole bug clusters are gateway/Python-side +(WS write-timeout/RPC pool, MCP-failure startup freezes, shell.exec denylist). +The engine swap addresses rendering/input/scroll/memory; **gateway bugs ride +along.** The Effect-boundary hardening does make those failures *visible* (typed +events → system lines instead of a frozen spinner) and the TUI auto-heals +(crash → backoff → respawn → resume, capped 3/60s). + +--- + +## 6. Feature parity matrix (vs the Ink TUI) + +Verbatim, detailed, surface-by-surface with `file:line` evidence: +**`docs/plans/opentui-ink-parity-matrix.md`** (interactive/filterable version in +the HTML write-up). Headline state: + +| Surface | State | +|---|---| +| Transcript rendering (scrollbox, markdown, code, diffs, collapsible tools, reasoning, chronological order, windowing) | **full parity (9/9)** | +| Blocking prompts (approval/clarify/sudo/secret/confirm) | **full parity (5/5)** | +| Theming (skins, light/dark, ANSI-256 norm) | **full parity** | +| Mouse / copy (tracking, selection, multi-click, OSC52, click-to-expand, wheel accel) | **full parity** | +| Resilience (crash auto-heal + resume) | **parity++ (exponential backoff)** | +| Composer / input | near parity — **missing: external editor (Ctrl+G → `$EDITOR`)**; ghost-text autosuggest partial | +| Slash commands | core parity — **missing: `/setup`, `/redraw`, `/plugins`, `/voice`**; `/undo` prefill + `/image` partial | +| Status bar / header chrome | almost all closed — **missing: MCP-servers panel, profile-in-prompt** | +| Agent surfaces | most shipped — **missing: voice indicators, browser/CDP indicator** | +| Utility commands | **missing: `/redraw`, `/setup`**; rest present | + +> The original PR-draft gap list was **substantially stale** — the WIP since +> shipped context %/token bar, cost, compressions, duration, update banner, todos +> panel, activity feed, notifications, background-task indicator, **and per-tool +> renderers** (the "every tool renders the same" claim is false: +> `view/tools/registry.tsx` has dedicated renderers). + +### Genuinely-remaining parity gaps + +- [ ] **External editor (Ctrl+G → `$EDITOR`)** — highest-impact missing composer affordance +- [ ] MCP-servers detail panel; profile-in-prompt marker +- [ ] Voice indicators (listening/transcribing/REC/STT) + `/voice` +- [ ] Browser/CDP connection indicator + `/browser` +- [ ] `/setup` wizard handoff, `/redraw`, `/plugins` hub +- [ ] Draggable scrollbar; sticky-prompt line +- [ ] `/undo` prefill into composer; model-picker persist-global toggle; skills-hub install/manage + +--- + +## 7. Rollout, runtime & risks + +- **Runtime:** plain Node 26 (FFI floor 26.3+) — one runtime, no Bun. (Note: the + upstream OpenTUI docs say "requires Bun"; this engine deliberately runs on Node + 26's experimental `node:ffi` instead — that's the load-bearing runtime decision.) +- **Rollback:** Ink is untouched and remains the fallback; reverting is a launcher + decision, not a code revert. +- **Default-engine selection:** auto-picks OpenTUI only when the host is genuinely + set up (Node ≥ 26.3 + built bundle), else Ink; explicit env/config bypasses the + probe. +- **Known sharp edges:** `libopentui.so` native-lib distribution (P1 upstream: + copies can fill `/tmp`); the Dockerfile Node major bump needs full-image CI + verification; tree-sitter grammars are fetched from GitHub on first use and + cached in `~/.hermes/cache/opentui-parsers/` — air-gapped hosts get plain-text + highlighting until the cache is pre-seeded (the fetch never blocks boot and + never throws). + +## 8. Try it + +```bash +hermes # auto-selects OpenTUI when the host supports it +HERMES_TUI_ENGINE=opentui hermes # force the native engine +HERMES_TUI_ENGINE=ink hermes # force the legacy Ink engine +# preview standalone (no backend), Node 26: +cd ui-opentui && npm install +node scripts/build.mjs scripts/demo.tsx .demo +DEMO_TOTAL=120 HERMES_TUI_MAX_MESSAGES=80 \ + node --experimental-ffi --no-warnings .demo/demo.js # inside a TTY +``` + +Requires Node 26.3+. On older Node / Windows / Termux it auto-falls-back to Ink. + +--- + +## Appendix — source-of-truth files in this repo + +| Topic | File | +|---|---| +| Non-core change audit (full) | `docs/research/opentui-noncore-change-audit.md` | +| Feature parity matrix (verbatim) | `docs/plans/opentui-ink-parity-matrix.md` | +| Benchmark report | `docs/plans/opentui-endgame-benchmark-report.md` | +| Gut-check verification | `docs/plans/opentui-gutcheck-verification.md` | +| Ink↔OpenTUI capture asymmetry | `docs/plans/opentui-ink-asymmetry-note.md` | +| UI screenshots | `docs/research/opentui-screenshots/{ink,opentui}-*.png` | +| PR description (prose) | `docs/pr-description-main-doc.md` | +| Interactive write-up | `~/projects/opentui-perf-writeup/index.html` (out-of-repo) | diff --git a/hermes_cli/main.py b/hermes_cli/main.py index 6c1baa0908c..15211ad6a28 100644 --- a/hermes_cli/main.py +++ b/hermes_cli/main.py @@ -2281,6 +2281,18 @@ def _launch_tui( ) os.close(active_session_fd) env["HERMES_TUI_ACTIVE_SESSION_FILE"] = active_session_file + # Tree-sitter grammar cache for the OpenTUI engine: grammars are fetched + # from GitHub on first use and cached here (profile-aware). Unset → OpenTUI + # falls back to its XDG default ($XDG_DATA_HOME/opentui). See + # ui-opentui/src/boundary/parsers.ts. + try: + from hermes_cli.config import get_hermes_home + + env["HERMES_TUI_PARSER_CACHE"] = str( + get_hermes_home() / "cache" / "opentui-parsers" + ) + except Exception: + logger.debug("Failed to resolve OpenTUI parser cache dir", exc_info=True) env["HERMES_PYTHON_SRC_ROOT"] = os.environ.get( "HERMES_PYTHON_SRC_ROOT", str(PROJECT_ROOT) ) diff --git a/ui-opentui/parsers/bash/highlights.scm b/ui-opentui/parsers/bash/highlights.scm deleted file mode 100644 index f33a7c2d3ae..00000000000 --- a/ui-opentui/parsers/bash/highlights.scm +++ /dev/null @@ -1,56 +0,0 @@ -[ - (string) - (raw_string) - (heredoc_body) - (heredoc_start) -] @string - -(command_name) @function - -(variable_name) @property - -[ - "case" - "do" - "done" - "elif" - "else" - "esac" - "export" - "fi" - "for" - "function" - "if" - "in" - "select" - "then" - "unset" - "until" - "while" -] @keyword - -(comment) @comment - -(function_definition name: (word) @function) - -(file_descriptor) @number - -[ - (command_substitution) - (process_substitution) - (expansion) -]@embedded - -[ - "$" - "&&" - ">" - ">>" - "<" - "|" -] @operator - -( - (command (_) @constant) - (#match? @constant "^-") -) diff --git a/ui-opentui/parsers/bash/tree-sitter-bash.wasm b/ui-opentui/parsers/bash/tree-sitter-bash.wasm deleted file mode 100644 index 28e2869fada..00000000000 Binary files a/ui-opentui/parsers/bash/tree-sitter-bash.wasm and /dev/null differ diff --git a/ui-opentui/parsers/c/highlights.scm b/ui-opentui/parsers/c/highlights.scm deleted file mode 100644 index 8ee118900ff..00000000000 --- a/ui-opentui/parsers/c/highlights.scm +++ /dev/null @@ -1,81 +0,0 @@ -(identifier) @variable - -((identifier) @constant - (#match? @constant "^[A-Z][A-Z\\d_]*$")) - -"break" @keyword -"case" @keyword -"const" @keyword -"continue" @keyword -"default" @keyword -"do" @keyword -"else" @keyword -"enum" @keyword -"extern" @keyword -"for" @keyword -"if" @keyword -"inline" @keyword -"return" @keyword -"sizeof" @keyword -"static" @keyword -"struct" @keyword -"switch" @keyword -"typedef" @keyword -"union" @keyword -"volatile" @keyword -"while" @keyword - -"#define" @keyword -"#elif" @keyword -"#else" @keyword -"#endif" @keyword -"#if" @keyword -"#ifdef" @keyword -"#ifndef" @keyword -"#include" @keyword -(preproc_directive) @keyword - -"--" @operator -"-" @operator -"-=" @operator -"->" @operator -"=" @operator -"!=" @operator -"*" @operator -"&" @operator -"&&" @operator -"+" @operator -"++" @operator -"+=" @operator -"<" @operator -"==" @operator -">" @operator -"||" @operator - -"." @delimiter -";" @delimiter - -(string_literal) @string -(system_lib_string) @string - -(null) @constant -(number_literal) @number -(char_literal) @number - -(field_identifier) @property -(statement_identifier) @label -(type_identifier) @type -(primitive_type) @type -(sized_type_specifier) @type - -(call_expression - function: (identifier) @function) -(call_expression - function: (field_expression - field: (field_identifier) @function)) -(function_declarator - declarator: (identifier) @function) -(preproc_function_def - name: (identifier) @function.special) - -(comment) @comment diff --git a/ui-opentui/parsers/c/tree-sitter-c.wasm b/ui-opentui/parsers/c/tree-sitter-c.wasm deleted file mode 100644 index ddfbcc1e66f..00000000000 Binary files a/ui-opentui/parsers/c/tree-sitter-c.wasm and /dev/null differ diff --git a/ui-opentui/parsers/css/highlights.scm b/ui-opentui/parsers/css/highlights.scm deleted file mode 100644 index 763661af761..00000000000 --- a/ui-opentui/parsers/css/highlights.scm +++ /dev/null @@ -1,64 +0,0 @@ -(comment) @comment - -(tag_name) @tag -(nesting_selector) @tag -(universal_selector) @tag - -"~" @operator -">" @operator -"+" @operator -"-" @operator -"*" @operator -"/" @operator -"=" @operator -"^=" @operator -"|=" @operator -"~=" @operator -"$=" @operator -"*=" @operator - -"and" @operator -"or" @operator -"not" @operator -"only" @operator - -(attribute_selector (plain_value) @string) -(pseudo_element_selector (tag_name) @attribute) -(pseudo_class_selector (class_name) @attribute) - -(class_name) @property -(id_name) @property -(namespace_name) @property -(property_name) @property -(feature_name) @property - -(attribute_name) @attribute - -(function_name) @function - -((property_name) @variable - (#match? @variable "^--")) -((plain_value) @variable - (#match? @variable "^--")) - -"@media" @keyword -"@import" @keyword -"@charset" @keyword -"@namespace" @keyword -"@supports" @keyword -"@keyframes" @keyword -(at_keyword) @keyword -(to) @keyword -(from) @keyword -(important) @keyword - -(string_value) @string -(color_value) @string.special - -(integer_value) @number -(float_value) @number -(unit) @type - -"#" @punctuation.delimiter -"," @punctuation.delimiter -":" @punctuation.delimiter diff --git a/ui-opentui/parsers/css/tree-sitter-css.wasm b/ui-opentui/parsers/css/tree-sitter-css.wasm deleted file mode 100644 index a958382d2ef..00000000000 Binary files a/ui-opentui/parsers/css/tree-sitter-css.wasm and /dev/null differ diff --git a/ui-opentui/parsers/go/highlights.scm b/ui-opentui/parsers/go/highlights.scm deleted file mode 100644 index 6a3c0ac8b82..00000000000 --- a/ui-opentui/parsers/go/highlights.scm +++ /dev/null @@ -1,123 +0,0 @@ -; Function calls - -(call_expression - function: (identifier) @function) - -(call_expression - function: (identifier) @function.builtin - (#match? @function.builtin "^(append|cap|close|complex|copy|delete|imag|len|make|new|panic|print|println|real|recover)$")) - -(call_expression - function: (selector_expression - field: (field_identifier) @function.method)) - -; Function definitions - -(function_declaration - name: (identifier) @function) - -(method_declaration - name: (field_identifier) @function.method) - -; Identifiers - -(type_identifier) @type -(field_identifier) @property -(identifier) @variable - -; Operators - -[ - "--" - "-" - "-=" - ":=" - "!" - "!=" - "..." - "*" - "*" - "*=" - "/" - "/=" - "&" - "&&" - "&=" - "%" - "%=" - "^" - "^=" - "+" - "++" - "+=" - "<-" - "<" - "<<" - "<<=" - "<=" - "=" - "==" - ">" - ">=" - ">>" - ">>=" - "|" - "|=" - "||" - "~" -] @operator - -; Keywords - -[ - "break" - "case" - "chan" - "const" - "continue" - "default" - "defer" - "else" - "fallthrough" - "for" - "func" - "go" - "goto" - "if" - "import" - "interface" - "map" - "package" - "range" - "return" - "select" - "struct" - "switch" - "type" - "var" -] @keyword - -; Literals - -[ - (interpreted_string_literal) - (raw_string_literal) - (rune_literal) -] @string - -(escape_sequence) @escape - -[ - (int_literal) - (float_literal) - (imaginary_literal) -] @number - -[ - (true) - (false) - (nil) - (iota) -] @constant.builtin - -(comment) @comment diff --git a/ui-opentui/parsers/go/tree-sitter-go.wasm b/ui-opentui/parsers/go/tree-sitter-go.wasm deleted file mode 100644 index 74cd9277156..00000000000 Binary files a/ui-opentui/parsers/go/tree-sitter-go.wasm and /dev/null differ diff --git a/ui-opentui/parsers/html/highlights.scm b/ui-opentui/parsers/html/highlights.scm deleted file mode 100644 index ea0ff4e3085..00000000000 --- a/ui-opentui/parsers/html/highlights.scm +++ /dev/null @@ -1,13 +0,0 @@ -(tag_name) @tag -(erroneous_end_tag_name) @tag.error -(doctype) @constant -(attribute_name) @attribute -(attribute_value) @string -(comment) @comment - -[ - "<" - ">" - "" -] @punctuation.bracket diff --git a/ui-opentui/parsers/html/tree-sitter-html.wasm b/ui-opentui/parsers/html/tree-sitter-html.wasm deleted file mode 100644 index d0fca0826ef..00000000000 Binary files a/ui-opentui/parsers/html/tree-sitter-html.wasm and /dev/null differ diff --git a/ui-opentui/parsers/json/highlights.scm b/ui-opentui/parsers/json/highlights.scm deleted file mode 100644 index ece8392f0bc..00000000000 --- a/ui-opentui/parsers/json/highlights.scm +++ /dev/null @@ -1,16 +0,0 @@ -(pair - key: (_) @string.special.key) - -(string) @string - -(number) @number - -[ - (null) - (true) - (false) -] @constant.builtin - -(escape_sequence) @escape - -(comment) @comment diff --git a/ui-opentui/parsers/json/tree-sitter-json.wasm b/ui-opentui/parsers/json/tree-sitter-json.wasm deleted file mode 100644 index a75bba7a655..00000000000 Binary files a/ui-opentui/parsers/json/tree-sitter-json.wasm and /dev/null differ diff --git a/ui-opentui/parsers/manifest.json b/ui-opentui/parsers/manifest.json deleted file mode 100644 index 5d9f32ae60e..00000000000 --- a/ui-opentui/parsers/manifest.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "comment": "Tree-sitter grammars vendored beyond @opentui/core's bundled 5 (ts/js/markdown/markdown_inline/zig). Source of truth for scripts/update-parsers.mjs (vendoring) and src/boundary/parsers.ts (boot registration). Curated in docs/plans/opentui-syntax-highlighting-languages.md — cpp deliberately dropped (3.28 MB, ~half the bundle). Aliases are belt-and-braces: core's extToFiletype/infoStringToFiletype already normalize py->python etc., but a literal alias filetype reaching the client still resolves.", - "parsers": [ - { "filetype": "python", "aliases": ["py"], "org": "tree-sitter", "tag": "v0.23.6" }, - { "filetype": "rust", "aliases": ["rs"], "org": "tree-sitter", "tag": "v0.23.2" }, - { "filetype": "go", "aliases": [], "org": "tree-sitter", "tag": "v0.23.4" }, - { "filetype": "bash", "aliases": ["sh", "shell", "zsh"], "org": "tree-sitter", "tag": "v0.23.3" }, - { "filetype": "json", "aliases": [], "org": "tree-sitter", "tag": "v0.24.8" }, - { "filetype": "c", "aliases": ["h"], "org": "tree-sitter", "tag": "v0.23.5" }, - { "filetype": "html", "aliases": [], "org": "tree-sitter", "tag": "v0.23.2" }, - { "filetype": "css", "aliases": [], "org": "tree-sitter", "tag": "v0.23.2" }, - { "filetype": "yaml", "aliases": ["yml"], "org": "tree-sitter-grammars", "tag": "v0.7.1" }, - { "filetype": "toml", "aliases": [], "org": "tree-sitter-grammars", "tag": "v0.7.0" } - ] -} diff --git a/ui-opentui/parsers/python/highlights.scm b/ui-opentui/parsers/python/highlights.scm deleted file mode 100644 index af744484432..00000000000 --- a/ui-opentui/parsers/python/highlights.scm +++ /dev/null @@ -1,137 +0,0 @@ -; Identifier naming conventions - -(identifier) @variable - -((identifier) @constructor - (#match? @constructor "^[A-Z]")) - -((identifier) @constant - (#match? @constant "^[A-Z][A-Z_]*$")) - -; Function calls - -(decorator) @function -(decorator - (identifier) @function) - -(call - function: (attribute attribute: (identifier) @function.method)) -(call - function: (identifier) @function) - -; Builtin functions - -((call - function: (identifier) @function.builtin) - (#match? - @function.builtin - "^(abs|all|any|ascii|bin|bool|breakpoint|bytearray|bytes|callable|chr|classmethod|compile|complex|delattr|dict|dir|divmod|enumerate|eval|exec|filter|float|format|frozenset|getattr|globals|hasattr|hash|help|hex|id|input|int|isinstance|issubclass|iter|len|list|locals|map|max|memoryview|min|next|object|oct|open|ord|pow|print|property|range|repr|reversed|round|set|setattr|slice|sorted|staticmethod|str|sum|super|tuple|type|vars|zip|__import__)$")) - -; Function definitions - -(function_definition - name: (identifier) @function) - -(attribute attribute: (identifier) @property) -(type (identifier) @type) - -; Literals - -[ - (none) - (true) - (false) -] @constant.builtin - -[ - (integer) - (float) -] @number - -(comment) @comment -(string) @string -(escape_sequence) @escape - -(interpolation - "{" @punctuation.special - "}" @punctuation.special) @embedded - -[ - "-" - "-=" - "!=" - "*" - "**" - "**=" - "*=" - "/" - "//" - "//=" - "/=" - "&" - "&=" - "%" - "%=" - "^" - "^=" - "+" - "->" - "+=" - "<" - "<<" - "<<=" - "<=" - "<>" - "=" - ":=" - "==" - ">" - ">=" - ">>" - ">>=" - "|" - "|=" - "~" - "@=" - "and" - "in" - "is" - "not" - "or" - "is not" - "not in" -] @operator - -[ - "as" - "assert" - "async" - "await" - "break" - "class" - "continue" - "def" - "del" - "elif" - "else" - "except" - "exec" - "finally" - "for" - "from" - "global" - "if" - "import" - "lambda" - "nonlocal" - "pass" - "print" - "raise" - "return" - "try" - "while" - "with" - "yield" - "match" - "case" -] @keyword diff --git a/ui-opentui/parsers/python/tree-sitter-python.wasm b/ui-opentui/parsers/python/tree-sitter-python.wasm deleted file mode 100644 index ebd96b5db34..00000000000 Binary files a/ui-opentui/parsers/python/tree-sitter-python.wasm and /dev/null differ diff --git a/ui-opentui/parsers/rust/highlights.scm b/ui-opentui/parsers/rust/highlights.scm deleted file mode 100644 index 48c7284ec2b..00000000000 --- a/ui-opentui/parsers/rust/highlights.scm +++ /dev/null @@ -1,161 +0,0 @@ -; Identifiers - -(type_identifier) @type -(primitive_type) @type.builtin -(field_identifier) @property - -; Identifier conventions - -; Assume all-caps names are constants -((identifier) @constant - (#match? @constant "^[A-Z][A-Z\\d_]+$'")) - -; Assume uppercase names are enum constructors -((identifier) @constructor - (#match? @constructor "^[A-Z]")) - -; Assume that uppercase names in paths are types -((scoped_identifier - path: (identifier) @type) - (#match? @type "^[A-Z]")) -((scoped_identifier - path: (scoped_identifier - name: (identifier) @type)) - (#match? @type "^[A-Z]")) -((scoped_type_identifier - path: (identifier) @type) - (#match? @type "^[A-Z]")) -((scoped_type_identifier - path: (scoped_identifier - name: (identifier) @type)) - (#match? @type "^[A-Z]")) - -; Assume all qualified names in struct patterns are enum constructors. (They're -; either that, or struct names; highlighting both as constructors seems to be -; the less glaring choice of error, visually.) -(struct_pattern - type: (scoped_type_identifier - name: (type_identifier) @constructor)) - -; Function calls - -(call_expression - function: (identifier) @function) -(call_expression - function: (field_expression - field: (field_identifier) @function.method)) -(call_expression - function: (scoped_identifier - "::" - name: (identifier) @function)) - -(generic_function - function: (identifier) @function) -(generic_function - function: (scoped_identifier - name: (identifier) @function)) -(generic_function - function: (field_expression - field: (field_identifier) @function.method)) - -(macro_invocation - macro: (identifier) @function.macro - "!" @function.macro) - -; Function definitions - -(function_item (identifier) @function) -(function_signature_item (identifier) @function) - -(line_comment) @comment -(block_comment) @comment - -(line_comment (doc_comment)) @comment.documentation -(block_comment (doc_comment)) @comment.documentation - -"(" @punctuation.bracket -")" @punctuation.bracket -"[" @punctuation.bracket -"]" @punctuation.bracket -"{" @punctuation.bracket -"}" @punctuation.bracket - -(type_arguments - "<" @punctuation.bracket - ">" @punctuation.bracket) -(type_parameters - "<" @punctuation.bracket - ">" @punctuation.bracket) - -"::" @punctuation.delimiter -":" @punctuation.delimiter -"." @punctuation.delimiter -"," @punctuation.delimiter -";" @punctuation.delimiter - -(parameter (identifier) @variable.parameter) - -(lifetime (identifier) @label) - -"as" @keyword -"async" @keyword -"await" @keyword -"break" @keyword -"const" @keyword -"continue" @keyword -"default" @keyword -"dyn" @keyword -"else" @keyword -"enum" @keyword -"extern" @keyword -"fn" @keyword -"for" @keyword -"gen" @keyword -"if" @keyword -"impl" @keyword -"in" @keyword -"let" @keyword -"loop" @keyword -"macro_rules!" @keyword -"match" @keyword -"mod" @keyword -"move" @keyword -"pub" @keyword -"raw" @keyword -"ref" @keyword -"return" @keyword -"static" @keyword -"struct" @keyword -"trait" @keyword -"type" @keyword -"union" @keyword -"unsafe" @keyword -"use" @keyword -"where" @keyword -"while" @keyword -"yield" @keyword -(crate) @keyword -(mutable_specifier) @keyword -(use_list (self) @keyword) -(scoped_use_list (self) @keyword) -(scoped_identifier (self) @keyword) -(super) @keyword - -(self) @variable.builtin - -(char_literal) @string -(string_literal) @string -(raw_string_literal) @string - -(boolean_literal) @constant.builtin -(integer_literal) @constant.builtin -(float_literal) @constant.builtin - -(escape_sequence) @escape - -(attribute_item) @attribute -(inner_attribute_item) @attribute - -"*" @operator -"&" @operator -"'" @operator diff --git a/ui-opentui/parsers/rust/tree-sitter-rust.wasm b/ui-opentui/parsers/rust/tree-sitter-rust.wasm deleted file mode 100644 index 7a5cf2c9495..00000000000 Binary files a/ui-opentui/parsers/rust/tree-sitter-rust.wasm and /dev/null differ diff --git a/ui-opentui/parsers/toml/highlights.scm b/ui-opentui/parsers/toml/highlights.scm deleted file mode 100644 index 456148935ea..00000000000 --- a/ui-opentui/parsers/toml/highlights.scm +++ /dev/null @@ -1,53 +0,0 @@ -; Properties -;----------- - -(bare_key) @type - -(quoted_key) @string - -(pair - (bare_key)) @property - -(pair - (dotted_key - (bare_key) @property)) - -; Literals -;--------- - -(boolean) @boolean - -(comment) @comment - -(string) @string - -[ - (integer) - (float) -] @number - -[ - (offset_date_time) - (local_date_time) - (local_date) - (local_time) -] @string.special - -; Punctuation -;------------ - -[ - "." - "," -] @punctuation.delimiter - -"=" @operator - -[ - "[" - "]" - "[[" - "]]" - "{" - "}" -] @punctuation.bracket diff --git a/ui-opentui/parsers/toml/tree-sitter-toml.wasm b/ui-opentui/parsers/toml/tree-sitter-toml.wasm deleted file mode 100644 index f6e1e4a354c..00000000000 Binary files a/ui-opentui/parsers/toml/tree-sitter-toml.wasm and /dev/null differ diff --git a/ui-opentui/parsers/yaml/highlights.scm b/ui-opentui/parsers/yaml/highlights.scm deleted file mode 100644 index cb9dcc622e3..00000000000 --- a/ui-opentui/parsers/yaml/highlights.scm +++ /dev/null @@ -1,79 +0,0 @@ -(boolean_scalar) @boolean - -(null_scalar) @constant.builtin - -[ - (double_quote_scalar) - (single_quote_scalar) - (block_scalar) - (string_scalar) -] @string - -[ - (integer_scalar) - (float_scalar) -] @number - -(comment) @comment - -[ - (anchor_name) - (alias_name) -] @label - -(tag) @type - -[ - (yaml_directive) - (tag_directive) - (reserved_directive) -] @attribute - -(block_mapping_pair - key: (flow_node - [ - (double_quote_scalar) - (single_quote_scalar) - ] @property)) - -(block_mapping_pair - key: (flow_node - (plain_scalar - (string_scalar) @property))) - -(flow_mapping - (_ - key: (flow_node - [ - (double_quote_scalar) - (single_quote_scalar) - ] @property))) - -(flow_mapping - (_ - key: (flow_node - (plain_scalar - (string_scalar) @property)))) - -[ - "," - "-" - ":" - ">" - "?" - "|" -] @punctuation.delimiter - -[ - "[" - "]" - "{" - "}" -] @punctuation.bracket - -[ - "*" - "&" - "---" - "..." -] @punctuation.special diff --git a/ui-opentui/parsers/yaml/tree-sitter-yaml.wasm b/ui-opentui/parsers/yaml/tree-sitter-yaml.wasm deleted file mode 100644 index 96e9e413732..00000000000 Binary files a/ui-opentui/parsers/yaml/tree-sitter-yaml.wasm and /dev/null differ diff --git a/ui-opentui/scripts/demo.tsx b/ui-opentui/scripts/demo.tsx index ab7fadd8964..291beb36010 100644 --- a/ui-opentui/scripts/demo.tsx +++ b/ui-opentui/scripts/demo.tsx @@ -16,14 +16,15 @@ import { createCliRenderer } from '@opentui/core' import { render } from '@opentui/solid' import { installMultiClickSelection } from '../src/boundary/multiClickSelect.ts' -import { registerVendoredParsers } from '../src/boundary/parsers.ts' +import { registerRemoteParsers } from '../src/boundary/parsers.ts' import { createSessionStore } from '../src/logic/store.ts' import { App } from '../src/view/App.tsx' import { ThemeProvider } from '../src/view/theme.tsx' import { materialize } from './fixture.ts' -// Same grammar registration as the live entry so fixture code blocks highlight. -registerVendoredParsers() +// Same grammar registration as the live entry so fixture code blocks highlight +// (fetched+cached on first use; no HERMES_TUI_PARSER_CACHE here → OpenTUI default). +registerRemoteParsers() const TOTAL = Number.parseInt(process.env.DEMO_TOTAL ?? '', 10) || 200 diff --git a/ui-opentui/scripts/update-parsers.mjs b/ui-opentui/scripts/update-parsers.mjs deleted file mode 100644 index c66005bbbc0..00000000000 --- a/ui-opentui/scripts/update-parsers.mjs +++ /dev/null @@ -1,71 +0,0 @@ -/** - * Vendor the extra Tree-sitter grammars listed in parsers/manifest.json into - * parsers// (wasm + highlights.scm). Plain Node fetch — no Bun - * (core's update-assets.js is bun-shebanged; its download logic is just - * fetch+fs, so we do the same two writes ourselves and keep the generated - * import-module out of the esbuild bundle entirely — registration reads the - * vendored files by PATH at runtime, see src/boundary/parsers.ts). - * - * Idempotent: existing valid files are kept unless --force. Validates wasm - * magic and a non-empty query so a bad download can never be committed. - * - * node scripts/update-parsers.mjs [--force] - * - * The vendored files are COMMITTED (build inputs, like @opentui/core's own - * assets/) — builds and offline machines never re-download. - */ -import { Buffer } from 'node:buffer' -import { mkdir, readFile, writeFile } from 'node:fs/promises' -import { dirname, join, resolve } from 'node:path' -import { fileURLToPath } from 'node:url' - -const root = resolve(dirname(fileURLToPath(import.meta.url)), '..') -const parsersDir = join(root, 'parsers') -const force = process.argv.includes('--force') - -const manifest = JSON.parse(await readFile(join(parsersDir, 'manifest.json'), 'utf8')) - -const wasmUrl = p => - `https://github.com/${p.org}/tree-sitter-${p.filetype}/releases/download/${p.tag}/tree-sitter-${p.filetype}.wasm` -const scmUrl = p => - `https://raw.githubusercontent.com/${p.org}/tree-sitter-${p.filetype}/${p.tag}/queries/highlights.scm` - -async function fetchBytes(url) { - const response = await globalThis.fetch(url) - if (!response.ok) throw new Error(`${response.status} ${response.statusText} for ${url}`) - return Buffer.from(await response.arrayBuffer()) -} - -async function haveValid(path, validate) { - try { - return validate(await readFile(path)) - } catch { - return false - } -} - -const isWasm = bytes => bytes.length > 8 && bytes.subarray(0, 4).toString('latin1') === '\0asm' -const isQuery = bytes => bytes.length > 0 && bytes.toString('utf8').trim().length > 0 - -let downloaded = 0 -for (const parser of manifest.parsers) { - const dir = join(parsersDir, parser.filetype) - await mkdir(dir, { recursive: true }) - const targets = [ - { name: `tree-sitter-${parser.filetype}.wasm`, url: wasmUrl(parser), validate: isWasm }, - { name: 'highlights.scm', url: scmUrl(parser), validate: isQuery } - ] - for (const target of targets) { - const path = join(dir, target.name) - if (!force && (await haveValid(path, target.validate))) { - console.log(`✓ kept ${parser.filetype}/${target.name}`) - continue - } - const bytes = await fetchBytes(target.url) - if (!target.validate(bytes)) throw new Error(`validation failed for ${target.url}`) - await writeFile(path, bytes) - downloaded += 1 - console.log(`↓ fetched ${parser.filetype}/${target.name} (${(bytes.length / 1024).toFixed(0)} KB)`) - } -} -console.log(`done — ${downloaded} file(s) fetched, ${manifest.parsers.length} grammars vendored`) diff --git a/ui-opentui/src/boundary/parsers.manifest.json b/ui-opentui/src/boundary/parsers.manifest.json new file mode 100644 index 00000000000..4f37d69caf6 --- /dev/null +++ b/ui-opentui/src/boundary/parsers.manifest.json @@ -0,0 +1,65 @@ +{ + "comment": "Tree-sitter grammars beyond @opentui/core's bundled 5 (ts/js/markdown/markdown_inline/zig). NOT vendored — declared as remote URLs and fetched+cached at runtime by OpenTUI's TreeSitterClient into HERMES_TUI_PARSER_CACHE (see src/boundary/parsers.ts). This file is the URL source of truth. Grammar versions are pinned via the release tag in each wasm URL; .scm highlight queries follow opencode's per-language source choices (parser-repo queries for python/html where nvim-treesitter's are parser-incompatible, nvim-treesitter master otherwise). cpp deliberately dropped (3.28 MB, ~half the old vendored bundle). Aliases are belt-and-braces: core's extToFiletype/infoStringToFiletype already normalize py->python etc., but a literal alias filetype reaching the client still resolves. To add/refresh a grammar, edit the wasm tag + the highlights URL here — no binaries land in the repo.", + "parsers": [ + { + "filetype": "python", + "aliases": ["py"], + "wasm": "https://github.com/tree-sitter/tree-sitter-python/releases/download/v0.23.6/tree-sitter-python.wasm", + "highlights": "https://github.com/tree-sitter/tree-sitter-python/raw/refs/heads/master/queries/highlights.scm" + }, + { + "filetype": "rust", + "aliases": ["rs"], + "wasm": "https://github.com/tree-sitter/tree-sitter-rust/releases/download/v0.23.2/tree-sitter-rust.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/rust/highlights.scm" + }, + { + "filetype": "go", + "aliases": [], + "wasm": "https://github.com/tree-sitter/tree-sitter-go/releases/download/v0.23.4/tree-sitter-go.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/go/highlights.scm" + }, + { + "filetype": "bash", + "aliases": ["sh", "shell", "zsh"], + "wasm": "https://github.com/tree-sitter/tree-sitter-bash/releases/download/v0.23.3/tree-sitter-bash.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/bash/highlights.scm" + }, + { + "filetype": "json", + "aliases": [], + "wasm": "https://github.com/tree-sitter/tree-sitter-json/releases/download/v0.24.8/tree-sitter-json.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/json/highlights.scm" + }, + { + "filetype": "c", + "aliases": ["h"], + "wasm": "https://github.com/tree-sitter/tree-sitter-c/releases/download/v0.23.5/tree-sitter-c.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/c/highlights.scm" + }, + { + "filetype": "html", + "aliases": [], + "wasm": "https://github.com/tree-sitter/tree-sitter-html/releases/download/v0.23.2/tree-sitter-html.wasm", + "highlights": "https://github.com/tree-sitter/tree-sitter-html/raw/refs/heads/master/queries/highlights.scm" + }, + { + "filetype": "css", + "aliases": [], + "wasm": "https://github.com/tree-sitter/tree-sitter-css/releases/download/v0.23.2/tree-sitter-css.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/css/highlights.scm" + }, + { + "filetype": "yaml", + "aliases": ["yml"], + "wasm": "https://github.com/tree-sitter-grammars/tree-sitter-yaml/releases/download/v0.7.1/tree-sitter-yaml.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/yaml/highlights.scm" + }, + { + "filetype": "toml", + "aliases": [], + "wasm": "https://github.com/tree-sitter-grammars/tree-sitter-toml/releases/download/v0.7.0/tree-sitter-toml.wasm", + "highlights": "https://raw.githubusercontent.com/nvim-treesitter/nvim-treesitter/refs/heads/master/queries/toml/highlights.scm" + } + ] +} diff --git a/ui-opentui/src/boundary/parsers.ts b/ui-opentui/src/boundary/parsers.ts index 4a84f298156..499f7736a02 100644 --- a/ui-opentui/src/boundary/parsers.ts +++ b/ui-opentui/src/boundary/parsers.ts @@ -2,37 +2,43 @@ * Extra Tree-sitter grammar registration — the syntax-highlighting language * expansion (docs/plans/opentui-syntax-highlighting-languages.md). * - * @opentui/core@0.4.0 bundles only 5 grammars (ts/js/markdown/markdown_inline/ - * zig); everything else rendered plain text. The cure is the public - * `addDefaultParsers()` API + the grammars vendored under `parsers//` - * (committed; refresh via `node scripts/update-parsers.mjs`). + * @opentui/core@0.4.x bundles only a handful of grammars (ts/js/markdown/ + * markdown_inline/zig); everything else renders plain text. The cure is the + * public `addDefaultParsers()` API fed with REMOTE grammar URLs — OpenTUI's + * TreeSitterClient fetches each `.wasm`/`.scm` lazily on first use of a + * filetype and caches it under the client's `dataPath`. We do NOT vendor any + * binaries (cf. opencode, which checks in zero `.wasm`/`.scm` and lets OpenTUI + * fetch+cache). The grammar set + its URLs live in `parsers.manifest.json`. * - * Why paths, not the generated import-module: core's `updateAssets` generates - * a Bun-flavored module (`import(... { with: { type: "file" } })`) that esbuild - * can't bundle; its own Node fallback resolves plain file paths anyway, and - * `FiletypeParserOptions.wasm`/`queries.highlights` accept local paths - * directly. So registration just points at the vendored files — resolved at - * RUNTIME by walking up from this module to the package root, which works from - * both the esbuild bundle (dist/main.js → ../parsers) and vitest's src tree - * (src/boundary → ../../parsers). + * Cache location: `HERMES_TUI_PARSER_CACHE` (set by the Python launcher to + * `~/.hermes/cache/opentui-parsers/`, profile-aware via get_hermes_home). When + * unset (dev/demo/CI), we leave OpenTUI's default data path + * (`$XDG_DATA_HOME/opentui` → `~/.local/share/opentui`) untouched. * - * Must run BEFORE the first ``/`` mount (they grab the global - * tree-sitter client lazily) — the entry imports + calls this at module load, - * ahead of renderer acquisition. Total: a missing assets dir degrades to - * core's bundled set (plain text for the extras), never a throw. + * `setDataPath()` on the GLOBAL client must run BEFORE the client initializes + * (it only mutates `options.dataPath` until init, then the worker boots with + * it). `addDefaultParsers()` must run BEFORE the first ``/`` + * mount (they grab the global client lazily and trigger init). The entry + * imports + calls `registerRemoteParsers()` at module load, ahead of renderer + * acquisition, so both orderings hold. + * + * Offline behavior: registration itself does NO network (it only declares the + * URL configs). The fetch happens on first highlight of a given language; if it + * fails (air-gapped / GitHub unreachable), OpenTUI degrades that filetype to + * plain text — never a throw. A registration error likewise degrades the whole + * extra set to plain text. */ -import { existsSync } from 'node:fs' -import { dirname, join } from 'node:path' -import { fileURLToPath } from 'node:url' - +import { getTreeSitterClient } from '@opentui/core' import { addDefaultParsers } from '@opentui/core' -import manifest from '../../parsers/manifest.json' +import manifest from './parsers.manifest.json' import { getLog } from './log.ts' interface ManifestParser { readonly filetype: string readonly aliases: readonly string[] + readonly wasm: string + readonly highlights: string } /** The registered parser configs (exported shape for tests/diagnostics). */ @@ -43,43 +49,44 @@ export interface RegisteredParser { queries: { highlights: string[] } } -/** Walk up from this module (bundle: dist/…; tests: src/boundary/…) to the - * package root's vendored `parsers/` dir. */ -function findParsersDir(): string | undefined { - let dir = dirname(fileURLToPath(import.meta.url)) - for (let hop = 0; hop < 5; hop += 1) { - const candidate = join(dir, 'parsers') - if (existsSync(join(candidate, 'manifest.json'))) return candidate - dir = dirname(dir) - } - return undefined +/** The cache dir for fetched grammar assets, or undefined to use OpenTUI's + * default ($XDG_DATA_HOME/opentui). The launcher sets this per-profile. */ +export function parserCacheDir(): string | undefined { + const dir = (process.env.HERMES_TUI_PARSER_CACHE ?? '').trim() + return dir.length ? dir : undefined } -/** Build the parser configs for every vendored grammar whose files exist. */ -export function vendoredParsers(parsersDir: string | undefined = findParsersDir()): RegisteredParser[] { - if (!parsersDir) return [] +/** Build the remote parser configs from the manifest. Pure — no network, no + * filesystem; just declares the URL configs OpenTUI fetches lazily. */ +export function remoteParsers(): RegisteredParser[] { const configs: RegisteredParser[] = [] for (const parser of (manifest as { parsers: ManifestParser[] }).parsers) { - const wasm = join(parsersDir, parser.filetype, `tree-sitter-${parser.filetype}.wasm`) - const highlights = join(parsersDir, parser.filetype, 'highlights.scm') - if (!existsSync(wasm) || !existsSync(highlights)) continue + if (!parser.wasm || !parser.highlights) continue configs.push({ filetype: parser.filetype, ...(parser.aliases.length ? { aliases: [...parser.aliases] } : {}), - wasm, - queries: { highlights: [highlights] } + wasm: parser.wasm, + queries: { highlights: [parser.highlights] } }) } return configs } -/** Register the vendored grammars with core's global default-parser list. +/** Point the global tree-sitter client's cache at our profile dir, then + * register the remote grammars with core's global default-parser list. * Returns what was registered (empty on any failure — plain-text fallback). */ -export function registerVendoredParsers(): RegisteredParser[] { +export function registerRemoteParsers(): RegisteredParser[] { try { - const parsers = vendoredParsers() + const cache = parserCacheDir() + if (cache) { + // Must precede the client's lazy initialize() (first / + // mount). Pre-init this only mutates options.dataPath; the returned + // promise resolves immediately (no worker yet) so we don't await it. + void getTreeSitterClient().setDataPath(cache) + } + const parsers = remoteParsers() if (!parsers.length) { - getLog().warn('parsers', 'no vendored tree-sitter grammars found — extras render plain', {}) + getLog().warn('parsers', 'no remote tree-sitter grammars declared — extras render plain', {}) return [] } addDefaultParsers(parsers) diff --git a/ui-opentui/src/entry/main.tsx b/ui-opentui/src/entry/main.tsx index d80860c67ce..057b8a3d823 100644 --- a/ui-opentui/src/entry/main.tsx +++ b/ui-opentui/src/entry/main.tsx @@ -32,7 +32,7 @@ import { getLog } from '../boundary/log.ts' import { startMemlog } from '../boundary/memlog.ts' import { startMemoryMonitor } from '../boundary/memoryMonitor.ts' import { startProactiveGc } from '../boundary/proactiveGc.ts' -import { registerVendoredParsers } from '../boundary/parsers.ts' +import { registerRemoteParsers } from '../boundary/parsers.ts' import { acquireRenderer } from '../boundary/renderer.ts' import { makeAppLayer } from '../boundary/runtime.ts' import { nthAssistantResponse } from '../logic/copy.ts' @@ -68,10 +68,11 @@ import { App } from '../view/App.tsx' import { seedLearnedNames } from '../view/composer.tsx' import { TerminalChrome } from '../view/terminalChrome.tsx' -// Syntax-highlighting language expansion: register the vendored tree-sitter +// Syntax-highlighting language expansion: register the remote tree-sitter // grammars (python/rust/go/bash/json/c/html/css/yaml/toml) before the first -// / mount initializes the global tree-sitter client. -registerVendoredParsers() +// / mount initializes the global tree-sitter client. Grammars +// are fetched from GitHub on first use and cached under HERMES_TUI_PARSER_CACHE. +registerRemoteParsers() import type { SessionPickerOps } from '../view/overlays/sessionPicker.tsx' import { ThemeProvider } from '../view/theme.tsx' import { makeFakeGatewayLayer, type FakeGatewayController } from './fakeGateway.ts' diff --git a/ui-opentui/src/test/parsers.test.tsx b/ui-opentui/src/test/parsers.test.tsx index b45b91c1bbb..4db2dc93fb6 100644 --- a/ui-opentui/src/test/parsers.test.tsx +++ b/ui-opentui/src/test/parsers.test.tsx @@ -1,49 +1,55 @@ /** - * Vendored tree-sitter grammar registration (syntax-highlighting language - * expansion). Layers: - * 1. config: every manifest grammar has valid vendored assets (wasm magic, - * non-empty query) and the built configs point at existing absolute paths. + * Remote tree-sitter grammar registration (syntax-highlighting language + * expansion). Grammars are NOT vendored — they're declared as remote URLs in + * parsers.manifest.json and fetched+cached by OpenTUI on first use. Layers: + * 1. config: every manifest grammar builds a well-formed config — a release + * `.wasm` URL + a highlights `.scm` URL (both https). * 2. resolution: core's filetype maps route our curated extensions/fence * labels to the registered filetype ids. - * Visual color is live-smoke territory (highlighting settles async — see - * codeBlock.tsx); these tests pin the wiring that makes it possible. + * Actual fetch + visual color is live-smoke territory (network + async settle — + * see codeBlock.tsx); these tests pin the wiring that makes it possible without + * hitting the network. */ -import { readFileSync } from 'node:fs' -import { isAbsolute } from 'node:path' - import { extToFiletype, infoStringToFiletype, pathToFiletype } from '@opentui/core' import { describe, expect, test } from 'vitest' -import { registerVendoredParsers, vendoredParsers } from '../boundary/parsers.ts' +import { parserCacheDir, registerRemoteParsers, remoteParsers } from '../boundary/parsers.ts' const EXPECTED = ['python', 'rust', 'go', 'bash', 'json', 'c', 'html', 'css', 'yaml', 'toml'] -describe('vendored grammar configs', () => { - test('all 10 curated grammars resolve with existing absolute asset paths', () => { - const configs = vendoredParsers() +describe('remote grammar configs', () => { + test('all 10 curated grammars build configs from the manifest', () => { + const configs = remoteParsers() expect(configs.map(c => c.filetype).sort()).toEqual([...EXPECTED].sort()) - for (const config of configs) { - expect(isAbsolute(config.wasm)).toBe(true) - expect(isAbsolute(config.queries.highlights[0]!)).toBe(true) + }) + + test('each config carries an https .wasm URL and a non-empty .scm URL', () => { + for (const config of remoteParsers()) { + expect(config.wasm, config.filetype).toMatch(/^https:\/\/.+\.wasm$/) + const scm = config.queries.highlights[0]! + expect(scm.startsWith('https://'), config.filetype).toBe(true) + expect(scm.endsWith('.scm'), config.filetype).toBe(true) } }) - test('vendored wasm files carry the wasm magic; queries are non-empty', () => { - for (const config of vendoredParsers()) { - const wasm = readFileSync(config.wasm) - expect(wasm.subarray(0, 4).toString('latin1'), config.filetype).toBe('\0asm') - const query = readFileSync(config.queries.highlights[0]!, 'utf8') - expect(query.trim().length, config.filetype).toBeGreaterThan(0) - } - }) - - test('registerVendoredParsers registers and reports the full set', () => { - const registered = registerVendoredParsers() + test('registerRemoteParsers registers and reports the full set', () => { + const registered = registerRemoteParsers() expect(registered.map(r => r.filetype).sort()).toEqual([...EXPECTED].sort()) }) - test('a missing assets dir degrades to empty (plain-text fallback), no throw', () => { - expect(vendoredParsers('/nonexistent/parsers')).toEqual([]) + test('parserCacheDir reflects HERMES_TUI_PARSER_CACHE (undefined when unset)', () => { + const prev = process.env.HERMES_TUI_PARSER_CACHE + try { + delete process.env.HERMES_TUI_PARSER_CACHE + expect(parserCacheDir()).toBeUndefined() + process.env.HERMES_TUI_PARSER_CACHE = '/tmp/hermes-parser-cache' + expect(parserCacheDir()).toBe('/tmp/hermes-parser-cache') + process.env.HERMES_TUI_PARSER_CACHE = ' ' + expect(parserCacheDir()).toBeUndefined() + } finally { + if (prev === undefined) delete process.env.HERMES_TUI_PARSER_CACHE + else process.env.HERMES_TUI_PARSER_CACHE = prev + } }) })