Chased the "stream frame p95 = 60ms with ZERO longtasks" mystery to its actual
cause: the default stream chunk had no paragraph breaks, so it grew into one
giant ~22KB block that re-rendered fully every flush — defeating the block
memoization real streaming relies on. Plain text = 21ms; realistic chunk with
`\n\n` breaks (blocks settle, only the tail re-renders) = 23ms. Fixed the
default chunk to model real LLM output; a break-less `--chunk` remains available
as a single-block worst-case stress.
Also hardened the isolated instance so measurements reflect real cost:
- Wait for the gateway socket to actually connect before measuring (a booting/
absent backend's reconnect backoff churns the main thread). Exposed via a new
__PERF_DRIVE__.connected() probe reading $gateway.connectionState.
- Focus emulation + anti-throttle/occlusion flags so a backgrounded perf window
isn't frame-throttled (no OS focus stealing).
- Generation-guarded the rAF frame recorder so repeated runs don't leave
overlapping recorders polluting frame intervals.
Baseline re-captured as the median of 5 --spawn runs (darwin-arm64); all three
CI scenarios now green and stable. Absolute values are dev-build (noted in
_meta) — regression guards, not shipped numbers.
- Resolve the vite CLI via vite/package.json `bin` (Vite 8's exports block
importing vite/bin/vite.js directly — --spawn failed with ERR_PACKAGE_PATH_NOT_EXPORTED).
- Add a post-launch settle so cold-start contention (vite dep pre-bundling,
first backend-connect attempts) doesn't contaminate the first scenario.
- Drop the raw autolink from the default stream chunk (resolvable URLs trigger
link-embed DNS lookups unrelated to render cost).
- Replace seed baseline with real numbers from a darwin-arm64 --spawn run.
keystroke + transcript are clean; stream is a clean single-run capture (the
isolated backend may not connect, and its reconnect churn inflates frame
pacing — re-capture on a connected instance for tighter tolerances).
Replaces the dozen ad-hoc measure-*/profile-* scripts (each reinventing the
CDP client — 4 different copies — plus its own arg parsing, stats, output
path, and none with a baseline) with one framework under scripts/perf/:
- lib/cdp.mjs one CDP client + target discovery + typing + CPU-profile wrapper + DOM selectors
- lib/stats.mjs percentiles, histograms, CPU-profile self-time ranking
- lib/baseline.mjs load/compare/update baseline + regression gate (new capability)
- lib/launch.mjs attach, OR spawn a fully ISOLATED instance
- scenarios/* one module per measurement, registered in scenarios/index.mjs
- run.mjs / serve.mjs, baseline.json, README.md
Isolation solves the long-standing measurement blocker: a running `hgui` held
the Electron single-instance lock, so a second instance quit. `--spawn` /
`perf:serve` launch with their own --user-data-dir (separate lock scope), their
own HERMES_HOME (separate backend/sessions, config seeded from ~/.hermes so it
reaches a chat view without onboarding), and their own --remote-debugging-port.
Synthetic scenarios drive $messages via window.__PERF_DRIVE__, so no LLM credits.
Scenario -> sunset script mapping:
stream <- measure-synthetic-stream, profile-synth-stream, profile-long-stream
stream --real <- measure-real-stream, profile-real-stream
keystroke <- measure-latency, profile-typing, leak-typing
transcript <- (new: long-transcript mount cost)
submit <- measure-submit, measure-jump
session-switch <- profile-session-switch
profile-switch <- measure-profile-switch
CPU profiling is now a cross-cutting --cpuprofile flag, not 5 separate scripts.
CI-tier scenarios (stream, keystroke, transcript) need no backend/credits and
are gated against baseline.json (seed values; re-capture with --update-baseline
on a reference device). Backend-tier scenarios are report-only.
perf-probe.tsx gains loadTranscript() for the transcript scenario. No core
files touched; isolation is via CLI args, not env-gated app changes.
Verified: node --check all modules, tsc, eslint, and a unit smoke of the
stats + regression-gate logic. The end-to-end GUI run (which opens a window)
is left to run interactively via `npm run perf -- --spawn`.
