Profiling the boot answered "is there a real cold-start win?": no wasteful
hotspot — the renderer does only ~tens of ms of work at mount, no heavy library
(shiki/mermaid/katex/d3/motion) initializes at startup; the rest is Electron
runtime + waiting, near the Electron floor.
It also exposed that the cold-start number was pessimistic: a fresh
--user-data-dir per run means a COLD V8 code cache and worst-case bundle
recompile every launch. Real users reuse their profile. Measured delta:
fresh (cold cache): spawn→interactive ~1.48s
reused (warm cache): ~1.0s
So representative launch is ~1.0s; only first-launch-after-install pays ~+400ms.
- coldStartSamples() reuses one profile (run 0 warms the cache, discarded;
runs 1..N are warm samples), stepping ports + pausing so the single-instance
lock releases. `--cold-fresh` measures the first-launch worst case.
- Re-baselined cold-start with the representative warm numbers.
Net: nothing high-ROI left to optimize. The only lever is shipping a pre-warmed
V8 code cache to make first launch match warm (~400ms, once per update) — real
packaging complexity for a marginal win, deliberately not pursued.
* bench(desktop): measure the full picture — prod build, cold-start, first-token
Stop drip-feeding scenarios: extend the harness to cover the latencies that
actually dominate perceived speed, and measure them on a REAL production build.
- --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1,
off in normal builds) and launch it from dist/. Measures minified React, so
numbers are representative shipped figures instead of ~3x-inflated dev ones.
- cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a
fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms.
- first-token scenario (backend tier): Enter → first assistant token painted —
the TTFT latency an agent app is uniquely judged on.
- run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates
ci+cold tiers against the baseline.
Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all
green. Representative numbers:
cold-start spawn→interactive ~1.6s, FCP ~0.5s
stream frame p95 22ms, 1 longtask
keystroke p50 2ms, p95 8.7ms
transcript mount 145ms, 82ms longtask (400-msg open)
The prod build also settled the open question from the dev numbers: the
transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not
actionable. Measurement did its job.
* bench(desktop): trustworthy cold-start measurement (code-splitting is NOT the lever)
Investigated code-splitting the ~22MB renderer bundle to cut cold start. It is
the wrong fix on both counts:
1. Intentional design: vite.config disables codeSplitting because Shiki emits
thousands of dynamic chunks and electron-builder OOMs scanning them — a
packaging/installer constraint, not an oversight.
2. The data says it wouldn't help. Fixing the cold-start measurement to be
trustworthy and reading the boot composition (prod build):
spawn → interactive ~1.5s
renderer nav → DOMInteractive ~0.8s, → DOMContentLoaded ~1.06s
so the whole 22MB bundle EVAL is only ~0.27s (DCL − DOMInteractive) of the
~1.5s. The dominant costs are Electron/window startup and React app mount —
neither touched by splitting.
The measurement fixes (the real content of this PR — no app change, since the
optimization was rejected):
- Drop HERMES_DESKTOP_BOOT_FAKE from spawned instances — it injected artificial
per-phase boot-overlay sleeps that inflated cold-start (and slowed every run).
- Unique debug/dev port per cold-start run — a just-killed instance can hold
:9222 briefly, so reusing it made CDP attach to the DYING instance and report
garbage (spawn_to_cdp of ~4ms). Stepping the port per run fixes the race.
- Richer boot marks (dom_interactive, dom_content_loaded, main-script size) so
cold-start composition is visible, not just a single number.
- Forward all numeric boot marks from the cold-start loop.
- Re-baseline cold-start with the clean numbers.
A real cold-start win would target Electron startup / app-mount (e.g. V8 code
cache, deferred non-critical mount) — a future pass, now that it's measurable.
Stop drip-feeding scenarios: extend the harness to cover the latencies that
actually dominate perceived speed, and measure them on a REAL production build.
- --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1,
off in normal builds) and launch it from dist/. Measures minified React, so
numbers are representative shipped figures instead of ~3x-inflated dev ones.
- cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a
fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms.
- first-token scenario (backend tier): Enter → first assistant token painted —
the TTFT latency an agent app is uniquely judged on.
- run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates
ci+cold tiers against the baseline.
Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all
green. Representative numbers:
cold-start spawn→interactive ~1.6s, FCP ~0.5s
stream frame p95 22ms, 1 longtask
keystroke p50 2ms, p95 8.7ms
transcript mount 145ms, 82ms longtask (400-msg open)
The prod build also settled the open question from the dev numbers: the
transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not
actionable. Measurement did its job.
Chased the "stream frame p95 = 60ms with ZERO longtasks" mystery to its actual
cause: the default stream chunk had no paragraph breaks, so it grew into one
giant ~22KB block that re-rendered fully every flush — defeating the block
memoization real streaming relies on. Plain text = 21ms; realistic chunk with
`\n\n` breaks (blocks settle, only the tail re-renders) = 23ms. Fixed the
default chunk to model real LLM output; a break-less `--chunk` remains available
as a single-block worst-case stress.
Also hardened the isolated instance so measurements reflect real cost:
- Wait for the gateway socket to actually connect before measuring (a booting/
absent backend's reconnect backoff churns the main thread). Exposed via a new
__PERF_DRIVE__.connected() probe reading $gateway.connectionState.
- Focus emulation + anti-throttle/occlusion flags so a backgrounded perf window
isn't frame-throttled (no OS focus stealing).
- Generation-guarded the rAF frame recorder so repeated runs don't leave
overlapping recorders polluting frame intervals.
Baseline re-captured as the median of 5 --spawn runs (darwin-arm64); all three
CI scenarios now green and stable. Absolute values are dev-build (noted in
_meta) — regression guards, not shipped numbers.
- Resolve the vite CLI via vite/package.json `bin` (Vite 8's exports block
importing vite/bin/vite.js directly — --spawn failed with ERR_PACKAGE_PATH_NOT_EXPORTED).
- Add a post-launch settle so cold-start contention (vite dep pre-bundling,
first backend-connect attempts) doesn't contaminate the first scenario.
- Drop the raw autolink from the default stream chunk (resolvable URLs trigger
link-embed DNS lookups unrelated to render cost).
- Replace seed baseline with real numbers from a darwin-arm64 --spawn run.
keystroke + transcript are clean; stream is a clean single-run capture (the
isolated backend may not connect, and its reconnect churn inflates frame
pacing — re-capture on a connected instance for tighter tolerances).
Replaces the dozen ad-hoc measure-*/profile-* scripts (each reinventing the
CDP client — 4 different copies — plus its own arg parsing, stats, output
path, and none with a baseline) with one framework under scripts/perf/:
- lib/cdp.mjs one CDP client + target discovery + typing + CPU-profile wrapper + DOM selectors
- lib/stats.mjs percentiles, histograms, CPU-profile self-time ranking
- lib/baseline.mjs load/compare/update baseline + regression gate (new capability)
- lib/launch.mjs attach, OR spawn a fully ISOLATED instance
- scenarios/* one module per measurement, registered in scenarios/index.mjs
- run.mjs / serve.mjs, baseline.json, README.md
Isolation solves the long-standing measurement blocker: a running `hgui` held
the Electron single-instance lock, so a second instance quit. `--spawn` /
`perf:serve` launch with their own --user-data-dir (separate lock scope), their
own HERMES_HOME (separate backend/sessions, config seeded from ~/.hermes so it
reaches a chat view without onboarding), and their own --remote-debugging-port.
Synthetic scenarios drive $messages via window.__PERF_DRIVE__, so no LLM credits.
Scenario -> sunset script mapping:
stream <- measure-synthetic-stream, profile-synth-stream, profile-long-stream
stream --real <- measure-real-stream, profile-real-stream
keystroke <- measure-latency, profile-typing, leak-typing
transcript <- (new: long-transcript mount cost)
submit <- measure-submit, measure-jump
session-switch <- profile-session-switch
profile-switch <- measure-profile-switch
CPU profiling is now a cross-cutting --cpuprofile flag, not 5 separate scripts.
CI-tier scenarios (stream, keystroke, transcript) need no backend/credits and
are gated against baseline.json (seed values; re-capture with --update-baseline
on a reference device). Backend-tier scenarios are report-only.
perf-probe.tsx gains loadTranscript() for the transcript scenario. No core
files touched; isolation is via CLI args, not env-gated app changes.
Verified: node --check all modules, tsc, eslint, and a unit smoke of the
stats + regression-gate logic. The end-to-end GUI run (which opens a window)
is left to run interactively via `npm run perf -- --spawn`.
