a894879d28
Captures medians of 5 runs for multitab and render-churn so tonight's
wins can't silently regress.
idle-cost is deliberately NOT gated. Its render attribution and idle
commit rate are trustworthy and are what the scenario exists for, but the
drag fps it reports (~0.6fps, p95 814ms) contradicts a direct
single-clock probe of the same gesture on the same build (57fps). I ruled
out sash selection, tile setup, render-counter residue, and a 20s soak,
and could not explain the gap — so the metric ships as a report, not a
gate. Gating CI on a number I can't defend would either fire on a phantom
or mask a real stall.
tier: 'report' is outside GATED ('ci','cold'), so the scenario still runs
and prints but neither compares nor writes a baseline.
4.6 KiB
Desktop perf harness
One systematized way to measure desktop rendering/interaction performance,
diff it against a committed baseline, and fail on regressions. It replaces the
dozen one-off measure-* / profile-* scripts that each reinvented the CDP
client, arg parsing, stats, and output (and never had a baseline).
Quick start
# Isolated instance (recommended) — no running app or LLM credits needed.
# Its own --user-data-dir + HERMES_HOME means it never collides with `hgui`.
npm run perf -- --spawn
# Or: launch an isolated instance once, attach repeatedly (faster iteration).
npm run perf:serve # leaves an instance on :9222
npm run perf # attaches, runs the CI suite, gates on baseline
# One scenario, with a CPU profile:
npm run perf -- stream --cpuprofile --tokens 800
# Representative PRODUCTION numbers (minified React, not the ~3x-slower dev build):
npm run perf -- cold-start stream keystroke transcript --spawn --prod
# Re-capture the baseline on your reference device, then commit baseline.json:
npm run perf -- cold-start stream keystroke transcript --spawn --prod --update-baseline
Dev vs prod
By default the harness measures the dev renderer (fast to spin up, good for
relative regression checks). Pass --prod (with --spawn) to build a
production renderer with the probe included (VITE_PERF_PROBE=1) and measure
minified React — the representative shipped numbers. The committed baseline is
captured with --prod.
Why isolation matters
The measurement this harness exists to run was historically blocked: a running
hgui holds the Electron single-instance lock, so a second instance quit
immediately. --spawn / perf:serve launch with their own --user-data-dir
(separate lock scope), their own HERMES_HOME (separate backend + sessions),
and their own --remote-debugging-port. Synthetic scenarios drive $messages
directly via window.__PERF_DRIVE__, so no LLM credits are spent.
Scenarios
| scenario | tier | measures | replaces |
|---|---|---|---|
stream |
ci | streaming longtasks, frame p95/p99, mutation cadence | measure-synthetic-stream, profile-synth-stream, profile-long-stream |
stream --real |
backend | same, from a real LLM stream | measure-real-stream, profile-real-stream |
keystroke |
ci | composer keystroke → paint latency | measure-latency, profile-typing, leak-typing |
transcript |
ci | large-transcript mount + paint cost | (new) |
render-churn |
ci | per-component render attribution + store churn while N tabs stream | (new) |
idle-cost |
report | busy-but-silent tiles: idle commit rate, + fps while resizing / typing | (new) |
cold-start |
cold | launch → CDP → driver → first paint (fresh spawn/run) | (new) |
first-token |
backend | Enter → first assistant token painted (TTFT) | (new) |
submit |
backend | Enter → cleared → user msg painted, scroll jump | measure-submit, measure-jump |
session-switch |
backend | route → first-paint → settle | profile-session-switch |
profile-switch |
backend | rail click → sidebar settled | measure-profile-switch |
ci + cold scenarios need no backend/credits and are gated against
baseline.json (cold-start requires --spawn since it measures a fresh
launch, and must be run in its own invocation). backend scenarios need a live
backend (and --spawn or a real session/credits) and are report-only.
CPU profiling is a cross-cutting --cpuprofile flag on any scenario (it wraps
the run in Profiler.start/stop and prints a top-self-time table), replacing
every standalone profile-* script.
Adding a scenario
Create scenarios/<name>.mjs exporting { name, tier, description, run(cdp, opts) }
where run returns { metrics, detail } (metrics = flat numbers, lower is
better), then register it in scenarios/index.mjs. If it's ci, add a
baseline.json entry (or run --update-baseline).
Layout
lib/cdp.mjs— the one CDP client + target discovery + typing + CPU-profile wrapper + DOM selectors.lib/stats.mjs— percentiles, histograms, CPU-profile self-time ranking.lib/baseline.mjs— load/compare/update the baseline + regression gate.lib/launch.mjs— attach, or spawn a fully isolated instance.scenarios/— one module per measurement.run.mjs— entrypoint.serve.mjs— standalone isolated launcher.
Not migrated (kept as dev utilities)
eval.mjs, reload.mjs, reload-renderer.mjs, probe-renderer.mjs,
probe-thread.mjs, click-session.mjs, diag-*.mjs are interactive dev
helpers, not benchmarks. They can adopt lib/cdp.mjs in a follow-up.