Captures medians of 5 runs for multitab and render-churn so tonight's
wins can't silently regress.
idle-cost is deliberately NOT gated. Its render attribution and idle
commit rate are trustworthy and are what the scenario exists for, but the
drag fps it reports (~0.6fps, p95 814ms) contradicts a direct
single-clock probe of the same gesture on the same build (57fps). I ruled
out sash selection, tile setup, render-counter residue, and a 20s soak,
and could not explain the gap — so the metric ships as a report, not a
gate. Gating CI on a number I can't defend would either fire on a phantom
or mask a real stall.
tier: 'report' is outside GATED ('ci','cold'), so the scenario still runs
and prints but neither compares nor writes a baseline.
withFrames ran its own requestAnimationFrame ticker while the gesture body
independently awaited rAF per step. Two rAF consumers, so the observer's
deltas counted the driver's frames as well as the app's — it reported
~3fps for a drag that a single-clock probe measures at ~23fps, and it
never moved no matter what got fixed underneath.
Timing now comes from the same callbacks the body drives (__MARK__).
This also fixes a silent false-negative on the typing pass: it paced on
setTimeout, so the independent ticker was mostly sampling idle waits
between keystrokes and reported a flat 61fps. On the driving clock the
same interaction reports ~30fps with 27 of 40 frames over 33ms — which
matches the 'typing feels slow' symptom I previously could not reproduce.
TYPE now records __TYPE_TARGET__ and the runner throws when no composer is
found, so a pass that measures nothing fails loudly instead of scoring a
perfect 0 deficit — same guard DRAG already had.
TreeGroup called useStore($layoutTree) to build its right-click menu's
move/split directions. That subscribes every zone — and therefore every
mounted pane and its entire transcript — to the whole layout tree. A sash
drag rewrites the tree once per frame, so dragging the sidebar re-rendered
all five tiles' message lists on every pointermove, for a context menu
nobody had open.
The directions are only read when the menu renders, so read the tree there
with .get() instead. Same lazy shape the neighbouring `closable` prop
already uses.
Measured over one 60px sash drag with five busy tiles:
commits 83 -> 12
ChatView 150 -> 10 (4465ms -> 353ms)
AuiProvider 9450 -> 630 (9868ms -> 774ms)
TreeGroup 180 -> 12
TreeSplit 90 -> 6
Also fixes an observer effect in the harness: idle-cost recorded render
attribution *during* the timed gesture, and the counter walks the fiber
tree on every commit. That was large enough to hide this 15x reduction
behind an unchanged fps, so timing and attribution are separate passes now
and `record` defaults off.
Adds scripts/diag-drag-churn.mjs — the probe that found this. It reports
the transcript chain (who above the messages re-rendered) plus every atom
that notified, which is what named TreeGroup instead of leaving it to be
guessed at. Notably the atom list came back EMPTY: this was never store
churn, so the render-attribution path was the only thing that could have
found it.
The synthetic gesture oscillated +/-3px, which nets to zero displacement
and can clamp to a no-op — so it reported a confident fps number for a
drag that never moved the sash. Sweeps monotonically now, dispatches
pointer events React's synthetic system accepts (isPrimary/button/buttons),
and records dragTarget + dragMoved so a drag that silently did nothing is
visible in the output rather than passing as a measurement.
Verified: dragMoved now reports 60px where it previously reported 0.
Adds an `idle-cost` scenario for the symptom Brooklyn reported: with a
thread spinning, resizing the sidebar feels slow. It holds N tiles busy,
pushes NO tokens, and measures the renderer's self-inflicted commit rate
plus fps while dragging the splitter and while typing.
It reproduces immediately. Five busy tiles, nothing streaming:
idle commits 17.7/sec (should be 0 — nothing is happening)
drag 1.4 fps p95 812ms, worst frame 1.9s
typing 61 fps (fine — this is specific to resize)
Attributing the drag window showed 105,385 TooltipProvider renders and
~15s of component time across a 60-frame gesture. Cause: `Tip` mounts a
full Radix provider + Tooltip per call site, and there are ~107 of them.
Radix's Tooltip holds real state and Popper subscribes to layout, so an
unrelated interaction re-rendered all of them.
Mounts the machinery lazily instead, on first hover/focus. Tooltip churn
drops ~4x (105k -> 26k) and drag doubles to 3fps.
Note `defaultOpen` on the armed Tooltip is load-bearing: the pointerenter
that armed it has already fired, so Radix never sees it and the tip mounts
silently closed. A test caught exactly that, and now guards it.
3fps is still bad — the remaining cost is the whole transcript
re-rendering per resize frame (MessagePrimitive.Parts 12,600 renders /
10.5s, Block/Ct 24,300 each, all 100% wasted). Separate fix.
Two problems, both found by distrusting the harness's own numbers.
1. The scenario slept a fixed 1s after mounting tabs, then recorded. Boot
and session hydration are not reliably done by then, so a variable
amount of unrelated work landed inside the measurement window. Three
back-to-back runs on identical code spread 2.2x on total_renders and
3.8x on wasted_renders — wide enough that a single-run before/after
delta could be mostly noise. Replaced with a quiesce gate that waits
for commits to hold still before recording, and reports 'quiet:N' or
'timeout:...' so a contaminated run is visible instead of silent.
2. The counter attributed a context-driven re-render as 'wasted', which
pointed at memo() as the fix when memo cannot block context at all.
Adds contextChanged via the fiber's context dependency list, and
excludes it from wasted.
The gate also turned up a finding worth more than the fix: with five busy
tiles and NO driver running, the renderer still commits ~18x/sec. The
report now names the cascade roots (own state changed, props did not)
rather than leaving them to be guessed at — Streamdown re-renders itself
105 times while idle, which is what drives Block/Ct.
Drives the same synthetic multi-tab streaming workload as `multitab`
(publishSessionState per session per flush, no backend, no credits) but
reports render attribution instead of frame pacing — sidebar_renders,
wasted_renders, and wasted_notifies.
Answers 'does the sidebar re-render while an agent is typing' directly.
The existing `stream` scenario measures streaming into an empty thread, which
is exactly the case where per-delta transcript work does not show up. This adds
`--historyTurns`, which mounts a settled transcript and lets it drain BEFORE
the recorders start, so the measurement window contains streaming work only —
and `stream-history`, a report-only preset that turns it on.
Report-only (tier: manual) rather than gated: how much history a host can mount
varies, so the absolute number is not comparable across machines. It is meant
for same-machine before/after runs.
Co-authored-by: Jakub Wolniewicz <frizikk@users.noreply.github.com>
N session tiles stacked as tabs, all mounted (keep-alive) and all
streaming concurrently through the real publishSessionState path — the
"several PR reviews at once" workload. Frame pacing + longtask metrics,
no backend or credits needed; the workload that exposed both fixes above
and the regression gate that keeps them fixed.
Profiling the boot answered "is there a real cold-start win?": no wasteful
hotspot — the renderer does only ~tens of ms of work at mount, no heavy library
(shiki/mermaid/katex/d3/motion) initializes at startup; the rest is Electron
runtime + waiting, near the Electron floor.
It also exposed that the cold-start number was pessimistic: a fresh
--user-data-dir per run means a COLD V8 code cache and worst-case bundle
recompile every launch. Real users reuse their profile. Measured delta:
fresh (cold cache): spawn→interactive ~1.48s
reused (warm cache): ~1.0s
So representative launch is ~1.0s; only first-launch-after-install pays ~+400ms.
- coldStartSamples() reuses one profile (run 0 warms the cache, discarded;
runs 1..N are warm samples), stepping ports + pausing so the single-instance
lock releases. `--cold-fresh` measures the first-launch worst case.
- Re-baselined cold-start with the representative warm numbers.
Net: nothing high-ROI left to optimize. The only lever is shipping a pre-warmed
V8 code cache to make first launch match warm (~400ms, once per update) — real
packaging complexity for a marginal win, deliberately not pursued.
* bench(desktop): measure the full picture — prod build, cold-start, first-token
Stop drip-feeding scenarios: extend the harness to cover the latencies that
actually dominate perceived speed, and measure them on a REAL production build.
- --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1,
off in normal builds) and launch it from dist/. Measures minified React, so
numbers are representative shipped figures instead of ~3x-inflated dev ones.
- cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a
fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms.
- first-token scenario (backend tier): Enter → first assistant token painted —
the TTFT latency an agent app is uniquely judged on.
- run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates
ci+cold tiers against the baseline.
Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all
green. Representative numbers:
cold-start spawn→interactive ~1.6s, FCP ~0.5s
stream frame p95 22ms, 1 longtask
keystroke p50 2ms, p95 8.7ms
transcript mount 145ms, 82ms longtask (400-msg open)
The prod build also settled the open question from the dev numbers: the
transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not
actionable. Measurement did its job.
* bench(desktop): trustworthy cold-start measurement (code-splitting is NOT the lever)
Investigated code-splitting the ~22MB renderer bundle to cut cold start. It is
the wrong fix on both counts:
1. Intentional design: vite.config disables codeSplitting because Shiki emits
thousands of dynamic chunks and electron-builder OOMs scanning them — a
packaging/installer constraint, not an oversight.
2. The data says it wouldn't help. Fixing the cold-start measurement to be
trustworthy and reading the boot composition (prod build):
spawn → interactive ~1.5s
renderer nav → DOMInteractive ~0.8s, → DOMContentLoaded ~1.06s
so the whole 22MB bundle EVAL is only ~0.27s (DCL − DOMInteractive) of the
~1.5s. The dominant costs are Electron/window startup and React app mount —
neither touched by splitting.
The measurement fixes (the real content of this PR — no app change, since the
optimization was rejected):
- Drop HERMES_DESKTOP_BOOT_FAKE from spawned instances — it injected artificial
per-phase boot-overlay sleeps that inflated cold-start (and slowed every run).
- Unique debug/dev port per cold-start run — a just-killed instance can hold
:9222 briefly, so reusing it made CDP attach to the DYING instance and report
garbage (spawn_to_cdp of ~4ms). Stepping the port per run fixes the race.
- Richer boot marks (dom_interactive, dom_content_loaded, main-script size) so
cold-start composition is visible, not just a single number.
- Forward all numeric boot marks from the cold-start loop.
- Re-baseline cold-start with the clean numbers.
A real cold-start win would target Electron startup / app-mount (e.g. V8 code
cache, deferred non-critical mount) — a future pass, now that it's measurable.
Stop drip-feeding scenarios: extend the harness to cover the latencies that
actually dominate perceived speed, and measure them on a REAL production build.
- --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1,
off in normal builds) and launch it from dist/. Measures minified React, so
numbers are representative shipped figures instead of ~3x-inflated dev ones.
- cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a
fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms.
- first-token scenario (backend tier): Enter → first assistant token painted —
the TTFT latency an agent app is uniquely judged on.
- run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates
ci+cold tiers against the baseline.
Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all
green. Representative numbers:
cold-start spawn→interactive ~1.6s, FCP ~0.5s
stream frame p95 22ms, 1 longtask
keystroke p50 2ms, p95 8.7ms
transcript mount 145ms, 82ms longtask (400-msg open)
The prod build also settled the open question from the dev numbers: the
transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not
actionable. Measurement did its job.
Chased the "stream frame p95 = 60ms with ZERO longtasks" mystery to its actual
cause: the default stream chunk had no paragraph breaks, so it grew into one
giant ~22KB block that re-rendered fully every flush — defeating the block
memoization real streaming relies on. Plain text = 21ms; realistic chunk with
`\n\n` breaks (blocks settle, only the tail re-renders) = 23ms. Fixed the
default chunk to model real LLM output; a break-less `--chunk` remains available
as a single-block worst-case stress.
Also hardened the isolated instance so measurements reflect real cost:
- Wait for the gateway socket to actually connect before measuring (a booting/
absent backend's reconnect backoff churns the main thread). Exposed via a new
__PERF_DRIVE__.connected() probe reading $gateway.connectionState.
- Focus emulation + anti-throttle/occlusion flags so a backgrounded perf window
isn't frame-throttled (no OS focus stealing).
- Generation-guarded the rAF frame recorder so repeated runs don't leave
overlapping recorders polluting frame intervals.
Baseline re-captured as the median of 5 --spawn runs (darwin-arm64); all three
CI scenarios now green and stable. Absolute values are dev-build (noted in
_meta) — regression guards, not shipped numbers.
- Resolve the vite CLI via vite/package.json `bin` (Vite 8's exports block
importing vite/bin/vite.js directly — --spawn failed with ERR_PACKAGE_PATH_NOT_EXPORTED).
- Add a post-launch settle so cold-start contention (vite dep pre-bundling,
first backend-connect attempts) doesn't contaminate the first scenario.
- Drop the raw autolink from the default stream chunk (resolvable URLs trigger
link-embed DNS lookups unrelated to render cost).
- Replace seed baseline with real numbers from a darwin-arm64 --spawn run.
keystroke + transcript are clean; stream is a clean single-run capture (the
isolated backend may not connect, and its reconnect churn inflates frame
pacing — re-capture on a connected instance for tighter tolerances).
Replaces the dozen ad-hoc measure-*/profile-* scripts (each reinventing the
CDP client — 4 different copies — plus its own arg parsing, stats, output
path, and none with a baseline) with one framework under scripts/perf/:
- lib/cdp.mjs one CDP client + target discovery + typing + CPU-profile wrapper + DOM selectors
- lib/stats.mjs percentiles, histograms, CPU-profile self-time ranking
- lib/baseline.mjs load/compare/update baseline + regression gate (new capability)
- lib/launch.mjs attach, OR spawn a fully ISOLATED instance
- scenarios/* one module per measurement, registered in scenarios/index.mjs
- run.mjs / serve.mjs, baseline.json, README.md
Isolation solves the long-standing measurement blocker: a running `hgui` held
the Electron single-instance lock, so a second instance quit. `--spawn` /
`perf:serve` launch with their own --user-data-dir (separate lock scope), their
own HERMES_HOME (separate backend/sessions, config seeded from ~/.hermes so it
reaches a chat view without onboarding), and their own --remote-debugging-port.
Synthetic scenarios drive $messages via window.__PERF_DRIVE__, so no LLM credits.
Scenario -> sunset script mapping:
stream <- measure-synthetic-stream, profile-synth-stream, profile-long-stream
stream --real <- measure-real-stream, profile-real-stream
keystroke <- measure-latency, profile-typing, leak-typing
transcript <- (new: long-transcript mount cost)
submit <- measure-submit, measure-jump
session-switch <- profile-session-switch
profile-switch <- measure-profile-switch
CPU profiling is now a cross-cutting --cpuprofile flag, not 5 separate scripts.
CI-tier scenarios (stream, keystroke, transcript) need no backend/credits and
are gated against baseline.json (seed values; re-capture with --update-baseline
on a reference device). Backend-tier scenarios are report-only.
perf-probe.tsx gains loadTranscript() for the transcript scenario. No core
files touched; isolation is via CLI args, not env-gated app changes.
Verified: node --check all modules, tsc, eslint, and a unit smoke of the
stats + regression-gate logic. The end-to-end GUI run (which opens a window)
is left to run interactively via `npm run perf -- --spawn`.
