mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-07-21 16:18:55 +00:00

bench(desktop): measure the full picture — prod build, cold-start, first-token (#67697 )

Stop drip-feeding scenarios: extend the harness to cover the latencies that
actually dominate perceived speed, and measure them on a REAL production build.

- --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1,
  off in normal builds) and launch it from dist/. Measures minified React, so
  numbers are representative shipped figures instead of ~3x-inflated dev ones.
- cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a
  fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms.
- first-token scenario (backend tier): Enter → first assistant token painted —
  the TTFT latency an agent app is uniquely judged on.
- run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates
  ci+cold tiers against the baseline.

Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all
green. Representative numbers:
  cold-start  spawn→interactive ~1.6s, FCP ~0.5s
  stream      frame p95 22ms, 1 longtask
  keystroke   p50 2ms, p95 8.7ms
  transcript  mount 145ms, 82ms longtask (400-msg open)

The prod build also settled the open question from the dev numbers: the
transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not
actionable. Measurement did its job.

2026-07-19 17:52:39 -05:00

4.4 KiB

Raw Blame History

Desktop perf harness

One systematized way to measure desktop rendering/interaction performance, diff it against a committed baseline, and fail on regressions. It replaces the dozen one-off measure-* / profile-* scripts that each reinvented the CDP client, arg parsing, stats, and output (and never had a baseline).

Quick start

# Isolated instance (recommended) — no running app or LLM credits needed.
# Its own --user-data-dir + HERMES_HOME means it never collides with `hgui`.
npm run perf -- --spawn

# Or: launch an isolated instance once, attach repeatedly (faster iteration).
npm run perf:serve            # leaves an instance on :9222
npm run perf                  # attaches, runs the CI suite, gates on baseline

# One scenario, with a CPU profile:
npm run perf -- stream --cpuprofile --tokens 800

# Representative PRODUCTION numbers (minified React, not the ~3x-slower dev build):
npm run perf -- cold-start stream keystroke transcript --spawn --prod

# Re-capture the baseline on your reference device, then commit baseline.json:
npm run perf -- cold-start stream keystroke transcript --spawn --prod --update-baseline

Dev vs prod

By default the harness measures the dev renderer (fast to spin up, good for relative regression checks). Pass --prod (with --spawn) to build a production renderer with the probe included (VITE_PERF_PROBE=1) and measure minified React — the representative shipped numbers. The committed baseline is captured with --prod.

Why isolation matters

The measurement this harness exists to run was historically blocked: a running hgui holds the Electron single-instance lock, so a second instance quit immediately. --spawn / perf:serve launch with their own --user-data-dir (separate lock scope), their own HERMES_HOME (separate backend + sessions), and their own --remote-debugging-port. Synthetic scenarios drive $messages directly via window.__PERF_DRIVE__, so no LLM credits are spent.

Scenarios

scenario	tier	measures	replaces
`stream`	ci	streaming longtasks, frame p95/p99, mutation cadence	measure-synthetic-stream, profile-synth-stream, profile-long-stream
`stream --real`	backend	same, from a real LLM stream	measure-real-stream, profile-real-stream
`keystroke`	ci	composer keystroke → paint latency	measure-latency, profile-typing, leak-typing
`transcript`	ci	large-transcript mount + paint cost	(new)
`cold-start`	cold	launch → CDP → driver → first paint (fresh spawn/run)	(new)
`first-token`	backend	Enter → first assistant token painted (TTFT)	(new)
`submit`	backend	Enter → cleared → user msg painted, scroll jump	measure-submit, measure-jump
`session-switch`	backend	route → first-paint → settle	profile-session-switch
`profile-switch`	backend	rail click → sidebar settled	measure-profile-switch

ci + cold scenarios need no backend/credits and are gated against baseline.json (cold-start requires --spawn since it measures a fresh launch, and must be run in its own invocation). backend scenarios need a live backend (and --spawn or a real session/credits) and are report-only.

CPU profiling is a cross-cutting --cpuprofile flag on any scenario (it wraps the run in Profiler.start/stop and prints a top-self-time table), replacing every standalone profile-* script.

Adding a scenario

Create scenarios/<name>.mjs exporting { name, tier, description, run(cdp, opts) } where run returns { metrics, detail } (metrics = flat numbers, lower is better), then register it in scenarios/index.mjs. If it's ci, add a baseline.json entry (or run --update-baseline).

Layout

lib/cdp.mjs — the one CDP client + target discovery + typing + CPU-profile wrapper + DOM selectors.
lib/stats.mjs — percentiles, histograms, CPU-profile self-time ranking.
lib/baseline.mjs — load/compare/update the baseline + regression gate.
lib/launch.mjs — attach, or spawn a fully isolated instance.
scenarios/ — one module per measurement.
run.mjs — entrypoint. serve.mjs — standalone isolated launcher.

Not migrated (kept as dev utilities)

eval.mjs, reload.mjs, reload-renderer.mjs, probe-renderer.mjs, probe-thread.mjs, click-session.mjs, diag-*.mjs are interactive dev helpers, not benchmarks. They can adopt lib/cdp.mjs in a follow-up.

4.4 KiB Raw Blame History