mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-21 16:18:55 +00:00
b6ae910d8c
* bench(desktop): measure the full picture — prod build, cold-start, first-token
Stop drip-feeding scenarios: extend the harness to cover the latencies that
actually dominate perceived speed, and measure them on a REAL production build.
- --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1,
off in normal builds) and launch it from dist/. Measures minified React, so
numbers are representative shipped figures instead of ~3x-inflated dev ones.
- cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a
fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms.
- first-token scenario (backend tier): Enter → first assistant token painted —
the TTFT latency an agent app is uniquely judged on.
- run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates
ci+cold tiers against the baseline.
Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all
green. Representative numbers:
cold-start spawn→interactive ~1.6s, FCP ~0.5s
stream frame p95 22ms, 1 longtask
keystroke p50 2ms, p95 8.7ms
transcript mount 145ms, 82ms longtask (400-msg open)
The prod build also settled the open question from the dev numbers: the
transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not
actionable. Measurement did its job.
* bench(desktop): trustworthy cold-start measurement (code-splitting is NOT the lever)
Investigated code-splitting the ~22MB renderer bundle to cut cold start. It is
the wrong fix on both counts:
1. Intentional design: vite.config disables codeSplitting because Shiki emits
thousands of dynamic chunks and electron-builder OOMs scanning them — a
packaging/installer constraint, not an oversight.
2. The data says it wouldn't help. Fixing the cold-start measurement to be
trustworthy and reading the boot composition (prod build):
spawn → interactive ~1.5s
renderer nav → DOMInteractive ~0.8s, → DOMContentLoaded ~1.06s
so the whole 22MB bundle EVAL is only ~0.27s (DCL − DOMInteractive) of the
~1.5s. The dominant costs are Electron/window startup and React app mount —
neither touched by splitting.
The measurement fixes (the real content of this PR — no app change, since the
optimization was rejected):
- Drop HERMES_DESKTOP_BOOT_FAKE from spawned instances — it injected artificial
per-phase boot-overlay sleeps that inflated cold-start (and slowed every run).
- Unique debug/dev port per cold-start run — a just-killed instance can hold
:9222 briefly, so reusing it made CDP attach to the DYING instance and report
garbage (spawn_to_cdp of ~4ms). Stepping the port per run fixes the race.
- Richer boot marks (dom_interactive, dom_content_loaded, main-script size) so
cold-start composition is visible, not just a single number.
- Forward all numeric boot marks from the cold-start loop.
- Re-baseline cold-start with the clean numbers.
A real cold-start win would target Electron startup / app-mount (e.g. V8 code
cache, deferred non-critical mount) — a future pass, now that it's measurable.
60 lines
1.6 KiB
JSON
60 lines
1.6 KiB
JSON
{
|
|
"_meta": {
|
|
"note": "Median of 5 runs, darwin-arm64, `--spawn --prod` (PRODUCTION minified renderer, real boot — no fake-boot). Representative shipped numbers, not dev-inflated. cold-start uses a fresh unique-port spawn per run and its marks are process-spawn wall clock (spawn_to_*) or renderer nav-relative (dom_*). Re-baseline per device with `--update-baseline`; tolerances are loose for cross-machine/disk variance.",
|
|
"platform": "darwin-arm64",
|
|
"node": "v24.11.0",
|
|
"updated": "2026-07-19T22:59:21.605Z"
|
|
},
|
|
"scenarios": {
|
|
"stream": {
|
|
"tolerance": {
|
|
"tolFrac": 0.6,
|
|
"tolAbs": 5
|
|
},
|
|
"metrics": {
|
|
"longtasks_n": 1,
|
|
"longtask_max_ms": 67,
|
|
"frame_p95_ms": 22,
|
|
"frame_p99_ms": 23.7,
|
|
"slow_frames_33": 1,
|
|
"intermut_p95_ms": 36.1
|
|
}
|
|
},
|
|
"keystroke": {
|
|
"tolerance": {
|
|
"tolFrac": 0.6,
|
|
"tolAbs": 4
|
|
},
|
|
"metrics": {
|
|
"keystroke_p50_ms": 2.1,
|
|
"keystroke_p95_ms": 8.7,
|
|
"keystroke_p99_ms": 16.9,
|
|
"keystroke_slow_16": 2
|
|
}
|
|
},
|
|
"transcript": {
|
|
"tolerance": {
|
|
"tolFrac": 0.75,
|
|
"tolAbs": 40
|
|
},
|
|
"metrics": {
|
|
"transcript_mount_ms": 145,
|
|
"transcript_longtask_ms": 82,
|
|
"transcript_longtask_max_ms": 82
|
|
}
|
|
},
|
|
"cold-start": {
|
|
"tolerance": {
|
|
"tolFrac": 0.6,
|
|
"tolAbs": 150
|
|
},
|
|
"metrics": {
|
|
"spawn_to_cdp_ms": 1098,
|
|
"spawn_to_driver_ms": 1482,
|
|
"dom_interactive_ms": 794,
|
|
"dom_content_loaded_ms": 1057,
|
|
"nav_to_read_ms": 1209
|
|
}
|
|
}
|
|
}
|
|
}
|