mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-24 16:54:43 +00:00
b1fb3c5285
Stop drip-feeding scenarios: extend the harness to cover the latencies that actually dominate perceived speed, and measure them on a REAL production build. - --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1, off in normal builds) and launch it from dist/. Measures minified React, so numbers are representative shipped figures instead of ~3x-inflated dev ones. - cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms. - first-token scenario (backend tier): Enter → first assistant token painted — the TTFT latency an agent app is uniquely judged on. - run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates ci+cold tiers against the baseline. Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all green. Representative numbers: cold-start spawn→interactive ~1.6s, FCP ~0.5s stream frame p95 22ms, 1 longtask keystroke p50 2ms, p95 8.7ms transcript mount 145ms, 82ms longtask (400-msg open) The prod build also settled the open question from the dev numbers: the transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not actionable. Measurement did its job.
18 lines
919 B
JavaScript
18 lines
919 B
JavaScript
// Cold start — launch → renderer → interactive. Unlike the other scenarios this
|
|
// measures the LAUNCH itself, so it can't run against an already-up instance:
|
|
// the runner spawns a fresh isolated instance per run (requires --spawn) and
|
|
// reads the timings/boot-marks the launcher captures. Registered here so it's a
|
|
// known name with a baseline entry; the actual measurement lives in run.mjs.
|
|
//
|
|
// Metrics (lower is better):
|
|
// spawn_to_cdp_ms process spawn → CDP page target reachable (electron/V8 up)
|
|
// spawn_to_driver_ms process spawn → renderer mounted + perf driver present
|
|
// fcp_ms renderer nav start → first contentful paint
|
|
export default {
|
|
name: 'cold-start',
|
|
tier: 'cold',
|
|
description: 'Launch → first paint → interactive (fresh spawn per run).',
|
|
run() {
|
|
throw new Error('cold-start is measured by the runner via fresh spawns; use `--spawn`.')
|
|
}
|
|
}
|