mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-25 17:18:11 +00:00
b1fb3c5285
Stop drip-feeding scenarios: extend the harness to cover the latencies that actually dominate perceived speed, and measure them on a REAL production build. - --prod: build a production renderer with the probe included (VITE_PERF_PROBE=1, off in normal builds) and launch it from dist/. Measures minified React, so numbers are representative shipped figures instead of ~3x-inflated dev ones. - cold-start scenario (tier "cold"): launch → CDP → driver → first paint, via a fresh isolated spawn per run. Captures spawn_to_cdp_ms, spawn_to_driver_ms, fcp_ms. - first-token scenario (backend tier): Enter → first assistant token painted — the TTFT latency an agent app is uniquely judged on. - run.mjs gained --prod (build once), cold-start fresh-spawn loop, and gates ci+cold tiers against the baseline. Baseline re-captured on a PRODUCTION build (median of 5), darwin-arm64 — all green. Representative numbers: cold-start spawn→interactive ~1.6s, FCP ~0.5s stream frame p95 22ms, 1 longtask keystroke p50 2ms, p95 8.7ms transcript mount 145ms, 82ms longtask (400-msg open) The prod build also settled the open question from the dev numbers: the transcript-mount "lead" (221ms longtask in dev) is only ~72-82ms in prod — not actionable. Measurement did its job.
27 lines
968 B
JavaScript
27 lines
968 B
JavaScript
// Scenario registry. Add a scenario module here and it's automatically
|
|
// available to the runner, the default suite (tier 'ci'), and the baseline gate.
|
|
|
|
import coldStart from './cold-start.mjs'
|
|
import firstToken from './first-token.mjs'
|
|
import keystroke from './keystroke.mjs'
|
|
import profileSwitch from './profile-switch.mjs'
|
|
import sessionSwitch from './session-switch.mjs'
|
|
import stream from './stream.mjs'
|
|
import submit from './submit.mjs'
|
|
import transcript from './transcript.mjs'
|
|
|
|
export const SCENARIOS = {
|
|
[stream.name]: stream,
|
|
[keystroke.name]: keystroke,
|
|
[transcript.name]: transcript,
|
|
[coldStart.name]: coldStart,
|
|
[firstToken.name]: firstToken,
|
|
[submit.name]: submit,
|
|
[sessionSwitch.name]: sessionSwitch,
|
|
[profileSwitch.name]: profileSwitch
|
|
}
|
|
|
|
/** Scenarios safe to run with no LLM credits / no live backend — the default suite. */
|
|
export const CI_SCENARIOS = Object.values(SCENARIOS)
|
|
.filter(s => s.tier === 'ci')
|
|
.map(s => s.name)
|