mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-03 02:11:48 +00:00
62a5d7207d
* feat(plugins): bundle hermes-achievements, scan full session history Ships @PCinkusz's hermes-achievements dashboard plugin (https://github.com/PCinkusz/hermes-achievements) as a bundled plugin at plugins/hermes-achievements/ and fixes a bug in the scan path that made the plugin only see the first 200 sessions — making lifetime badges (50k tool calls, 75k errors, etc.) unreachable on long-running installs. Changes: - plugins/hermes-achievements/: vendor v0.3.1 verbatim (manifest, dist/, plugin_api.py, tests, docs, README). - plugins/hermes-achievements/dashboard/plugin_api.py: * scan_sessions(): limit=None now scans ALL sessions via SQLite LIMIT -1. Previously capped at 200, so users with 8000+ sessions saw ~2% of their history. * evaluate_all(): first-ever scans run in a background thread so the dashboard request path never blocks. Stale snapshots serve immediately while a background refresh runs. force=True still blocks synchronously for manual /rescan. * _build_pending_snapshot(), _start_background_scan(), _run_scan_and_update_cache(): supporting plumbing + idempotent thread spawn. - tests/plugins/test_achievements_plugin.py: new tests covering the 200-cap regression, the background-scan first-run flow, stale-serve-plus-background-refresh, forced sync rescan, and scan-thread idempotency. - website/docs/user-guide/features/built-in-plugins.md: lists hermes-achievements in the bundled-plugins table and documents API endpoints, state files, and performance characteristics. E2E validated against a real 8564-session ~6.4GB state.db: * Cold scan: 13m 19s (one-time, backgrounded — UI never blocks) * Warm rescan: 1.47s (8563/8564 sessions reused from checkpoint cache) * 57/60 achievements unlocked, 3 discovered — aggregates like total_tool_calls=259958, total_errors=164213, skill_events=368243 correctly surface lifetime badges that the 200-cap made unreachable. Original credit: @PCinkusz (MIT-licensed). Upstream repo remains the staging ground for new badges; this bundle keeps the dashboard feature parity with Hermes core changes. * feat(achievements): publish partial snapshots during cold scan Previously a cold scan on a large session DB (13min on 8564 sessions) showed zero badges for the entire duration, then every badge at once when the scan completed. A dashboard refresh mid-scan was indistinguishable from a fresh install with no history. Now the scanner publishes a partial snapshot to _SNAPSHOT_CACHE every 250 sessions, so each refresh during a cold scan surfaces more badges incrementally. Mechanism: - scan_sessions() takes an optional progress_callback fired every progress_every sessions with (sessions_so_far, scanned, total). - _compute_from_scan() is extracted from compute_all() and gains an is_partial flag that skips writing to state.json — we don't want to record unlocked_at based on a half-complete aggregate that a later session might rebalance. - _run_scan_and_update_cache() installs a publisher callback that builds a partial snapshot, marks it mode='in_progress', and writes it to the cache with age=0 so the UI keeps polling /scan-status and picks up the final snapshot when the scan completes. - Manual /rescan (force=True) disables partial publishing — the caller is blocking on the final result anyway. E2E against real 8564-session state.db (polled cache every 10s): t=10s: cache empty t=20s: 250/8564 scanned, 35 unlocked, 25 discovered t=40s: 500/8564 scanned, 42 unlocked, 18 discovered t=60s: 1000/8564 scanned, 49 unlocked, 11 discovered ... Tests: 9/9 pass (2 new — partial snapshot publication + no-persist-on-partial). Upstream unittest suite: 10/10 pass. * feat(achievements): in-progress scan banner with live % progress Previously the dashboard showed zero badges silently during long cold scans (13min on 8564 sessions). The backend was publishing partial snapshots every 250 sessions, but the bundled UI didn't surface any indicator that a scan was running — it just rendered the main page with whatever counts were currently published and no way for the user to know more progress was coming. UI changes (dist/index.js, dist/style.css): - Added a scan-in-progress banner rendered between the hero and stats when scan_meta.mode is 'pending' or 'in_progress'. Shows: BUILDING ACHIEVEMENT PROFILE… Scanned 1,750 of 8,564 sessions · 20%. Badges unlock as more history streams in. with a pulsing teal indicator and a filling teal/cyan progress bar. Disappears the moment the backend flips to 'full' or 'incremental'. - Added an auto-poller via useEffect — while scanInFlight is true the page re-fetches /achievements every 4s WITHOUT toggling the loading skeleton, so unlock counts tick up visibly without the user refreshing. The effect cleans itself up when the scan finishes. - Added refresh() (re-fetch, no loading flip) alongside the existing load() (full reload, used by the Rescan button). Attribution preserved: - Added a header comment to index.js crediting @PCinkusz (https://github.com/PCinkusz/hermes-achievements, MIT) as the original author, noting the banner is a layered addition on top of the original dist bundle. - Matching header comment in style.css, flagging the new .ha-scan-banner* rules as the local addition. Live-verified end to end: - Spun up `hermes dashboard --port 9229 --no-open` against a fresh HERMES_HOME symlinked to the real 8564-session state.db. - Opened /achievements in a browser, confirmed the banner renders with live progress: 'Scanned 1,000 of 8,564 sessions · 11%' → updates to '1,250 ... · 14%' → '1,750 ... · 20%' without user interaction, matching the backend's partial publications. - Stats row simultaneously climbed from 35 → 49 → 53 unlocked as more history streamed in. - Vision analysis of the rendered page confirms the banner styling matches the rest of the dashboard (dark card bg, teal accent, same small-caps typography, pulsing indicator reusing ha-pulse keyframes).
157 lines
4.6 KiB
Markdown
157 lines
4.6 KiB
Markdown
# Hermes Achievements Performance Implementation Plan
|
|
|
|
Status: Ready for execution after hackathon review window
|
|
Constraint: Plugin remains frozen until judging is complete
|
|
Decision: `/overview` and top-banner slots are out of scope and will be removed.
|
|
|
|
---
|
|
|
|
## Phase 0 — Baseline & Safety (no behavior change)
|
|
|
|
### Task 0.1: Add perf benchmark script (local)
|
|
Objective: Repro baseline before/after.
|
|
|
|
Acceptance:
|
|
- Can print endpoint timings for `/achievements` (3 runs each, cold + warm).
|
|
|
|
### Task 0.2: Define acceptance thresholds
|
|
Objective: Lock success criteria now.
|
|
|
|
Acceptance:
|
|
- Documented SLOs:
|
|
- `/achievements` p95 < 1s (cached)
|
|
- max active scan jobs = 1
|
|
|
|
---
|
|
|
|
## Phase 1 — Remove unused overview/slot surface (highest certainty)
|
|
|
|
### Task 1.1: Remove `/overview` backend route
|
|
Objective: Eliminate duplicate heavy endpoint path.
|
|
|
|
Acceptance:
|
|
- `plugin_api.py` no longer exposes `/overview`.
|
|
|
|
### Task 1.2: Remove slot registration and SummarySlot frontend code
|
|
Objective: Remove cross-tab banner fetch behavior.
|
|
|
|
Acceptance:
|
|
- No `registerSlot(..."sessions:top"...)` or `registerSlot(..."analytics:top"...)`.
|
|
- No frontend call to `api("/overview")`.
|
|
|
|
### Task 1.3: Update plugin manifest
|
|
Objective: Reflect final UI scope.
|
|
|
|
Acceptance:
|
|
- `manifest.json` removes `slots` declarations.
|
|
- Tab registration remains intact.
|
|
|
|
---
|
|
|
|
## Phase 2 — Shared snapshot persistence + single-flight for `/achievements`
|
|
|
|
### Task 2.1: Introduce snapshot store abstraction + on-disk persistence
|
|
Objective: Single source of truth for Achievements data that survives process restarts.
|
|
|
|
Acceptance:
|
|
- One structure contains dataset consumed by `/achievements`.
|
|
- Repeated requests do not recompute when cache is fresh.
|
|
- Snapshot persisted at `~/.hermes/plugins/hermes-achievements/scan_snapshot.json`.
|
|
|
|
### Task 2.2: Single-flight scan coordinator
|
|
Objective: Prevent concurrent recomputes.
|
|
|
|
Acceptance:
|
|
- Simultaneous requests result in one compute run.
|
|
|
|
### Task 2.3: Refactor `/achievements` to read snapshot
|
|
Objective: Remove direct repeated compute from request path.
|
|
|
|
Acceptance:
|
|
- `/achievements` does not run independent full recompute per request when cache is valid.
|
|
|
|
---
|
|
|
|
## Phase 3 — Stale-While-Revalidate
|
|
|
|
### Task 3.1: TTL state (`FRESH`/`STALE`)
|
|
Objective: Serve immediately when stale, refresh in background.
|
|
|
|
Acceptance:
|
|
- Cached response returned quickly even when expired.
|
|
- Refresh is asynchronous.
|
|
|
|
### Task 3.2: Add `scan-status` endpoint (optional)
|
|
Objective: Let UI/ops inspect scan state.
|
|
|
|
Acceptance:
|
|
- Returns state, last success time, last duration, last error.
|
|
|
|
### Task 3.3: Add metadata fields to `/achievements`
|
|
Objective: Improve transparency.
|
|
|
|
Acceptance:
|
|
- Response includes `generated_at`, `is_stale`, maybe `scan_id`.
|
|
|
|
---
|
|
|
|
## Phase 4 — Incremental Scanning (optional but recommended)
|
|
|
|
### Task 4.1: Add per-session checkpoint file
|
|
Objective: Track session-level changes, not just global scan time.
|
|
|
|
Acceptance:
|
|
- Checkpoint persisted at `~/.hermes/plugins/hermes-achievements/scan_checkpoint.json`.
|
|
- For each session: `session_id`, fingerprint (`updated_at`/message_count/hash), and cached contribution.
|
|
|
|
### Task 4.2: Incremental aggregation
|
|
Objective: Recompute only changed/new sessions and reuse unchanged contributions.
|
|
|
|
Acceptance:
|
|
- Typical refresh time drops materially below full scan.
|
|
- Aggregate rebuild uses: subtract old contribution + add new contribution for changed sessions.
|
|
|
|
### Task 4.3: Full rebuild fallback
|
|
Objective: Preserve correctness.
|
|
|
|
Acceptance:
|
|
- Manual full rescan always possible.
|
|
- Schema/version changes invalidate checkpoint and force full rebuild.
|
|
|
|
---
|
|
|
|
## Test Plan
|
|
|
|
1. Unit tests
|
|
- Snapshot lifecycle transitions
|
|
- Dedupe logic under parallel requests
|
|
- `/achievements` response compatibility
|
|
|
|
2. Integration tests
|
|
- Opening Achievements repeatedly causes <=1 heavy scan while in-flight
|
|
- `/achievements` warm-cache load is fast
|
|
- manual rescan updates snapshot and timestamps
|
|
|
|
3. Manual benchmarks
|
|
- Compare pre/post `/achievements` timings with same history dataset
|
|
|
|
---
|
|
|
|
## Rollout Plan
|
|
|
|
1. Release internal branch with Phase 1 (remove overview/slots).
|
|
2. Validate no UI regression in Achievements tab.
|
|
3. Add Phase 2 snapshot/dedupe.
|
|
4. Add Phase 3 stale-while-revalidate + status metadata.
|
|
5. Optional: incremental scanner.
|
|
|
|
Rollback: keep old compute path behind temporary feature flag for one release window.
|
|
|
|
---
|
|
|
|
## Definition of Done
|
|
|
|
- Achievements tab remains fully functional (counts, latest, tiers, cards, filters).
|
|
- No `/overview` endpoint or slot calls remain.
|
|
- Repeated Achievements loads feel immediate after warm cache.
|
|
- Metrics/unlocks remain unchanged versus baseline.
|