mirrors/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-05-03 02:11:48 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5
hermes-agent/plugins/hermes-achievements/docs/achievements-performance-implementation-plan.md
Teknium 62a5d7207d
feat(plugins): bundle hermes-achievements + scan full session history (#17754) 
* feat(plugins): bundle hermes-achievements, scan full session history

Ships @PCinkusz's hermes-achievements dashboard plugin (https://github.com/PCinkusz/hermes-achievements) as a bundled plugin at plugins/hermes-achievements/ and fixes a bug in the scan path that made the plugin only see the first 200 sessions — making lifetime badges (50k tool calls, 75k errors, etc.) unreachable on long-running installs.

Changes:

- plugins/hermes-achievements/: vendor v0.3.1 verbatim (manifest, dist/, plugin_api.py, tests, docs, README).
- plugins/hermes-achievements/dashboard/plugin_api.py:
  * scan_sessions(): limit=None now scans ALL sessions via SQLite LIMIT -1. Previously capped at 200, so users with 8000+ sessions saw ~2% of their history.
  * evaluate_all(): first-ever scans run in a background thread so the dashboard request path never blocks. Stale snapshots serve immediately while a background refresh runs. force=True still blocks synchronously for manual /rescan.
  * _build_pending_snapshot(), _start_background_scan(), _run_scan_and_update_cache(): supporting plumbing + idempotent thread spawn.
- tests/plugins/test_achievements_plugin.py: new tests covering the 200-cap regression, the background-scan first-run flow, stale-serve-plus-background-refresh, forced sync rescan, and scan-thread idempotency.
- website/docs/user-guide/features/built-in-plugins.md: lists hermes-achievements in the bundled-plugins table and documents API endpoints, state files, and performance characteristics.

E2E validated against a real 8564-session ~6.4GB state.db:
  * Cold scan: 13m 19s (one-time, backgrounded — UI never blocks)
  * Warm rescan: 1.47s (8563/8564 sessions reused from checkpoint cache)
  * 57/60 achievements unlocked, 3 discovered — aggregates like total_tool_calls=259958, total_errors=164213, skill_events=368243 correctly surface lifetime badges that the 200-cap made unreachable.

Original credit: @PCinkusz (MIT-licensed). Upstream repo remains the staging ground for new badges; this bundle keeps the dashboard feature parity with Hermes core changes.

* feat(achievements): publish partial snapshots during cold scan

Previously a cold scan on a large session DB (13min on 8564 sessions)
showed zero badges for the entire duration, then every badge at once
when the scan completed. A dashboard refresh mid-scan was indistinguishable
from a fresh install with no history.

Now the scanner publishes a partial snapshot to _SNAPSHOT_CACHE every
250 sessions, so each refresh during a cold scan surfaces more badges
incrementally.

Mechanism:
- scan_sessions() takes an optional progress_callback fired every
  progress_every sessions with (sessions_so_far, scanned, total).
- _compute_from_scan() is extracted from compute_all() and gains an
  is_partial flag that skips writing to state.json — we don't want
  to record unlocked_at based on a half-complete aggregate that a
  later session might rebalance.
- _run_scan_and_update_cache() installs a publisher callback that
  builds a partial snapshot, marks it mode='in_progress', and writes
  it to the cache with age=0 so the UI keeps polling /scan-status
  and picks up the final snapshot when the scan completes.
- Manual /rescan (force=True) disables partial publishing — the
  caller is blocking on the final result anyway.

E2E against real 8564-session state.db (polled cache every 10s):
  t=10s: cache empty
  t=20s: 250/8564 scanned, 35 unlocked, 25 discovered
  t=40s: 500/8564 scanned, 42 unlocked, 18 discovered
  t=60s: 1000/8564 scanned, 49 unlocked, 11 discovered
  ...

Tests: 9/9 pass (2 new — partial snapshot publication + no-persist-on-partial).
Upstream unittest suite: 10/10 pass.

* feat(achievements): in-progress scan banner with live % progress

Previously the dashboard showed zero badges silently during long cold
scans (13min on 8564 sessions). The backend was publishing partial
snapshots every 250 sessions, but the bundled UI didn't surface any
indicator that a scan was running — it just rendered the main page
with whatever counts were currently published and no way for the user
to know more progress was coming.

UI changes (dist/index.js, dist/style.css):

- Added a scan-in-progress banner rendered between the hero and stats
  when scan_meta.mode is 'pending' or 'in_progress'. Shows:
    BUILDING ACHIEVEMENT PROFILE…
    Scanned 1,750 of 8,564 sessions · 20%. Badges unlock as more history streams in.
  with a pulsing teal indicator and a filling teal/cyan progress bar.
  Disappears the moment the backend flips to 'full' or 'incremental'.

- Added an auto-poller via useEffect — while scanInFlight is true the
  page re-fetches /achievements every 4s WITHOUT toggling the loading
  skeleton, so unlock counts tick up visibly without the user refreshing.
  The effect cleans itself up when the scan finishes.

- Added refresh() (re-fetch, no loading flip) alongside the existing
  load() (full reload, used by the Rescan button).

Attribution preserved:

- Added a header comment to index.js crediting @PCinkusz
  (https://github.com/PCinkusz/hermes-achievements, MIT) as the
  original author, noting the banner is a layered addition on top
  of the original dist bundle.
- Matching header comment in style.css, flagging the new
  .ha-scan-banner* rules as the local addition.

Live-verified end to end:

- Spun up `hermes dashboard --port 9229 --no-open` against a fresh
  HERMES_HOME symlinked to the real 8564-session state.db.
- Opened /achievements in a browser, confirmed the banner renders with
  live progress: 'Scanned 1,000 of 8,564 sessions · 11%' → updates to
  '1,250 ... · 14%' → '1,750 ... · 20%' without user interaction,
  matching the backend's partial publications.
- Stats row simultaneously climbed from 35 → 49 → 53 unlocked as
  more history streamed in.
- Vision analysis of the rendered page confirms the banner styling
  matches the rest of the dashboard (dark card bg, teal accent, same
  small-caps typography, pulsing indicator reusing ha-pulse keyframes).
2026-04-29 23:23:57 -07:00

4.6 KiB
Raw Blame History

Hermes Achievements Performance Implementation Plan

Status: Ready for execution after hackathon review window Constraint: Plugin remains frozen until judging is complete Decision: /overview and top-banner slots are out of scope and will be removed.

Phase 0 — Baseline & Safety (no behavior change)

Task 0.1: Add perf benchmark script (local)

Objective: Repro baseline before/after.

Acceptance:

  • Can print endpoint timings for /achievements (3 runs each, cold + warm).

Task 0.2: Define acceptance thresholds

Objective: Lock success criteria now.

Acceptance:

  • Documented SLOs:
    • /achievements p95 < 1s (cached)
    • max active scan jobs = 1

Phase 1 — Remove unused overview/slot surface (highest certainty)

Task 1.1: Remove /overview backend route

Objective: Eliminate duplicate heavy endpoint path.

Acceptance:

  • plugin_api.py no longer exposes /overview.

Task 1.2: Remove slot registration and SummarySlot frontend code

Objective: Remove cross-tab banner fetch behavior.

Acceptance:

  • No registerSlot(..."sessions:top"...) or registerSlot(..."analytics:top"...).
  • No frontend call to api("/overview").

Task 1.3: Update plugin manifest

Objective: Reflect final UI scope.

Acceptance:

  • manifest.json removes slots declarations.
  • Tab registration remains intact.

Phase 2 — Shared snapshot persistence + single-flight for /achievements

Task 2.1: Introduce snapshot store abstraction + on-disk persistence

Objective: Single source of truth for Achievements data that survives process restarts.

Acceptance:

  • One structure contains dataset consumed by /achievements.
  • Repeated requests do not recompute when cache is fresh.
  • Snapshot persisted at ~/.hermes/plugins/hermes-achievements/scan_snapshot.json.

Task 2.2: Single-flight scan coordinator

Objective: Prevent concurrent recomputes.

Acceptance:

  • Simultaneous requests result in one compute run.

Task 2.3: Refactor /achievements to read snapshot

Objective: Remove direct repeated compute from request path.

Acceptance:

  • /achievements does not run independent full recompute per request when cache is valid.

Phase 3 — Stale-While-Revalidate

Task 3.1: TTL state (FRESH/STALE)

Objective: Serve immediately when stale, refresh in background.

Acceptance:

  • Cached response returned quickly even when expired.
  • Refresh is asynchronous.

Task 3.2: Add scan-status endpoint (optional)

Objective: Let UI/ops inspect scan state.

Acceptance:

  • Returns state, last success time, last duration, last error.

Task 3.3: Add metadata fields to /achievements

Objective: Improve transparency.

Acceptance:

  • Response includes generated_at, is_stale, maybe scan_id.

Phase 4 — Incremental Scanning (optional but recommended)

Task 4.1: Add per-session checkpoint file

Objective: Track session-level changes, not just global scan time.

Acceptance:

  • Checkpoint persisted at ~/.hermes/plugins/hermes-achievements/scan_checkpoint.json.
  • For each session: session_id, fingerprint (updated_at/message_count/hash), and cached contribution.

Task 4.2: Incremental aggregation

Objective: Recompute only changed/new sessions and reuse unchanged contributions.

Acceptance:

  • Typical refresh time drops materially below full scan.
  • Aggregate rebuild uses: subtract old contribution + add new contribution for changed sessions.

Task 4.3: Full rebuild fallback

Objective: Preserve correctness.

Acceptance:

  • Manual full rescan always possible.
  • Schema/version changes invalidate checkpoint and force full rebuild.

Test Plan

  1. Unit tests
  • Snapshot lifecycle transitions
  • Dedupe logic under parallel requests
  • /achievements response compatibility
  1. Integration tests
  • Opening Achievements repeatedly causes <=1 heavy scan while in-flight
  • /achievements warm-cache load is fast
  • manual rescan updates snapshot and timestamps
  1. Manual benchmarks
  • Compare pre/post /achievements timings with same history dataset

Rollout Plan

  1. Release internal branch with Phase 1 (remove overview/slots).
  2. Validate no UI regression in Achievements tab.
  3. Add Phase 2 snapshot/dedupe.
  4. Add Phase 3 stale-while-revalidate + status metadata.
  5. Optional: incremental scanner.

Rollback: keep old compute path behind temporary feature flag for one release window.

Definition of Done

  • Achievements tab remains fully functional (counts, latest, tiers, cards, filters).
  • No /overview endpoint or slot calls remain.
  • Repeated Achievements loads feel immediate after warm cache.
  • Metrics/unlocks remain unchanged versus baseline.