mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-16 14:32:34 +00:00
feat(agent): track per-model token usage for mid-session model switches
The `sessions` table records only the initial (model, billing_provider) for a session, so when a user switches models mid-session (via `/model` or programmatically) every token — including the switched model's — is attributed to the first model. Insights/billing reports then hide the cost of the new model entirely (e.g. a session that started on deepseek and switched to opus shows $0 for opus). Add a `session_model_usage` table keyed (session_id, model, billing_provider) that accumulates each per-API-call delta under the model active at the time of the call. `update_token_counts()` is the single chokepoint every per-call delta flows through (CLI, gateway, cron, delegated, codex), so recording there captures accurate attribution on every platform. Only the incremental path records — the gateway's `absolute=True` summary overwrite is skipped to avoid double-counting cumulative totals that can't be split per model. When a call omits the model, it falls back to the session's recorded model, matching the existing COALESCE-from-session summary behaviour. Insights `_compute_model_breakdown` now aggregates tokens and cost from `session_model_usage`, so a switched session splits correctly across models, with a defensive fallback to the per-session aggregate for any session lacking usage rows. A v17 migration backfills one usage row per existing token-bearing session from its aggregate totals (idempotent via INSERT OR IGNORE), validated lossless against a 1.3 GB production DB. Tests: per-model recording, mid-session split, model fallback, absolute no-double-count, v17 backfill, and an insights-level switch breakdown. Fixes #51607. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
022c4991fc
commit
cb7f6bbb2e
4 changed files with 438 additions and 20 deletions
|
|
@ -320,6 +320,35 @@ class TestInsightsPopulated:
|
|||
claude = next(m for m in models if "claude-sonnet" in m["model"])
|
||||
assert claude["sessions"] == 2
|
||||
|
||||
def test_model_breakdown_splits_mid_session_switch(self, db):
|
||||
"""A session that switches models mid-flight is split across both
|
||||
models in the breakdown, not dumped on the initial model (#51607).
|
||||
"""
|
||||
now = time.time()
|
||||
db.create_session(session_id="sw", source="cli",
|
||||
model="deepseek/deepseek-v4-pro")
|
||||
# 40k tokens on deepseek, then switch and 50k on opus.
|
||||
db.update_token_counts("sw", input_tokens=40000, output_tokens=8000,
|
||||
model="deepseek/deepseek-v4-pro",
|
||||
billing_provider="deepseek", api_call_count=2)
|
||||
db.update_session_model("sw", "anthropic/claude-opus-4.8")
|
||||
db.update_token_counts("sw", input_tokens=50000, output_tokens=4000,
|
||||
model="anthropic/claude-opus-4.8",
|
||||
billing_provider="openrouter", api_call_count=3)
|
||||
db._conn.commit()
|
||||
|
||||
report = InsightsEngine(db).generate(days=30)
|
||||
models = {m["model"]: m for m in report["models"]}
|
||||
assert "deepseek-v4-pro" in models
|
||||
assert "claude-opus-4.8" in models
|
||||
# Tokens attributed to the model that actually incurred them.
|
||||
assert models["deepseek-v4-pro"]["input_tokens"] == 40000
|
||||
assert models["claude-opus-4.8"]["input_tokens"] == 50000
|
||||
assert models["claude-opus-4.8"]["api_calls"] == 3
|
||||
# The summary row's single model would have hidden one of these.
|
||||
assert models["deepseek-v4-pro"]["total_tokens"] == 48000
|
||||
assert models["claude-opus-4.8"]["total_tokens"] == 54000
|
||||
|
||||
def test_platform_breakdown(self, populated_db):
|
||||
engine = InsightsEngine(populated_db)
|
||||
report = engine.generate(days=30)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue