mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-31 19:16:29 +00:00
feat(agent): token telemetry for micro-compaction
The existing log line reports message counts, which is the least informative number available here: absorbing one tool-heavy exchange can drop hundreds of tokens while moving the count by one. There was no way to answer "is this actually helping?" from a real session. Emit one content-free JSON line per pass, in the same shape as the batch compaction telemetry: before/after tokens, the delta, the size of the absorbed exchange, the rolling summary size, duration, and running per-session totals so a whole run can be read off the last line. No transcript content rides along. Add scripts/micro_compaction_report.py to aggregate those lines into passes, outcome mix, net tokens saved, mean exchange size and durations, with an optional per-session breakdown. Measuring it immediately surfaced something worth documenting: the first pass in a session normally *costs* tokens. The summary marker carries a fixed ~400 tokens of scaffolding, paid on pass one against a single absorbed exchange. From pass two the marker is replaced rather than added, so the overhead is already paid and each exchange is close to pure saving. Break-even is typically the second or third pass. Tests cover the telemetry contract, the cumulative totals, and that first-pass/later-pass shape so nobody reads a single turn and concludes it made things worse. The estimator costs ~5 ms at 600 messages and ~20 ms at 1200, taken twice per pass, post-turn — and only once an exchange is actually in hand, so turns that no-op early pay nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
a72f898d0f
commit
cac9526d2a
4 changed files with 339 additions and 2 deletions
|
|
@ -160,7 +160,47 @@ compression:
|
|||
Set it to `false` to disable micro-compaction and return to batch-only
|
||||
compaction. Everything else about compression is unchanged.
|
||||
|
||||
## What you'll see in the logs
|
||||
## Measuring it
|
||||
|
||||
Every pass emits one content-free JSON line, in the same style as the batch
|
||||
compaction telemetry:
|
||||
|
||||
```
|
||||
micro compaction telemetry: {"event":"micro_compaction","outcome":"absorbed",
|
||||
"tokens_before":12739,"tokens_after":12060,"tokens_delta":-679,
|
||||
"exchange_tokens":868,"rolling_summary_tokens":31,"passes_total":1,
|
||||
"tokens_saved_total":679,"duration_ms":14,...}
|
||||
```
|
||||
|
||||
`tokens_delta` is negative when the pass shrank the transcript.
|
||||
`tokens_saved_total` and `passes_total` accumulate across the session, so a whole
|
||||
run can be summarised from its last line. No transcript content appears in the
|
||||
payload — only counts.
|
||||
|
||||
To turn a log into an answer:
|
||||
|
||||
```
|
||||
python scripts/micro_compaction_report.py [--per-session] [LOGFILE ...]
|
||||
```
|
||||
|
||||
Defaults to `$HERMES_HOME/logs/agent.log`. It reports passes, outcome mix, net
|
||||
tokens saved, mean absorbed-exchange size and pass durations.
|
||||
|
||||
### Reading the numbers honestly
|
||||
|
||||
**The first pass in a session usually costs tokens rather than saving them.**
|
||||
Inserting the summary marker carries a fixed ~400 tokens of scaffolding — the
|
||||
compaction preamble, the historical heading, the end marker — and on pass one
|
||||
that is paid against a single absorbed exchange. A first pass showing
|
||||
`tokens_delta: +330` is not a malfunction.
|
||||
|
||||
From the second pass on, the marker is *replaced* rather than added, so the
|
||||
scaffolding is already paid for and each absorbed exchange is close to pure
|
||||
saving. The break-even is normally the second or third pass. This is why the
|
||||
per-session view matters more than any single line: judge the feature on a
|
||||
session's trajectory, not on one turn.
|
||||
|
||||
The plainer human-readable lines are still there too:
|
||||
|
||||
```
|
||||
Micro-compaction: 37 -> 36 messages
|
||||
|
|
@ -168,7 +208,7 @@ Micro-compaction defrag: rolling summary re-summarized (1843 chars)
|
|||
Micro-compaction: skipping exchange at cursor 12 after 3 consecutive failures
|
||||
```
|
||||
|
||||
Message counts move by small amounts — that's the point. The token count is where
|
||||
Message counts move by small amounts — that's expected. The token count is where
|
||||
the effect shows: absorbing one tool-heavy exchange can drop hundreds of tokens
|
||||
while changing the message count by one or two.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue