mirrors/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-07-31 19:16:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

feat(cache): persist the exact bytes sent to the API in an api_content sidecar

Browse source 
The first LLM call of every gateway turn gets ~0% provider prompt-cache
hit rate (in-turn calls: 97-99%) because the bytes sent for a turn's
user message are not the bytes replayed next turn: memory-prefetch and
pre_llm_call context are injected into the API copy only, the #48677
persist override writes cleaned content to the DB row, and
get_messages_as_conversation sanitize/strips user and assistant content
on load. Any of these diverges the request prefix at that message and
re-prefills everything after it — measured 27.9s for the first call vs
2.4-5.8s cached at a median ~156k-token context.

Persist what you send: a nullable messages.api_content column stores the
exact content string sent to the API when it differs from the clean
stored content, and replay substitutes it verbatim (no sanitize, no
strip). The injection composition lives in one helper
(turn_context.compose_user_api_content); the turn prologue stamps its
output onto the live user message, the api_messages build sends the
stamped bytes, and every outgoing copy pops the field so it never
reaches a provider. The crash-resilience user-turn persist moves after
prefetch/pre_llm_call so the user row is written once with its final
sidecar; _ensure_db_session stays before preflight compression (session
rotation needs the parent row under PRAGMA foreign_keys=ON). The
current-turn index trackers are re-anchored after compaction rebuilds
the message list, in-place preflight compaction backfills the stamp onto
the already-inserted row, and gateway replay forwards the sidecar only
when the replay pipeline did not rewrite the content. Rewrite paths that
would leave stale bytes (historical image strip, merge-summary-into-
tail, consecutive-user repair merge, stale-confirmation redaction) drop
the sidecar; the chat-completions transport and the max-iterations
summary path strip it defensively. codex_app_server and MoA turns are
excluded from stamping because their wire bytes differ from the
composition. A missing or dropped sidecar degrades to today's behavior
(one cache-boundary miss), never to wrong content.
This commit is contained in:
Soju06 2026-07-14 07:48:41 +00:00 • committed by kshitij
parent e53f87fe69
commit 7b3dcee928
16 changed files with 1532 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

1
scripts/release.py
View file

@ -83,6 +83,7 @@ LEGACY_AUTHOR_MAP = {
"neo@neodeMac-mini.local": "neo-claw-bot", # PR #58465 salvage (moa: drop empty user turns from advisory view)
"2024104039@mails.szu.edu.cn": "pixel4039", # PR #64420 salvage (streaming: retry zero-chunk streams)
"marceloparra.hm@gmail.com": "marcelohildebrand", # PR #42346 salvage (lmstudio: JIT load mode)
"qlskssk@gmail.com": "Soju06", # agent turn-latency perf PRs
"m.guttmann@journaway.com": "mguttmann", # PR #63738 salvage (Anthropic setup-token pool auth normalization)
"VrtxOmega@pm.me": "VrtxOmega", # PR #43809 salvage (desktop: WSL folder-picker path bridge)
"gn00742754@gmail.com": "SemonCat", # PR #56786 salvage (Slack Agent View manifests and Assistant APIs)