mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-21 16:18:55 +00:00
7b3dcee928
The first LLM call of every gateway turn gets ~0% provider prompt-cache hit rate (in-turn calls: 97-99%) because the bytes sent for a turn's user message are not the bytes replayed next turn: memory-prefetch and pre_llm_call context are injected into the API copy only, the #48677 persist override writes cleaned content to the DB row, and get_messages_as_conversation sanitize/strips user and assistant content on load. Any of these diverges the request prefix at that message and re-prefills everything after it — measured 27.9s for the first call vs 2.4-5.8s cached at a median ~156k-token context. Persist what you send: a nullable messages.api_content column stores the exact content string sent to the API when it differs from the clean stored content, and replay substitutes it verbatim (no sanitize, no strip). The injection composition lives in one helper (turn_context.compose_user_api_content); the turn prologue stamps its output onto the live user message, the api_messages build sends the stamped bytes, and every outgoing copy pops the field so it never reaches a provider. The crash-resilience user-turn persist moves after prefetch/pre_llm_call so the user row is written once with its final sidecar; _ensure_db_session stays before preflight compression (session rotation needs the parent row under PRAGMA foreign_keys=ON). The current-turn index trackers are re-anchored after compaction rebuilds the message list, in-place preflight compaction backfills the stamp onto the already-inserted row, and gateway replay forwards the sidecar only when the replay pipeline did not rewrite the content. Rewrite paths that would leave stale bytes (historical image strip, merge-summary-into- tail, consecutive-user repair merge, stale-confirmation redaction) drop the sidecar; the chat-completions transport and the max-iterations summary path strip it defensively. codex_app_server and MoA turns are excluded from stamping because their wire bytes differ from the composition. A missing or dropped sidecar degrades to today's behavior (one cache-boundary miss), never to wrong content. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| anthropic.py | ||
| base.py | ||
| bedrock.py | ||
| chat_completions.py | ||
| codex.py | ||
| codex_app_server.py | ||
| codex_app_server_session.py | ||
| codex_event_projector.py | ||
| hermes_tools_mcp_server.py | ||
| types.py | ||