mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-31 19:16:29 +00:00
e2dfa843ef
The first API call of a fresh desktop/TUI session pays provider-side ingestion of the entire uncached prefix (system prompt + tool schemas, commonly 50-70k tokens) — observed as ~20s first-message latency on Anthropic-cached routes, vs 4-7s on every later 100%-cache-hit turn. When agent.prewarm_prompt_cache is enabled (config.yaml, default off), the gateway issues one minimal non-streaming max_tokens=1 request right after the session agent is built — same tool schemas, same system prompt with the same [static, volatile] cache_control layout — so the provider writes the prompt-prefix cache BEFORE the first user message. The first real turn then reads a warm prefix instead of writing it cold. Measured on a live desktop-shaped session (nous / claude-fable-5, 67k prefix): prewarm 2.7s off the response path, first real turn 3.8s with cache=67045/67131 (100%) — down from 20.4s cold. Details: - agent/prompt_prewarm.py: pure helper; supported only where the request shape is reproducible (chat_completions / anthropic_messages, not MoA/ Codex/Bedrock/ACP) and _use_prompt_caching is on. Thinking/reasoning knobs are stripped (max_tokens=1 violates budget_tokens; thinking changes don't invalidate system/tools cache blocks). Fail-open: any failure returns False and the first real turn pays the write itself. - The exact sent prompt is handed to the first real turn via _prewarmed_system_prompt; _restore_or_build_system_prompt adopts it (gated on runtime-identity match, no history, no custom system message) so the volatile tail can't drift and split the just-warmed prefix. One-shot — cleared after every first-turn resolution. - tui_gateway/server.py: _schedule_prompt_prewarm fires from both agent build sites, waits for late MCP discovery first (tools are part of the cached prefix), and skips if the user already started the conversation. Cost note: the cache write (1.25x input, 5m TTL) is paid by the first real call today anyway; prewarming moves it earlier. Extra spend is one 0.1x cache read per session plus wasted writes for sessions opened but never used — which is why it ships default-off. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _stdin_recovery.py | ||
| compute_host.py | ||
| entry.py | ||
| event_publisher.py | ||
| git_probe.py | ||
| host_supervisor.py | ||
| loop_noise.py | ||
| project_tree.py | ||
| render.py | ||
| server.py | ||
| slash_worker.py | ||
| synthetic_turn.py | ||
| transport.py | ||
| turn_marker.py | ||
| ws.py | ||