mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-31 19:16:29 +00:00
feat: prompt-cache prewarm for TUI/desktop sessions (agent.prewarm_prompt_cache)
The first API call of a fresh desktop/TUI session pays provider-side ingestion of the entire uncached prefix (system prompt + tool schemas, commonly 50-70k tokens) — observed as ~20s first-message latency on Anthropic-cached routes, vs 4-7s on every later 100%-cache-hit turn. When agent.prewarm_prompt_cache is enabled (config.yaml, default off), the gateway issues one minimal non-streaming max_tokens=1 request right after the session agent is built — same tool schemas, same system prompt with the same [static, volatile] cache_control layout — so the provider writes the prompt-prefix cache BEFORE the first user message. The first real turn then reads a warm prefix instead of writing it cold. Measured on a live desktop-shaped session (nous / claude-fable-5, 67k prefix): prewarm 2.7s off the response path, first real turn 3.8s with cache=67045/67131 (100%) — down from 20.4s cold. Details: - agent/prompt_prewarm.py: pure helper; supported only where the request shape is reproducible (chat_completions / anthropic_messages, not MoA/ Codex/Bedrock/ACP) and _use_prompt_caching is on. Thinking/reasoning knobs are stripped (max_tokens=1 violates budget_tokens; thinking changes don't invalidate system/tools cache blocks). Fail-open: any failure returns False and the first real turn pays the write itself. - The exact sent prompt is handed to the first real turn via _prewarmed_system_prompt; _restore_or_build_system_prompt adopts it (gated on runtime-identity match, no history, no custom system message) so the volatile tail can't drift and split the just-warmed prefix. One-shot — cleared after every first-turn resolution. - tui_gateway/server.py: _schedule_prompt_prewarm fires from both agent build sites, waits for late MCP discovery first (tools are part of the cached prefix), and skips if the user already started the conversation. Cost note: the cache write (1.25x input, 5m TTL) is paid by the first real call today anyway; prewarming moves it earlier. Extra spend is one 0.1x cache read per session plus wasted writes for sessions opened but never used — which is why it ships default-off.
This commit is contained in:
parent
d83e858507
commit
e2dfa843ef
5 changed files with 511 additions and 2 deletions
|
|
@ -974,6 +974,17 @@ DEFAULT_CONFIG = {
|
|||
# on a genuinely hung build. Raise it for deployments with many slow
|
||||
# or unreachable MCP servers.
|
||||
"build_wait_timeout": 600,
|
||||
# Prompt-cache prewarm (TUI/desktop): right after a session's agent is
|
||||
# built, issue one minimal max_tokens=1 request so the provider writes
|
||||
# the prompt-prefix cache (system prompt + tool schemas) BEFORE the
|
||||
# first user message. Cuts cold first-message latency from provider
|
||||
# ingestion of a 50-70k-token uncached prefix (10-20s observed) down
|
||||
# to a cache read. Off by default: the write (1.25x input for the 5m
|
||||
# TTL) is wasted whenever a session is opened but never used, and the
|
||||
# first real turn additionally pays one cache read (0.1x). Only
|
||||
# applies to prompt-caching routes (Anthropic-compatible); no-op
|
||||
# elsewhere.
|
||||
"prewarm_prompt_cache": False,
|
||||
# Max app-level retry attempts for API errors (connection drops,
|
||||
# provider timeouts, 5xx, etc.) before the agent surfaces the
|
||||
# failure. The OpenAI SDK already does its own low-level retries
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue