feat: prompt-cache prewarm for TUI/desktop sessions (agent.prewarm_prompt_cache)

The first API call of a fresh desktop/TUI session pays provider-side
ingestion of the entire uncached prefix (system prompt + tool schemas,
commonly 50-70k tokens) — observed as ~20s first-message latency on
Anthropic-cached routes, vs 4-7s on every later 100%-cache-hit turn.

When agent.prewarm_prompt_cache is enabled (config.yaml, default off),
the gateway issues one minimal non-streaming max_tokens=1 request right
after the session agent is built — same tool schemas, same system prompt
with the same [static, volatile] cache_control layout — so the provider
writes the prompt-prefix cache BEFORE the first user message. The first
real turn then reads a warm prefix instead of writing it cold.

Measured on a live desktop-shaped session (nous / claude-fable-5, 67k
prefix): prewarm 2.7s off the response path, first real turn 3.8s with
cache=67045/67131 (100%) — down from 20.4s cold.

Details:
- agent/prompt_prewarm.py: pure helper; supported only where the request
  shape is reproducible (chat_completions / anthropic_messages, not MoA/
  Codex/Bedrock/ACP) and _use_prompt_caching is on. Thinking/reasoning
  knobs are stripped (max_tokens=1 violates budget_tokens; thinking
  changes don't invalidate system/tools cache blocks). Fail-open: any
  failure returns False and the first real turn pays the write itself.
- The exact sent prompt is handed to the first real turn via
  _prewarmed_system_prompt; _restore_or_build_system_prompt adopts it
  (gated on runtime-identity match, no history, no custom system message)
  so the volatile tail can't drift and split the just-warmed prefix.
  One-shot — cleared after every first-turn resolution.
- tui_gateway/server.py: _schedule_prompt_prewarm fires from both agent
  build sites, waits for late MCP discovery first (tools are part of the
  cached prefix), and skips if the user already started the conversation.

Cost note: the cache write (1.25x input, 5m TTL) is paid by the first
real call today anyway; prewarming moves it earlier. Extra spend is one
0.1x cache read per session plus wasted writes for sessions opened but
never used — which is why it ships default-off.
This commit is contained in:
Teknium 2026-07-27 18:04:16 -07:00
parent d83e858507
commit e2dfa843ef
5 changed files with 511 additions and 2 deletions

View file

@ -974,6 +974,17 @@ DEFAULT_CONFIG = {
# on a genuinely hung build. Raise it for deployments with many slow
# or unreachable MCP servers.
"build_wait_timeout": 600,
# Prompt-cache prewarm (TUI/desktop): right after a session's agent is
# built, issue one minimal max_tokens=1 request so the provider writes
# the prompt-prefix cache (system prompt + tool schemas) BEFORE the
# first user message. Cuts cold first-message latency from provider
# ingestion of a 50-70k-token uncached prefix (10-20s observed) down
# to a cache read. Off by default: the write (1.25x input for the 5m
# TTL) is wasted whenever a session is opened but never used, and the
# first real turn additionally pays one cache read (0.1x). Only
# applies to prompt-caching routes (Anthropic-compatible); no-op
# elsewhere.
"prewarm_prompt_cache": False,
# Max app-level retry attempts for API errors (connection drops,
# provider timeouts, 5xx, etc.) before the agent surfaces the
# failure. The OpenAI SDK already does its own low-level retries