mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-14 14:12:44 +00:00
docs: warn that mid-session model switches break prompt caching (#58747)
/model switches, primary-model fallback, and credential-pool key rotation all change the prompt-cache key (model and/or account), so the next turn re-reads the entire conversation at full input price. Add cost warnings everywhere docs recommend or describe these paths: - reference/slash-commands.md: cost note on both /model rows - user-guide/features/fallback-providers.md: warning admonition - user-guide/features/credential-pools.md: warning admonition - user-guide/configuring-models.md: mid-session switch warning - guides/tips.md: expand cache tip + /model tip - reference/faq.md: warning on the switch-back-and-forth example - user-guide/desktop.md: composer picker bullet - developer-guide/context-compression-and-caching.md: new cache-aware design pattern (model identity is part of the key)
This commit is contained in:
parent
7fde19afcc
commit
1c156736dc
8 changed files with 31 additions and 4 deletions
|
|
@ -361,6 +361,16 @@ The marker is applied differently based on content type:
|
|||
4. **TTL selection**: Default is `5m` (5 minutes). Use `1h` for long-running
|
||||
sessions where the user takes breaks between turns.
|
||||
|
||||
5. **Model identity is part of the cache key**: Provider-side caches are scoped
|
||||
to the model (and account/API key) serving the request. Any mid-conversation
|
||||
model change — an explicit `/model` switch, primary-model fallback, or a
|
||||
credential-pool rotation onto a different account — means the next request
|
||||
gets zero cache hits and re-reads the full conversation at undiscounted
|
||||
input price. This is inherent to how provider caches work, not something
|
||||
Hermes can avoid; user-facing docs for `/model`, fallback providers, and
|
||||
credential pools carry cost warnings for this reason. Don't add features
|
||||
that silently swap the model or credentials mid-session.
|
||||
|
||||
### Enabling Prompt Caching
|
||||
|
||||
Prompt caching is automatically enabled when:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue