mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-18 04:41:56 +00:00
7b76366552
Cuts input cost for first-turn Claude requests by ~85-90% on subsequent
sessions within an hour. Tools array (~13k tokens for default toolset) +
stable system prefix (~5-8k tokens) get a 1h cache_control marker; the
volatile suffix (memory, USER profile, timestamp, session id) sits in a
separate non-cached block at the end so it doesn't poison the cross-session
prefix when it changes.
Provider gate: Claude on native Anthropic (incl. OAuth subscription),
OpenRouter, and Nous Portal (which proxies to OpenRouter). All other
providers keep today's system_and_3 layout unchanged.
Layout (4 cache_control breakpoints, Anthropic max):
1. tools[-1] -> 1h (cross-session)
2. system content[0] -> 1h (cross-session, stable prefix)
3. messages[-2] -> 5m (within-session rolling)
4. messages[-1] -> 5m (within-session rolling)
Within-session rolling shrinks from 3 messages to 2 to free the breakpoint
budget. On Claude with realistic tool loadouts the long-lived tier carries
the bulk of cross-session value anyway.
System prompt is now always assembled cache-friendly: stable identity /
guidance / skills / platform hints first, then session-stable context
files (AGENTS.md, .cursorrules), then per-call volatile content. Old
single-string callers see the same logical content (same join order),
just reordered so volatile lives at the end.
Config knobs (defaults shown):
prompt_caching:
cache_ttl: "5m" # rolling-window TTL (unchanged)
long_lived_prefix: true # opt-out switch
long_lived_ttl: "1h" # cross-session prefix TTL
Live E2E (tests/agent/test_prompt_caching_live.py, gated on
OPENROUTER_API_KEY) on anthropic/claude-haiku-4.5 with default toolset:
Call 1 (cold): cache_write=13,415 cache_read=0
Call 2 (NEW agent + msg): cache_write=391 cache_read=13,025
Cross-session reuse: 97.09%
Implementation:
* agent/prompt_caching.py: new apply_anthropic_cache_control_long_lived()
+ mark_tools_for_long_lived_cache(); existing apply_anthropic_cache_control()
preserved verbatim for the fallback path.
* agent/anthropic_adapter.py: convert_tools_to_anthropic() now forwards
cache_control onto each Anthropic-format tool dict.
* run_agent.py: _build_system_prompt_parts() returns the 3-tier dict;
_build_system_prompt() joins them (backward compatible).
_supports_long_lived_anthropic_cache() policy added next to the existing
_anthropic_prompt_cache_policy() (which now also recognises Nous Portal
Claude — pre-existing gap fixed in passing).
_build_api_kwargs() resolves tools_for_api once and propagates the
marker through all four build paths (anthropic_messages, bedrock,
codex_responses, profile/legacy chat completions).
Long-lived flag plumbed into the runtime snapshot/restore + model-switch
+ fallback-promotion paths.
Tests:
* tests/agent/test_prompt_caching.py: +8 tests (TestMarkToolsForLongLivedCache,
TestApplyAnthropicCacheControlLongLived).
* tests/run_agent/test_anthropic_prompt_cache_policy.py: +9 tests
(TestSupportsLongLivedAnthropicCache matrix across 8 endpoint classes
+ a fallback-target case).
* tests/agent/test_prompt_caching_live.py: new live E2E (skipif when
OPENROUTER_API_KEY is unset; runs outside the hermetic suite).
* Targeted suites: 327/327 pass (caching/adapter/policy/builder).
* tests/agent/ + tests/run_agent/: 3992 pass, 17 skip, 1 pre-existing
flake (test_async_httpx_del_neuter::test_same_key_replaces_stale_loop_entry,
verified failing on pristine origin/main).
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_413_compression.py | ||
| test_860_dedup.py | ||
| test_1630_context_overflow_loop.py | ||
| test_agent_guardrails.py | ||
| test_agent_loop.py | ||
| test_agent_loop_tool_calling.py | ||
| test_agent_loop_vllm.py | ||
| test_anthropic_error_handling.py | ||
| test_anthropic_prompt_cache_policy.py | ||
| test_anthropic_third_party_oauth_guard.py | ||
| test_anthropic_truncation_continuation.py | ||
| test_api_max_retries_config.py | ||
| test_async_httpx_del_neuter.py | ||
| test_background_review.py | ||
| test_background_review_summary.py | ||
| test_background_review_toolset_restriction.py | ||
| test_codex_multimodal_tool_result.py | ||
| test_commit_memory_session_context_engine.py | ||
| test_compress_focus_plugin_fallback.py | ||
| test_compression_boundary.py | ||
| test_compression_boundary_hook.py | ||
| test_compression_feasibility.py | ||
| test_compression_persistence.py | ||
| test_compression_trigger_excludes_reasoning.py | ||
| test_compressor_fallback_update.py | ||
| test_concurrent_interrupt.py | ||
| test_context_token_tracking.py | ||
| test_copilot_native_vision_headers.py | ||
| test_create_openai_client_kwargs_isolation.py | ||
| test_create_openai_client_proxy_env.py | ||
| test_create_openai_client_reuse.py | ||
| test_deepseek_reasoning_content_echo.py | ||
| test_deepseek_v4_thinking_live.py | ||
| test_dict_tool_call_args.py | ||
| test_empty_response_recovery_persistence.py | ||
| test_exit_cleanup_interrupt.py | ||
| test_fallback_model.py | ||
| test_image_rejection_fallback.py | ||
| test_image_shrink_recovery.py | ||
| test_init_fallback_on_exhausted_pool.py | ||
| test_interactive_interrupt.py | ||
| test_interrupt_propagation.py | ||
| test_invalid_context_length_warning.py | ||
| test_iteration_budget_race.py | ||
| test_jsondecodeerror_retryable.py | ||
| test_last_reasoning_per_turn.py | ||
| test_long_context_tier_429.py | ||
| test_materialize_data_url_cleanup.py | ||
| test_memory_nudge_counter_hydration.py | ||
| test_memory_provider_init.py | ||
| test_memory_sync_interrupted.py | ||
| test_message_sequence_repair.py | ||
| test_openai_client_lifecycle.py | ||
| test_percentage_clamp.py | ||
| test_plugin_context_engine_init.py | ||
| test_primary_runtime_restore.py | ||
| test_provider_attribution_headers.py | ||
| test_provider_fallback.py | ||
| test_provider_parity.py | ||
| test_real_interrupt_subagent.py | ||
| test_redirect_stdout_issue.py | ||
| test_repair_tool_call_arguments.py | ||
| test_repair_tool_call_name.py | ||
| test_review_prompt_class_first.py | ||
| test_run_agent.py | ||
| test_run_agent_codex_responses.py | ||
| test_run_agent_multimodal_prologue.py | ||
| test_sequential_chats_live.py | ||
| test_session_meta_filtering.py | ||
| test_session_reset_fix.py | ||
| test_steer.py | ||
| test_stream_drop_logging.py | ||
| test_stream_interrupt_retry.py | ||
| test_streaming.py | ||
| test_streaming_tool_call_repair.py | ||
| test_strict_api_validation.py | ||
| test_strip_reasoning_tags_cli.py | ||
| test_switch_model_context.py | ||
| test_switch_model_fallback_prune.py | ||
| test_thinking_only_sanitizer.py | ||
| test_token_persistence_non_cli.py | ||
| test_tool_arg_coercion.py | ||
| test_tool_call_args_sanitizer.py | ||
| test_tool_call_guardrail_runtime.py | ||
| test_tool_executor_contextvar_propagation.py | ||
| test_unicode_ascii_codec.py | ||
| test_vision_aware_preprocessing.py | ||