hermes-agent/tests/agent
Chris Danis 28f4d6db63 fix(tool-schemas): reactive strip of pattern/format on llama.cpp grammar 400s
MCP servers commonly emit JSON Schema `pattern` (e.g. `\\d{4}-\\d{2}-\\d{2}`
for date-time params) and `format` keywords. llama.cpp's
`json-schema-to-grammar` converter rejects regex escape classes
(\\d/\\w/\\s) and most format values, returning HTTP 400
"parse: error parsing grammar: unknown escape at \\d" — the whole request
fails.

Cloud providers (OpenAI, Anthropic, OpenRouter, Gemini) accept these
keywords fine and use them as prompting hints. Stripping unconditionally
loses useful hints for every cloud user to fix a llama.cpp-only bug.

Approach: classify the llama.cpp grammar-parse 400 in the error
classifier, and on match do a one-shot in-place strip of pattern/format
from `self.tools`, then retry. Follows the existing
`thinking_signature` recovery pattern. Cloud users hit zero overhead;
llama.cpp users pay one failed request per session.

Changes
- agent/error_classifier.py: new `FailoverReason.llama_cpp_grammar_pattern`
  + narrow HTTP-400 branch matching "error parsing grammar",
  "json-schema-to-grammar", or "unable to generate parser ... template".
- tools/schema_sanitizer.py: new `strip_pattern_and_format()` helper —
  reactive, walks schema nodes, skips property names (search_files.pattern
  survives). Returns strip count for logging.
- run_agent.py: new one-shot recovery block in the retry loop. Strips,
  logs, continues. Falls through to normal retry if nothing to strip.
- tests: 4 classifier tests (3 variants + 1 non-400 negative), 7 strip
  tests including the property-name preservation and idempotency checks.

Co-authored-by: Chris Danis <cdanis@gmail.com>
2026-05-05 04:25:18 -07:00
..
transports fix(codex-transport): preserve request override headers for xai responses 2026-05-03 15:25:45 -07:00
__init__.py test: add unit tests for 8 modules (batch 2) 2026-02-26 13:54:20 +03:00
test_anthropic_adapter.py fix(anthropic): restrict fast mode to Opus 4.6 (Anthropic API contract) 2026-05-04 06:23:52 -07:00
test_anthropic_keychain.py fix: re-auth on stale OAuth token; read Claude Code credentials from macOS Keychain 2026-04-24 07:14:00 -07:00
test_auxiliary_client.py fix(auxiliary): propagate explicit_api_key to _try_anthropic() 2026-05-03 17:00:55 -07:00
test_auxiliary_client_anthropic_custom.py fix(anthropic): complete third-party Anthropic-compatible provider support (#12846) 2026-04-19 22:43:09 -07:00
test_auxiliary_config_bridge.py fix: remove legacy compression.summary_* config and env var fallbacks (#8992) 2026-04-13 04:59:26 -07:00
test_auxiliary_main_first.py fix(copilot): send vision header for Copilot vision requests 2026-04-27 08:35:50 -07:00
test_auxiliary_named_custom_providers.py fix(fallback): let custom_providers shadow built-in aliases 2026-04-30 20:18:44 -07:00
test_auxiliary_transport_autodetect.py fix(auxiliary): auto-detect Anthropic Messages transport for all aux clients (#17027) 2026-04-28 06:50:14 -07:00
test_bedrock_1m_context.py fix(bedrock): send context-1m-2025-08-07 beta so Opus 4.6/4.7 get 1M context (#16793) 2026-04-27 20:41:36 -07:00
test_bedrock_adapter.py fix(test): skip bedrock adapter tests when botocore is not installed 2026-05-04 04:41:55 -07:00
test_bedrock_integration.py fix(agent): handle aws_sdk auth type in resolve_provider_client 2026-04-24 07:26:07 -07:00
test_codex_cloudflare_headers.py fix(aux): remove hardcoded Codex fallback model, drop Codex from auto chain (#17765) 2026-04-29 23:23:50 -07:00
test_compress_focus.py fix: resolve CI test failures — add missing functions, fix stale tests (#9483) 2026-04-14 01:43:45 -07:00
test_compressor_image_tokens.py feat(image-input): native multimodal routing based on model vision capability (#16506) 2026-04-27 06:27:59 -07:00
test_context_compressor.py fix(compressor): _prune_old_tool_results boundary direction 2026-05-04 05:05:18 -07:00
test_context_engine.py feat: wire context engine plugin slot into agent and plugin system 2026-04-10 19:15:50 -07:00
test_context_references.py fix(agent): fall back when rg is blocked for @folder references 2026-04-20 01:56:41 -07:00
test_copilot_acp_client.py fix(ci): recover 38 failing tests on main (#17642) 2026-04-29 20:05:32 -07:00
test_credential_pool.py test(credential_pool): regression coverage for .env vs os.environ precedence 2026-05-02 02:00:32 -07:00
test_credential_pool_routing.py refactor: remove smart_model_routing feature (#12732) 2026-04-19 18:12:55 -07:00
test_crossloop_client_cache.py refactor(tests): re-architect tests + fix CI failures (#5946) 2026-04-07 17:19:07 -07:00
test_curator.py fix(skills): keep manual skills out of curator 2026-05-04 02:19:28 -07:00
test_curator_activity.py fix: use skill activity in curator status 2026-04-30 10:31:47 -07:00
test_curator_backup.py fix(curator): authoritative absorbed_into on delete + restore cron skill links on rollback (#18671) (#18731) 2026-05-02 01:29:57 -07:00
test_curator_classification.py fix(curator): prevent false-positive consolidation from substring matching 2026-05-04 01:21:23 -07:00
test_curator_reports.py fix(curator): rewrite cron job skill refs after consolidation (#18253) 2026-04-30 23:04:50 -07:00
test_deepseek_anthropic_thinking.py test(anthropic): regression guard for DeepSeek /anthropic thinking replay 2026-04-29 08:10:29 -07:00
test_direct_provider_url_detection.py fix: restrict provider URL detection to exact hostname matches 2026-04-20 22:14:29 -07:00
test_display.py fix(display): render <missing old_text> in memory previews instead of empty quotes (#12852) 2026-04-19 22:45:47 -07:00
test_display_emoji.py feat(tools): centralize tool emoji metadata in registry + skin integration 2026-03-15 20:21:21 -07:00
test_error_classifier.py fix(tool-schemas): reactive strip of pattern/format on llama.cpp grammar 400s 2026-05-05 04:25:18 -07:00
test_external_skills.py feat(skills): support external skill directories via config (#3678) 2026-03-29 00:33:30 -07:00
test_gemini_cloudcode.py fix(gemini): assign unique stream indices to parallel tool calls 2026-04-20 02:10:53 -07:00
test_gemini_free_tier_gate.py feat(gemini): block free-tier keys at setup + surface guidance on 429 (#15100) 2026-04-24 04:46:17 -07:00
test_gemini_native_adapter.py fix(gemini): fail fast on missing API key + surface it in hermes dump (#15133) 2026-04-24 05:35:17 -07:00
test_gemini_schema.py fix(gemini): drop integer/number/boolean enums from tool schemas (#15082) 2026-04-24 03:40:00 -07:00
test_image_gen_registry.py feat(plugins): pluggable image_gen backends + OpenAI provider (#13799) 2026-04-21 21:30:10 -07:00
test_image_routing.py feat(image-input): native multimodal routing based on model vision capability (#16506) 2026-04-27 06:27:59 -07:00
test_insights.py test: stop testing mutable data — convert change-detectors to invariants (#13363) 2026-04-20 23:20:33 -07:00
test_kimi_coding_anthropic_thinking.py fix(anthropic): broaden Kimi thinking-suppression to custom endpoints (#17455) 2026-04-29 06:35:42 -07:00
test_local_stream_timeout.py fix(agent): recognize Tailscale CGNAT (100.64.0.0/10) as local for Ollama timeouts 2026-04-22 14:46:10 -07:00
test_memory_provider.py fix(memory): add write origin metadata 2026-04-24 14:37:55 -07:00
test_memory_session_switch.py fix(hindsight): route flush-on-switch through writer queue, not raw thread 2026-04-29 08:09:03 -07:00
test_memory_user_id.py feat(hindsight): richer session-scoped retain metadata 2026-04-22 05:27:10 -07:00
test_minimax_auxiliary_url.py fix: provider/model resolution — salvage 4 PRs + MiniMax aux URL fix (#5983) 2026-04-07 22:23:28 -07:00
test_minimax_provider.py fix(ci): stabilize main test suite regressions (#17660) 2026-04-29 23:18:55 -07:00
test_model_metadata.py revert: computer-use cua-driver (PR #16919) (#16927) 2026-04-28 01:57:21 -07:00
test_model_metadata_local_ctx.py fix(tui): show correct context length 2026-04-28 12:27:36 -07:00
test_model_metadata_ssl.py fix(auth): honor SSL CA env vars across httpx + requests callsites 2026-04-24 03:00:33 -07:00
test_models_dev.py feat: add Step Plan provider support (salvage #6005) 2026-04-22 02:59:58 -07:00
test_moonshot_schema.py fix(moonshot): also strip nullable/enum after anyOf collapse 2026-04-30 23:14:31 -07:00
test_nous_rate_guard.py fix(nous): don't trip cross-session rate breaker on upstream-capacity 429s (#15898) 2026-04-26 04:53:42 -07:00
test_onboarding.py docs(onboarding): lead OpenClaw residue banner with migrate, warn that cleanup breaks OpenClaw (#17507) 2026-04-29 08:08:36 -07:00
test_openrouter_response_cache.py feat(openrouter): add response caching support (#19132) 2026-05-03 01:54:24 -07:00
test_prompt_builder.py feat(agent): add PLATFORM_HINTS for matrix, mattermost, and feishu (#14428) 2026-04-23 12:50:22 +05:30
test_prompt_caching.py fix(prompt-caching): skip top-level cache_control on role:tool for OpenRouter 2026-03-21 16:54:43 -07:00
test_proxy_and_url_validation.py fix(agent): normalize socks:// env proxies for httpx/anthropic 2026-04-21 05:52:46 -07:00
test_rate_limit_tracker.py feat: capture provider rate limit headers and show in /usage (#6541) 2026-04-09 03:43:14 -07:00
test_redact.py feat: replace kimi-k2.5 with kimi-k2.6 on OpenRouter and Nous Portal (#13148) 2026-04-20 11:49:54 -07:00
test_shell_hooks.py feat: shell hooks — wire shell scripts as Hermes hook callbacks 2026-04-20 20:53:51 -07:00
test_shell_hooks_consent.py fix(shell_hooks): parse hooks_auto_accept as strict bool/string, not bool() (#16322) 2026-04-26 20:48:35 -07:00
test_skill_commands.py fix(skills): rescan skill_commands cache when platform scope changes (#18739) 2026-05-02 01:36:53 -07:00
test_skill_commands_reload.py refactor(reload-skills): queue note for next turn, drop cache invalidation + agent tool 2026-04-29 21:07:47 -07:00
test_skill_utils.py test(skill_utils): add regression tests for non-dict metadata in extract_skill_conditions 2026-04-30 20:37:15 -07:00
test_streaming_context_scrubber.py style: trim verbose comment blocks added by previous commit 2026-04-27 12:37:33 -07:00
test_subagent_progress.py feat(delegate): orchestrator role and configurable spawn depth (default flat) 2026-04-21 14:23:45 -07:00
test_subagent_stop_hook.py feat: shell hooks — wire shell scripts as Hermes hook callbacks 2026-04-20 20:53:51 -07:00
test_subdirectory_hints.py fix(agent): catch PermissionError in subdirectory hint discovery 2026-04-09 03:10:30 -07:00
test_title_generator.py fix: improve telegram topic mode setup 2026-05-04 12:07:17 -07:00
test_tool_guardrails.py fix(agent): make tool loop guardrails warning-first 2026-04-30 20:43:15 -07:00
test_unsupported_parameter_retry.py fix(auxiliary): generalize unsupported-parameter detector and harden max_tokens retry (#15633) 2026-04-25 05:50:34 -07:00
test_unsupported_temperature_retry.py refactor(memory): remove flush_memories entirely (#15696) 2026-04-25 08:21:14 -07:00
test_usage_pricing.py fix(usage): read top-level Anthropic cache fields from OAI-compatible proxies 2026-04-22 17:40:49 -07:00
test_vision_resolved_args.py fix(vision): preserve explicit provider auth with custom base_url 2026-05-04 05:05:43 -07:00