hermes-agent/tests/agent
Teknium 77c0bc6b13
fix(curator): defer first run and add --dry-run preview (#18373) (#18389)
* fix(curator): defer first run and add --dry-run preview (#18373)

Curator was meant to run 7 days after install, not on the very first
gateway tick. On a fresh install (no .curator_state), should_run_now()
returned True immediately because last_run_at was None — so the gateway
cron ticker fired Curator against a fresh skill library moments after
'hermes update'. Combined with the binary 'agent-created' provenance
model (anything not bundled and not hub-installed), this consolidated
hand-authored user workflow skills without consent.

Changes:
- should_run_now(): first observation seeds last_run_at='now' and returns
  False. The next real pass fires one full interval_hours later (7 days
  by default), matching the original design intent.
- hermes curator run --dry-run: produces the same review report without
  applying automatic transitions OR permitting the LLM to call
  skill_manage / terminal mv. A DRY-RUN banner is prepended to the
  prompt and the caller skips apply_automatic_transitions. State is
  NOT advanced so a preview doesn't defer the next scheduled real pass.
- hermes update: prints a one-liner on fresh installs pointing at
  --dry-run, pause, and the docs. Silent on steady state.
- Docs: curator.md and cli-commands.md explain the deferred first-run
  behavior and warn that hand-written SKILL.md files share the
  'agent-created' bucket, with guidance to pin or preview before the
  first pass.

Tests:
- test_first_run_defers replaces the old 'first run always eligible'
  assertion — same fixture, inverted expectation.
- test_maybe_run_curator_defers_on_fresh_install covers the gateway tick
  path end-to-end.
- Three new dry-run tests cover state-advance suppression, prompt
  banner injection, and apply_automatic_transitions skipping.

Fixes #18373.

* feat(curator): pre-run backup + rollback (#18373)

Every real curator pass now snapshots ~/.hermes/skills/ into
~/.hermes/skills/.curator_backups/<utc-iso>/skills.tar.gz before calling
apply_automatic_transitions or the LLM review. If a run consolidates or
archives something the user didn't want touched, 'hermes curator
rollback' restores the tree in one command. Dry-run is skipped — no
mutation means no snapshot needed.

Changes:
- agent/curator_backup.py (new): tar.gz snapshot + safe rollback. The
  snapshot excludes .curator_backups/ (would recurse) and .hub/ (managed
  by the skills hub). Extract refuses absolute paths and .. components,
  and uses tarfile's filter='data' on Python 3.12+. Rollback takes a
  pre-rollback safety snapshot FIRST, stages the current tree into
  .rollback-staging-<ts>/ so the extract lands in an empty dir, and
  cleans the staging dir on success. A failed extract restores the
  staged contents.
- agent/curator.py: run_curator_review() calls curator_backup.
  snapshot_skills(reason='pre-curator-run') before apply_automatic_
  transitions. Best-effort — a failed snapshot logs at debug and the
  run continues (a transient disk issue shouldn't silently disable
  curator forever).
- hermes_cli/curator.py: new 'hermes curator backup' and 'hermes curator
  rollback' subcommands. rollback supports --list, --id <ts>, -y.
- hermes_cli/config.py: curator.backup.{enabled, keep} config block
  with sane defaults (enabled=true, keep=5).
- Docs: curator.md gets a 'Backups and rollback' section; cli-commands
  .md table gets the new rows.

Tests (new file tests/agent/test_curator_backup.py, 16 cases):
- snapshot creates tarball + manifest with correct counts
- snapshot excludes .curator_backups/ (recursion guard) and .hub/
- snapshot disabled via config returns None without creating anything
- snapshot uniquifies ids within the same second (-01 suffix)
- prune honors keep count, newest-first
- list_backups + _resolve_backup cover newest-default and unknown-id
- rollback restores a deleted skill with content intact
- rollback is itself undoable — safety snapshot shows up in list_backups
- rollback with no snapshots returns an error
- rollback refuses tarballs with absolute paths or .. components
- real curator runs take a 'pre-curator-run' snapshot; dry-runs do not

All curator tests: 210 passing locally.
2026-05-01 09:49:59 -07:00
..
transports fix(deepseek): preserve v4 reasoning_content on replay 2026-04-30 11:18:39 -07:00
__init__.py test: add unit tests for 8 modules (batch 2) 2026-02-26 13:54:20 +03:00
test_anthropic_adapter.py fix(anthropic): reactive recovery for OAuth 1M-context beta rejection (#17752) 2026-04-29 21:56:54 -07:00
test_anthropic_keychain.py fix: re-auth on stale OAuth token; read Claude Code credentials from macOS Keychain 2026-04-24 07:14:00 -07:00
test_auxiliary_client.py fix(aux): remove hardcoded Codex fallback model, drop Codex from auto chain (#17765) 2026-04-29 23:23:50 -07:00
test_auxiliary_client_anthropic_custom.py fix(anthropic): complete third-party Anthropic-compatible provider support (#12846) 2026-04-19 22:43:09 -07:00
test_auxiliary_config_bridge.py fix: remove legacy compression.summary_* config and env var fallbacks (#8992) 2026-04-13 04:59:26 -07:00
test_auxiliary_main_first.py fix(copilot): send vision header for Copilot vision requests 2026-04-27 08:35:50 -07:00
test_auxiliary_named_custom_providers.py fix(fallback): let custom_providers shadow built-in aliases 2026-04-30 20:18:44 -07:00
test_auxiliary_transport_autodetect.py fix(auxiliary): auto-detect Anthropic Messages transport for all aux clients (#17027) 2026-04-28 06:50:14 -07:00
test_bedrock_1m_context.py fix(bedrock): send context-1m-2025-08-07 beta so Opus 4.6/4.7 get 1M context (#16793) 2026-04-27 20:41:36 -07:00
test_bedrock_adapter.py test(bedrock): add model picker and region routing tests 2026-04-28 03:53:11 -07:00
test_bedrock_integration.py fix(agent): handle aws_sdk auth type in resolve_provider_client 2026-04-24 07:26:07 -07:00
test_codex_cloudflare_headers.py fix(aux): remove hardcoded Codex fallback model, drop Codex from auto chain (#17765) 2026-04-29 23:23:50 -07:00
test_compress_focus.py fix: resolve CI test failures — add missing functions, fix stale tests (#9483) 2026-04-14 01:43:45 -07:00
test_compressor_image_tokens.py feat(image-input): native multimodal routing based on model vision capability (#16506) 2026-04-27 06:27:59 -07:00
test_context_compressor.py test(context_compressor): regression test for PR #17025 tail-protection off-by-one 2026-04-30 20:00:01 -07:00
test_context_engine.py feat: wire context engine plugin slot into agent and plugin system 2026-04-10 19:15:50 -07:00
test_context_references.py fix(agent): fall back when rg is blocked for @folder references 2026-04-20 01:56:41 -07:00
test_copilot_acp_client.py fix(ci): recover 38 failing tests on main (#17642) 2026-04-29 20:05:32 -07:00
test_credential_pool.py fix(codex): resync pool entry from auth.json after reauth (#17001) 2026-04-28 05:43:09 -07:00
test_credential_pool_routing.py refactor: remove smart_model_routing feature (#12732) 2026-04-19 18:12:55 -07:00
test_crossloop_client_cache.py refactor(tests): re-architect tests + fix CI failures (#5946) 2026-04-07 17:19:07 -07:00
test_curator.py fix(curator): defer first run and add --dry-run preview (#18373) (#18389) 2026-05-01 09:49:59 -07:00
test_curator_activity.py fix: use skill activity in curator status 2026-04-30 10:31:47 -07:00
test_curator_backup.py fix(curator): defer first run and add --dry-run preview (#18373) (#18389) 2026-05-01 09:49:59 -07:00
test_curator_classification.py feat(curator): split archived into consolidated vs pruned with model + heuristic classification (#17941) 2026-04-30 10:31:23 -07:00
test_curator_reports.py fix(curator): rewrite cron job skill refs after consolidation (#18253) 2026-04-30 23:04:50 -07:00
test_deepseek_anthropic_thinking.py test(anthropic): regression guard for DeepSeek /anthropic thinking replay 2026-04-29 08:10:29 -07:00
test_direct_provider_url_detection.py fix: restrict provider URL detection to exact hostname matches 2026-04-20 22:14:29 -07:00
test_display.py fix(display): render <missing old_text> in memory previews instead of empty quotes (#12852) 2026-04-19 22:45:47 -07:00
test_display_emoji.py feat(tools): centralize tool emoji metadata in registry + skin integration 2026-03-15 20:21:21 -07:00
test_error_classifier.py fix(anthropic): reactive recovery for OAuth 1M-context beta rejection (#17752) 2026-04-29 21:56:54 -07:00
test_external_skills.py feat(skills): support external skill directories via config (#3678) 2026-03-29 00:33:30 -07:00
test_gemini_cloudcode.py fix(gemini): assign unique stream indices to parallel tool calls 2026-04-20 02:10:53 -07:00
test_gemini_free_tier_gate.py feat(gemini): block free-tier keys at setup + surface guidance on 429 (#15100) 2026-04-24 04:46:17 -07:00
test_gemini_native_adapter.py fix(gemini): fail fast on missing API key + surface it in hermes dump (#15133) 2026-04-24 05:35:17 -07:00
test_gemini_schema.py fix(gemini): drop integer/number/boolean enums from tool schemas (#15082) 2026-04-24 03:40:00 -07:00
test_image_gen_registry.py feat(plugins): pluggable image_gen backends + OpenAI provider (#13799) 2026-04-21 21:30:10 -07:00
test_image_routing.py feat(image-input): native multimodal routing based on model vision capability (#16506) 2026-04-27 06:27:59 -07:00
test_insights.py test: stop testing mutable data — convert change-detectors to invariants (#13363) 2026-04-20 23:20:33 -07:00
test_kimi_coding_anthropic_thinking.py fix(anthropic): broaden Kimi thinking-suppression to custom endpoints (#17455) 2026-04-29 06:35:42 -07:00
test_local_stream_timeout.py fix(agent): recognize Tailscale CGNAT (100.64.0.0/10) as local for Ollama timeouts 2026-04-22 14:46:10 -07:00
test_memory_provider.py fix(memory): add write origin metadata 2026-04-24 14:37:55 -07:00
test_memory_session_switch.py fix(hindsight): route flush-on-switch through writer queue, not raw thread 2026-04-29 08:09:03 -07:00
test_memory_user_id.py feat(hindsight): richer session-scoped retain metadata 2026-04-22 05:27:10 -07:00
test_minimax_auxiliary_url.py fix: provider/model resolution — salvage 4 PRs + MiniMax aux URL fix (#5983) 2026-04-07 22:23:28 -07:00
test_minimax_provider.py fix(ci): stabilize main test suite regressions (#17660) 2026-04-29 23:18:55 -07:00
test_model_metadata.py revert: computer-use cua-driver (PR #16919) (#16927) 2026-04-28 01:57:21 -07:00
test_model_metadata_local_ctx.py fix(tui): show correct context length 2026-04-28 12:27:36 -07:00
test_model_metadata_ssl.py fix(auth): honor SSL CA env vars across httpx + requests callsites 2026-04-24 03:00:33 -07:00
test_models_dev.py feat: add Step Plan provider support (salvage #6005) 2026-04-22 02:59:58 -07:00
test_moonshot_schema.py fix(moonshot): also strip nullable/enum after anyOf collapse 2026-04-30 23:14:31 -07:00
test_nous_rate_guard.py fix(nous): don't trip cross-session rate breaker on upstream-capacity 429s (#15898) 2026-04-26 04:53:42 -07:00
test_onboarding.py docs(onboarding): lead OpenClaw residue banner with migrate, warn that cleanup breaks OpenClaw (#17507) 2026-04-29 08:08:36 -07:00
test_prompt_builder.py feat(agent): add PLATFORM_HINTS for matrix, mattermost, and feishu (#14428) 2026-04-23 12:50:22 +05:30
test_prompt_caching.py fix(prompt-caching): skip top-level cache_control on role:tool for OpenRouter 2026-03-21 16:54:43 -07:00
test_proxy_and_url_validation.py fix(agent): normalize socks:// env proxies for httpx/anthropic 2026-04-21 05:52:46 -07:00
test_rate_limit_tracker.py feat: capture provider rate limit headers and show in /usage (#6541) 2026-04-09 03:43:14 -07:00
test_redact.py feat: replace kimi-k2.5 with kimi-k2.6 on OpenRouter and Nous Portal (#13148) 2026-04-20 11:49:54 -07:00
test_shell_hooks.py feat: shell hooks — wire shell scripts as Hermes hook callbacks 2026-04-20 20:53:51 -07:00
test_shell_hooks_consent.py fix(shell_hooks): parse hooks_auto_accept as strict bool/string, not bool() (#16322) 2026-04-26 20:48:35 -07:00
test_skill_commands.py refactor(commands): drop /provider, /plan handler, and clean up slash registry (#15047) 2026-04-24 03:10:52 -07:00
test_skill_commands_reload.py refactor(reload-skills): queue note for next turn, drop cache invalidation + agent tool 2026-04-29 21:07:47 -07:00
test_skill_utils.py test(skill_utils): add regression tests for non-dict metadata in extract_skill_conditions 2026-04-30 20:37:15 -07:00
test_streaming_context_scrubber.py style: trim verbose comment blocks added by previous commit 2026-04-27 12:37:33 -07:00
test_subagent_progress.py feat(delegate): orchestrator role and configurable spawn depth (default flat) 2026-04-21 14:23:45 -07:00
test_subagent_stop_hook.py feat: shell hooks — wire shell scripts as Hermes hook callbacks 2026-04-20 20:53:51 -07:00
test_subdirectory_hints.py fix(agent): catch PermissionError in subdirectory hint discovery 2026-04-09 03:10:30 -07:00
test_title_generator.py fix(auxiliary): custom provider URL rewrite + main_runtime model for title gen 2026-04-28 01:47:25 -07:00
test_tool_guardrails.py fix(agent): make tool loop guardrails warning-first 2026-04-30 20:43:15 -07:00
test_unsupported_parameter_retry.py fix(auxiliary): generalize unsupported-parameter detector and harden max_tokens retry (#15633) 2026-04-25 05:50:34 -07:00
test_unsupported_temperature_retry.py refactor(memory): remove flush_memories entirely (#15696) 2026-04-25 08:21:14 -07:00
test_usage_pricing.py fix(usage): read top-level Anthropic cache fields from OAI-compatible proxies 2026-04-22 17:40:49 -07:00
test_vision_resolved_args.py fix: pass resolved args to resolve_vision_provider_client() 2026-04-16 07:45:13 -07:00