mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-06-27 11:22:03 +00:00
7 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Brooklyn Nicholson
|
a5849917a8 |
test(pets): make slow pet generation suite opt-in
The pet generation image-processing suite is deterministic but expensive enough to blow the per-file CI timeout on Linux (140s), and it is not relevant to the fast timeout PR's normal signal. Keep it available for manual validation, but do not run it by default. Set HERMES_RUN_SLOW_PET_TESTS=1 to enable the suite. The canonical test wrapper now preserves that opt-in variable through its hermetic env. |
||
|
ethernet
|
2f9d18711f |
fix(ci): remove pytest-timeout, use per-file timeout only
fix(ci): write a new cache for test durations every time change(ci): rip out error 4 retries because we found the real bug |
||
|
ethernet
|
48be2e0e4d
|
test: use subprocesses for each test file (#29016)
* ci(tests): install ripgrep from prebuilt tarball instead of apt
apt-get update + install of ripgrep takes ~4 min on the GHA Ubuntu
runners (the apt-get update against archive.ubuntu.com is the slow
part; ripgrep itself is small). Switching to the upstream musl
binary tarball cuts the step to a few seconds.
- Pinned to ripgrep 15.1.0 with sha256 verification (same hash as
published in the releases sha256 sidecar file).
- Drops the `rg` binary into /usr/local/bin so it is on PATH for
every subsequent step without GITHUB_PATH manipulation.
- Applied to both the test and e2e jobs in tests.yml.
* fix(cli): compile syntax check to tempdir, not source __pycache__
`_validate_critical_files_syntax` runs `py_compile.compile()` on each
critical bootstrap file after a successful `git pull`. The default
`py_compile` writes the resulting `.pyc` next to the source under
`__pycache__/`, which causes two real problems:
1. Parallel test workers walking the same source tree (e.g. running
the suite under per-file process isolation) can race against each
other on the `__pycache__` write — manifests as flaky 'directory
not empty' errors during teardown.
2. In production, the post-pull syntax check leaves a `.pyc` behind
that the next interpreter run might pick up — fine when the
interpreter version matches, sketchy if it doesn't.
Fix: write the compiled output to a `tempfile.TemporaryDirectory()`
that's discarded on function exit. We only care about the compile-or-not
signal, not the artifact.
* test(runner): per-file process isolation, drop manual state reset + xdist
Replace fragile manual _reset_module_state test fixtures with robust
per-file subprocess isolation. Each test file runs in a fresh
`python -m pytest <file>` subprocess via ThreadPoolExecutor. No xdist,
no custom pytest plugin, no shared worker state.
Key changes:
* scripts/run_tests_parallel.py — new runner: discovers test files,
runs N in parallel via ThreadPoolExecutor, captures stdout per file,
treats exit code 5 (no tests collected) as pass, kills all children
on exit. Change from cpu_count to cpu_count*2. The runner is
I/O-bound (waiting on subprocess.communicate() from pytest children)
The parent process does almost no CPU work, so 2x oversubscription
keeps more pipes full. When a file fails, immediately show the last
30 lines of pytest output (stack traces + FAILED summary) plus a
ready-to-copy repro command:
python -m pytest tests/agent/test_auxiliary_client.py
* scripts/run_tests.sh — delegates to run_tests_parallel.py
* .github/workflows/tests.yml — test step: python
scripts/run_tests_parallel.py
* pyproject.toml — drop pytest-xdist, pytest-split; simplify addopts
* tests/conftest.py — remove ~200 lines of manual state-reset fixtures
* AGENTS.md — update Testing section for per-file design
* test(runner): speed gateway test antipattern scan up
* fix(test): web search provider plugin test missing xai
* fix(tests): make 14 test files pass under per-file subprocess isolation
Tests that relied on cross-file state pollution from xdist workers
fail when run in isolation (per-file subprocess model). Root causes
and fixes:
Tool registry not populated:
- test_video_generation_tool_surface_matrix: add discover_builtin_tools()
- test_web_providers_brave_free/ddgs/searxng/general: autouse fixtures
registering all 8 bundled web providers, reset after each test
- test_website_policy: same provider registration pattern
- test_web_tools_tavily: same pattern across 3 dispatch test classes
- Also add is_safe_url/check_website_access mocks where SSRF check
blocks example.com (DNS resolution fails in isolated envs)
Stale check_fn cache:
- test_kanban_tools: invalidate_check_fn_cache() + _clear_tool_defs_cache()
in both kanban guidance tests (prior test cached False for kanban_show)
- test_discord_tool: cache invalidation in setup/teardown
- test_homeassistant_tool: invalidate_check_fn_cache() before registry queries
Module-level state pollution:
- test_auxiliary_client: autouse fixture clearing _aux_unhealthy_until cache
- test_skill_commands: set_session_vars() instead of patch.dict(os.environ)
(ContextVar takes precedence over os.environ)
- test_dm_topics: overwrite sys.modules + separate telegram.constants mock
+ force-reimport of gateway.platforms.telegram
- test_terminal_tool_requirements: removed duplicate class declaration,
autouse _clear_caches fixture
* change(tests): run_tests.sh explicitly includes env vars
instead of manually dropping some vars, now we just only include some
* fix(tests): 5 more isolation/NixOS fixes
- test_approval_plugin_hooks: isolate HERMES_HOME so real user's
command_allowlist doesn't short-circuit the approval path
- test_google_chat: skipif when Platform.GOOGLE_CHAT not in enum
(feature not merged on this branch)
- test_write_deny: test systemd prefix against tmp_path instead of
/etc/systemd which resolves to /nix/store on NixOS
- test_pty_bridge: use shutil.which('cat') instead of /bin/cat
(doesn't exist on NixOS)
- profiles.py: rmtree onexc handler chmod's parent dirs too, fixing
profile deletion when copytree preserved read-only modes from
nix store
* fix(tests): clear unhealthy cache in autouse fixture for auxiliary_client
* fix(tests): skip send_message when telegram not installed; handle missing worker_id in browser_supervisor
* fix: py3.11 rmtree onexc compat + belt-and-suspenders unhealthy cache clear for expired codex test
* fix: address PR #29016 review feedback
- Remove tracked .pytest-cache/ artifact and add to .gitignore
- Fix stale 'xdist worker' comment in conftest.py
- Deduplicate web provider registration into tests/tools/conftest.py
shared helper (register_all_web_providers), replacing 8 copy-pasted
blocks across 6 test files
- Update PR description: remove stale recovered-test-files claim,
fix worker count to match code (cpu_count*2)
* fix: eliminate race in stale-cache achievements test
The background scan thread could complete and overwrite _SNAPSHOT_CACHE
before evaluate_all() returned the stale data — only 10 fake sessions
made the scan finish instantly. Added scan_delay param to _FakeSessionDB
and set it to 2s in the stale-cache test so the background thread can't
win the race.
|
||
|
Teknium
|
e2fd462ebe
|
ci(tests): add pytest-timeout 60s hard cap to break suite-teardown deadlock (#28861)
* ci(tests): add pytest-timeout 60s hard cap to break suite-teardown deadlock The full pytest suite reliably hangs at ~96% on origin/main, blowing through the 20-minute GHA job timeout on every CI push since yesterday. Individual tests complete in <30s — the deadlock builds up at session teardown after all tests run, when leaked threads and atexit handlers from thousands of tests interact and one of them lands in a futex-wait that never resolves. This PR is a stopgap that unblocks CI immediately + speeds up several slow tests we found while diagnosing. Changes - pyproject.toml: add pytest-timeout==2.4.0 to dev deps; bake --timeout=60 --timeout-method=thread into the default addopts. - scripts/run_tests.sh: re-add --timeout flags directly because the script wipes pyproject addopts with -o 'addopts='. - .github/workflows/tests.yml: explicit --timeout/--timeout-method on the CI pytest invocation for clarity. - gateway/run.py: in _run_agent, if the stream consumer was never created (e.g. non-streaming agent or test stub), cancel the stream_task immediately instead of waiting out the 5s wait_for timeout. ~5s saved per non-streaming gateway test run. - tests/run_agent/conftest.py: extend _fast_retry_backoff to patch agent.conversation_loop.jittered_backoff alongside run_agent.jittered_backoff. The retry loop was extracted into agent.conversation_loop which holds its own import — patching the run_agent reference alone left tests burning real wall-clock backoff seconds. - tests/run_agent/test_anthropic_error_handling.py tests/run_agent/test_run_agent.py (TestRetryExhaustion) tests/run_agent/test_fallback_model.py: same conversation_loop fix for per-test fixtures (defensive — the conftest covers them too). - tests/gateway/test_gateway_inactivity_timeout.py: trim run_duration 10.0 → 2.0 / 5.0 → 2.0 on three tests that wait the full SlowFakeAgent duration. Adjusted thresholds proportionally. - tests/gateway/test_api_server_runs.py: test_stop_interrupt_exception_does_not_crash trips the interrupted event in addition to raising, so the slow_run thread unblocks at teardown instead of waiting 10s. - tests/hermes_cli/test_update_gateway_restart.py: also patch time.monotonic in the autouse fixture. _wait_for_service_active loops on a wall-clock deadline; with sleep no-op'd the loop spun on real monotonic until 10s real-time per restart attempt (20s+ per test). - tests/tools/test_zombie_process_cleanup.py: cut runner._restart_drain_timeout 5.0 → 0.1 in test_gateway_stop_calls_close. Suite still hangs at 96% on full no-timeout runs; with these changes CI runs through to a real pass/fail signal. * chore(lock): regenerate uv.lock after adding pytest-timeout * ci: drop pytest-timeout 60 → 30s + bump GHA job 20 → 30 min Prior commit's timeout=60 was too generous — CI test job still hit the 20-min wall-clock cap with the suite hung at 96% (orphan agent-browser subprocesses blocking pytest session teardown). The local timeout=20 run completed in 6:17, so 30s is conservative enough to let real tests finish but aggressive enough to short-circuit deadlocks. Also bump GHA job timeout to 30 min as a safety margin. * test: delete 11 pre-existing failing tests + revert monotonic patch The previous PR commit landed pytest-timeout=30s and the suite now completes in 18:14 instead of hanging at 96%, but 11 pre-existing tests fail with real assertions. Per Teknium: nuke them. Deleted (no replacements): - tests/gateway/test_restart_resume_pending.py::test_clean_drain_does_not_mark_resume_pending - tests/gateway/test_restart_resume_pending.py::test_drain_timeout_only_marks_still_running_sessions - tests/hermes_cli/test_gateway_service.py::TestGatewaySystemServiceRouting::test_gateway_install_passes_system_flags - tests/hermes_cli/test_gateway_wsl.py::TestGatewayCommandWSLMessages::test_install_wsl_with_systemd_warns - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_detects_launchd_and_skips_manual_restart_message - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_restarts_profile_manual_gateways - tests/tools/test_file_operations.py::TestGitBaselineCheck::* (6 tests, entire class — _check_git_baseline helper doesn't exist) Also reverted my time.monotonic autouse-fixture hack in test_update_gateway_restart.py — it was causing worker crashes in CI by poisoning later tests in the same xdist worker. The two slow tests in that file (~24s and ~20s) will go back to taking real time but should still finish under the 30s pytest-timeout. * test: delete more pre-existing CI failures After previous push 3 more tests failed on CI; cull them all. Removed: - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_without_launchd_shows_manual_restart - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_profile_manual_gateway_falls_back_to_sigterm - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateResetFailedBeforeRestart::test_reset_failed_also_runs_before_retry_restart - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateResetFailedBeforeRestart::test_final_failure_message_tells_user_to_reset_failed - tests/run_agent/test_tool_call_args_sanitizer.py::test_marker_message_inserted_when_missing The 4 update_gateway_restart tests trigger `_wait_for_service_active` polling on a real wall-clock deadline that occasionally exceeds the 30s pytest-timeout cap and crashes xdist workers. The marker test has a pre-existing assertion mismatch. * test: nuke entire TestCmdUpdateLaunchdRestart class After surgical deletes of 4 tests this class keeps producing new worker-crashing tests. The pattern is consistent: any test in this class that triggers cmd_update's _wait_for_service_active polling spins on real wall-clock time and trips pytest-timeout's thread method, crashing the xdist worker. Just delete the whole class (285 lines, ~10 tests). These exercise macOS-only launchd behavior that's better tested on a real macOS runner than in linux xdist. * test: stub the 2 fallback_model tests that crash xdist workers on CI * test: delete test_anthropic_error_handling.py + test_fallback_model.py entirely These two files exercise the agent retry/fallback code paths and consistently crash xdist workers under pytest-timeout's thread method. Whack-a-mole-stubbing individual tests just surfaces the next ones. Nuke both files. * test: delete tests/hermes_cli/test_update_gateway_restart.py entirely This file's cmd_update integration tests consistently crash xdist workers under pytest-timeout's thread method. Surgical deletes just surface the next set. Removing the whole file. * ci(tests): switch pytest-timeout method thread → signal Thread-method has been crashing xdist workers when it interrupts code that's not interruption-safe (retry loops, threading.Event waits, etc). Signal method uses SIGALRM which is interpreter-level and cleanly raises a Failed: Timeout exception in test code. Should stop the worker crash cascade — failures will surface as proper Timeout markers we can diagnose individually. |
||
|
Teknium
|
771b8c4a36
|
test(conftest): plug every gateway-kill leak path (#23486)
The existing _live_system_guard (PR #23397) blocked os.kill / os.killpg and a narrow subset of subprocess invocations. Tests still SIGTERMed the live gateway today (May 10) because the guard had structural holes. Plug them all: - subprocess: also wrap getoutput, getstatusoutput - os.system, os.popen - completely unwrapped before - pty.spawn - completely unwrapped before - asyncio.create_subprocess_exec / create_subprocess_shell - bypassed the subprocess module entirely; now wrapped - Subprocess command inspection now looks at the WHOLE command string, not just tokens[0]. Catches sudo systemctl, env systemctl, bash -c 'systemctl', setsid systemctl, /usr/bin/systemctl, etc. - New process-killer block: pkill / killall / taskkill / fuser targeting hermes/python patterns is now refused - os.kill PID 0 (own group) allowed; PID -1 (every process we can signal) refused - subprocess.Popen wrapper preserves __class_getitem__ so third-party packages that use Popen[bytes] as a type annotation still import Coverage is locked in by tests/test_live_system_guard_self_test.py - exercises every primitive against a guaranteed-foreign PID and asserts the guard fires. Adding a new kill primitive without updating the guard breaks CI. scripts/run_tests.sh now also force-loads ~/.hermes/pytest_live_guard.py when present (developer-machine convenience), so even worktrees that predate this commit get the protection on subsequent test runs through the canonical wrapper. |
||
|
Teknium
|
f00dc6d7a3
|
fix(tests): harden run_tests.sh — uv-aware bootstrap + scrub HERMES_CRON_SESSION (#22767)
Two unrelated but co-located fixes to scripts/run_tests.sh: 1. pytest-split bootstrap (#22401): the script tried '$PYTHON -m pip install pytest-split' on first run, but uv-created venvs ship without pip. Result: 'No module named pip' before any test ran. Add a uv fallback (uv pip install --python $PYTHON), keep pip as a secondary path, and emit a clear error pointing at 'uv pip install -e ".[dev]"' when neither is available. Also declare pytest-split in pyproject.toml dev extra so a normal '.[dev]' install provisions it. 2. HERMES_CRON_SESSION leak (#22400): the hermetic env scrub already unsets HERMES_GATEWAY_SESSION and HERMES_INTERACTIVE but missed the sibling HERMES_CRON_SESSION. When run_tests.sh is invoked from a Hermes cron job, that variable leaks into pytest, flipping tools/approval.py into cron-deny mode and breaking tests/acp/test_approval_isolation.py and friends. Closes #22400. Closes #22401. |
||
|
Teknium
|
d404849351
|
test: make test env hermetic; enforce CI parity via scripts/run_tests.sh (#11577)
* test: make test env hermetic; enforce CI parity via scripts/run_tests.sh
Fixes the recurring 'works locally, fails in CI' (and vice versa) class
of flakes by making tests hermetic and providing a canonical local runner
that matches CI's environment.
## Layer 1 — hermetic conftest.py (tests/conftest.py)
Autouse fixture now unsets every credential-shaped env var before every
test, so developer-local API keys can't leak into tests that assert
'auto-detect provider when key present'.
Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD,
_CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of
credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID,
FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that
change auto-detect behavior.
Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET,
HERMES_SESSION_*, etc.) that mutate agent behavior.
Also:
- Redirects HOME to a per-test tempdir (not just HERMES_HOME), so
code reading ~/.hermes/* directly can't touch the real dir.
- Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to
match CI's deterministic runtime.
The old _isolate_hermes_home fixture name is preserved as an alias so
any test that yields it explicitly still works.
## Layer 2 — scripts/run_tests.sh canonical runner
'Always use scripts/run_tests.sh, never call pytest directly' is the
new rule (documented in AGENTS.md). The script:
- Unsets all credential env vars (belt-and-suspenders for callers
who bypass conftest — e.g. IDE integrations)
- Pins TZ/LANG/PYTHONHASHSEED
- Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on
a 20-core workstation surfaces test-ordering flakes CI will never
see, causing the infamous 'passes in CI, fails locally' drift)
- Finds the venv in .venv, venv, or main checkout's venv
- Passes through arbitrary pytest args
Installs pytest-split on demand so the script can also be used to run
matrix-split subsets locally for debugging.
## Remove 3 module-level dotenv stubs that broke test isolation
tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a
module-level:
if 'dotenv' not in sys.modules:
fake_dotenv = types.ModuleType('dotenv')
fake_dotenv.load_dotenv = lambda *a, **kw: None
sys.modules['dotenv'] = fake_dotenv
This patches sys.modules['dotenv'] to a fake at import time with no
teardown. Under pytest-xdist LoadScheduling, whichever worker collected
one of these files first poisoned its sys.modules; subsequent tests in
the same worker that imported load_dotenv transitively (e.g.
test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and
saw their assertions fail.
dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml),
so the defensive stub was never needed. Removed.
## Validation
- tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4
failures in test_env_loader.py before this fix)
- tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py,
tests/test_hermes_logging.py combined: 123 passed (the caplog
regression tests from PR #11453 still pass)
- Local full run shows no F/E clusters in the 0-55% range that were
previously present before the conftest hardening
## Background
See AGENTS.md 'Testing' section for the full list of drift sources
this closes. Matrix split (closed as #11566) will be re-attempted
once this foundation lands — cross-test pollution was the root cause
of the shard-3 hang in that PR.
* fix(conftest): don't redirect HOME — it broke CI subprocesses
PR #11577's autouse fixture was setting HOME to a per-test tempdir.
CI started timing out at 97% complete with dozens of E/F markers and
orphan python processes at cleanup — tests (or transitive deps)
spawn subprocesses that expect a stable HOME, and the redirect broke
them in non-obvious ways.
Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift
fixes) are unchanged and still in place. HERMES_HOME redirection is
also unchanged — that's the canonical way to isolate tests from
~/.hermes/, not HOME.
Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"`
instead of `get_hermes_home()` is a bug to fix at the callsite, not
something to paper over in conftest.
|