mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-06 02:41:48 +00:00
d404849351
* test: make test env hermetic; enforce CI parity via scripts/run_tests.sh
Fixes the recurring 'works locally, fails in CI' (and vice versa) class
of flakes by making tests hermetic and providing a canonical local runner
that matches CI's environment.
## Layer 1 — hermetic conftest.py (tests/conftest.py)
Autouse fixture now unsets every credential-shaped env var before every
test, so developer-local API keys can't leak into tests that assert
'auto-detect provider when key present'.
Pattern: unset any var ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD,
_CREDENTIALS, _ACCESS_KEY, _PRIVATE_KEY, etc. Plus an explicit list of
credential names that don't fit the suffix pattern (AWS_ACCESS_KEY_ID,
FAL_KEY, GH_TOKEN, etc.) and all the provider BASE_URL overrides that
change auto-detect behavior.
Also unsets HERMES_* behavioral vars (HERMES_YOLO_MODE, HERMES_QUIET,
HERMES_SESSION_*, etc.) that mutate agent behavior.
Also:
- Redirects HOME to a per-test tempdir (not just HERMES_HOME), so
code reading ~/.hermes/* directly can't touch the real dir.
- Pins TZ=UTC, LANG=C.UTF-8, LC_ALL=C.UTF-8, PYTHONHASHSEED=0 to
match CI's deterministic runtime.
The old _isolate_hermes_home fixture name is preserved as an alias so
any test that yields it explicitly still works.
## Layer 2 — scripts/run_tests.sh canonical runner
'Always use scripts/run_tests.sh, never call pytest directly' is the
new rule (documented in AGENTS.md). The script:
- Unsets all credential env vars (belt-and-suspenders for callers
who bypass conftest — e.g. IDE integrations)
- Pins TZ/LANG/PYTHONHASHSEED
- Uses -n 4 xdist workers (matches GHA ubuntu-latest; -n auto on
a 20-core workstation surfaces test-ordering flakes CI will never
see, causing the infamous 'passes in CI, fails locally' drift)
- Finds the venv in .venv, venv, or main checkout's venv
- Passes through arbitrary pytest args
Installs pytest-split on demand so the script can also be used to run
matrix-split subsets locally for debugging.
## Remove 3 module-level dotenv stubs that broke test isolation
tests/hermes_cli/test_{arcee,xiaomi,api_key}_provider.py each had a
module-level:
if 'dotenv' not in sys.modules:
fake_dotenv = types.ModuleType('dotenv')
fake_dotenv.load_dotenv = lambda *a, **kw: None
sys.modules['dotenv'] = fake_dotenv
This patches sys.modules['dotenv'] to a fake at import time with no
teardown. Under pytest-xdist LoadScheduling, whichever worker collected
one of these files first poisoned its sys.modules; subsequent tests in
the same worker that imported load_dotenv transitively (e.g.
test_env_loader.py via hermes_cli.env_loader) got the no-op lambda and
saw their assertions fail.
dotenv is a required dependency (python-dotenv>=1.2.1 in pyproject.toml),
so the defensive stub was never needed. Removed.
## Validation
- tests/hermes_cli/ alone: 2178 passed, 1 skipped, 0 failed (was 4
failures in test_env_loader.py before this fix)
- tests/test_plugin_skills.py, tests/hermes_cli/test_plugins.py,
tests/test_hermes_logging.py combined: 123 passed (the caplog
regression tests from PR #11453 still pass)
- Local full run shows no F/E clusters in the 0-55% range that were
previously present before the conftest hardening
## Background
See AGENTS.md 'Testing' section for the full list of drift sources
this closes. Matrix split (closed as #11566) will be re-attempted
once this foundation lands — cross-test pollution was the root cause
of the shard-3 hang in that PR.
* fix(conftest): don't redirect HOME — it broke CI subprocesses
PR #11577's autouse fixture was setting HOME to a per-test tempdir.
CI started timing out at 97% complete with dozens of E/F markers and
orphan python processes at cleanup — tests (or transitive deps)
spawn subprocesses that expect a stable HOME, and the redirect broke
them in non-obvious ways.
Env-var unsetting and TZ/LANG/hashseed pinning (the actual CI-drift
fixes) are unchanged and still in place. HERMES_HOME redirection is
also unchanged — that's the canonical way to isolate tests from
~/.hermes/, not HOME.
Any code in the codebase reading ~/.hermes/* via `Path.home() / ".hermes"`
instead of `get_hermes_home()` is a bug to fix at the callsite, not
something to paper over in conftest.
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_anthropic_oauth_flow.py | ||
| test_anthropic_provider_persistence.py | ||
| test_api_key_providers.py | ||
| test_arcee_provider.py | ||
| test_argparse_flag_propagation.py | ||
| test_atomic_json_write.py | ||
| test_atomic_yaml_write.py | ||
| test_auth_codex_provider.py | ||
| test_auth_commands.py | ||
| test_auth_nous_provider.py | ||
| test_auth_provider_gate.py | ||
| test_auth_qwen_provider.py | ||
| test_backup.py | ||
| test_banner.py | ||
| test_banner_git_state.py | ||
| test_banner_skills.py | ||
| test_chat_skills_flag.py | ||
| test_claw.py | ||
| test_clear_stale_base_url.py | ||
| test_cmd_update.py | ||
| test_coalesce_session_args.py | ||
| test_codex_cli_model_picker.py | ||
| test_codex_models.py | ||
| test_commands.py | ||
| test_completion.py | ||
| test_config.py | ||
| test_config_env_expansion.py | ||
| test_config_validation.py | ||
| test_container_aware_cli.py | ||
| test_copilot_auth.py | ||
| test_cron.py | ||
| test_custom_provider_model_switch.py | ||
| test_debug.py | ||
| test_deprecated_cwd_warning.py | ||
| test_dingtalk_auth.py | ||
| test_doctor.py | ||
| test_doctor_command_install.py | ||
| test_env_loader.py | ||
| test_env_sanitize_on_load.py | ||
| test_gateway.py | ||
| test_gateway_linger.py | ||
| test_gateway_runtime_health.py | ||
| test_gateway_service.py | ||
| test_gateway_wsl.py | ||
| test_gemini_provider.py | ||
| test_launcher.py | ||
| test_logs.py | ||
| test_managed_installs.py | ||
| test_mcp_config.py | ||
| test_mcp_tools_config.py | ||
| test_memory_reset.py | ||
| test_model_normalize.py | ||
| test_model_provider_persistence.py | ||
| test_model_switch_copilot_api_mode.py | ||
| test_model_switch_custom_providers.py | ||
| test_model_switch_opencode_anthropic.py | ||
| test_model_switch_variant_tags.py | ||
| test_model_validation.py | ||
| test_models.py | ||
| test_non_ascii_credential.py | ||
| test_nous_hermes_non_agentic.py | ||
| test_nous_subscription.py | ||
| test_ollama_cloud_auth.py | ||
| test_ollama_cloud_provider.py | ||
| test_opencode_go_in_model_list.py | ||
| test_overlay_slug_resolution.py | ||
| test_path_completion.py | ||
| test_placeholder_usage.py | ||
| test_plugin_cli_registration.py | ||
| test_plugins.py | ||
| test_plugins_cmd.py | ||
| test_profile_export_credentials.py | ||
| test_profiles.py | ||
| test_reasoning_effort_menu.py | ||
| test_runtime_provider_resolution.py | ||
| test_session_browse.py | ||
| test_sessions_delete.py | ||
| test_set_config_value.py | ||
| test_setup.py | ||
| test_setup_hermes_script.py | ||
| test_setup_matrix_e2ee.py | ||
| test_setup_model_provider.py | ||
| test_setup_noninteractive.py | ||
| test_setup_openclaw_migration.py | ||
| test_setup_prompt_menus.py | ||
| test_skills_config.py | ||
| test_skills_hub.py | ||
| test_skills_install_flags.py | ||
| test_skills_skip_confirm.py | ||
| test_skills_subparser.py | ||
| test_skin_engine.py | ||
| test_status.py | ||
| test_status_model_provider.py | ||
| test_subparser_routing_fallback.py | ||
| test_subprocess_timeouts.py | ||
| test_terminal_menu_fallbacks.py | ||
| test_tips.py | ||
| test_tool_token_estimation.py | ||
| test_tools_config.py | ||
| test_tools_disable_enable.py | ||
| test_update_autostash.py | ||
| test_update_check.py | ||
| test_update_gateway_restart.py | ||
| test_user_providers_model_switch.py | ||
| test_web_server.py | ||
| test_webhook_cli.py | ||
| test_xiaomi_provider.py | ||