hermes-agent/tests/conftest.py
Teknium 597615ade4
fix(ci): make tests, workflows, and attribution reliable under load (#66373)
* feat(attribution): conflict-free contributor mappings via contributors/emails/ directory

The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet:
every concurrent salvage PR appended entries to the same lines of the
same file, so parallel PRs re-conflicted on every merge to main.

New system: one file per email under contributors/emails/ — filename is
the commit-author email, first non-comment line is the GitHub login.
File additions never conflict, so any number of PRs can add mappings
concurrently.

- scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen)
  merged with the directory at import time (directory wins). All
  existing consumers (resolve_author, contributor_audit.py) unchanged.
- scripts/add_contributor.py: idempotent CLI to add a mapping; refuses
  conflicting reassignments (incl. against the legacy map), validates
  email/login shapes.
- contributor-check.yml: attribution gate now accepts a mapping file OR
  a legacy entry; failure message prints the exact add_contributor
  command. Also auto-resolves bare <login>@users.noreply.github.com
  emails is intentionally NOT added (kept id+login form only, matching
  previous behavior).
- contributor_audit.py: guidance now points at add_contributor.py.
- tests/scripts/test_contributor_map.py: 12 tests covering loader,
  merge precedence, CLI idempotency/conflict/validation, subprocess E2E.

* feat(ci): one-shot per-file flake retry in the parallel test runner

A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry
counts as green but is loudly reported in a '⚠ FLAKY' summary section
(with both attempts' output preserved) so the flake gets fixed instead
of eating a full-run rerun. Deterministic failures fail both attempts —
regressions cannot be laundered green.

- --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables)
- E2E verified: simulated first-run-fail flake goes green with banner;
  deterministic failure still exits 1; retries=0 restores old behavior.

This converts the dominant CI failure mode (one timing-sensitive test
flaking a 4600-test shard, requiring a manual 10-minute rerun and an
agent triage loop) into a self-healing retry that costs one file's
runtime.

* test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s

These guard against catastrophic regex backtracking (seconds-to-minutes
class), but 0.15s is within scheduler-stall noise on loaded shared CI
runners — test_max_accepted_separator_free_input_is_fast failed a CI
shard this week on runner load alone. 2.0s still catches the regression
class with zero flake surface.

* fix(ci): job timeouts everywhere + retries on all network installs

Reliability pass over every workflow:
- timeout-minutes on all 21 jobs that lacked one (a hung job previously
  burned the 6-hour default runner budget)
- ./.github/actions/retry wrapped around every network-fetching install
  that lacked it: pip installs (deploy-site, skills-index), npm ci
  (deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker
  test deps). Deterministic build steps (npm run build) deliberately
  NOT retried — split into separate steps so a real build failure fails
  fast instead of retrying 3x.

* docs(agents): document the file-retry flake policy

* fix(ci): curl retries on deploy hook + skills-index probe

* fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile

From the workflow reliability audit:
- tests.yml: duration-cache restore had NO restore-keys while saves use
  run_id-suffixed keys — the cache never matched once, so LPT slicing
  always ran blind and unbalanced slices pushed heavy files toward the
  per-file timeout. One-line restore-keys fixes slice balancing.
- Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed):
  'gh pr view || true' turned an API blip into 'label absent' → false
  BLOCKING failure. Now 3x retry, and API failure is reported as an API
  failure instead of a missing label.
- detect-changes action: compare API retried before failing open (was
  silently running all lanes on any blip).
- uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried
  so registry blips don't read as 'lockfile stale'.
- docker.yml merge job: imagetools create retried (Docker Hub eventual
  consistency on just-pushed digests).
- Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to
  curl --retry 3 (ADD cannot retry; checksums still enforced); npm
  --fetch-retries=5; playwright chromium fetch retried 3x.
- Advisory artifact uploads (per-slice durations, ci-timings report)
  get continue-on-error so an artifact-service blip can't fail a green
  test slice.

* fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list

- test_tui_gateway_server.py: session.create / non-eager session.resume
  arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
  test and fires into the NEXT test's _make_agent mock, racily
  corrupting captured state (the recurring session_resume shard
  failures). Replaced the per-test whack-a-mole stub with a module-wide
  autouse fixture; the 3 worker-lifecycle tests that genuinely need the
  deferred build opt back in via @pytest.mark.real_agent_prewarm (new
  marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
  live PROVIDER_REGISTRY instead of a hand-list that had drifted
  (missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
  tests failed on any machine with HF_TOKEN exported. E2E-verified with
  HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.

* test: de-flake 30 timing-sensitive test files for loaded CI runners

Root-cause fixes from the flake audit (session-DB mining + repo sweep):

Event-based sync instead of sleep-sync:
- title_generator: mock sets threading.Event, wait(10) replaces
  sleep(0.3) hoping the daemon thread got scheduled
- docker zombie_reaping / profile_gateway: poll-for-state helpers
  replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async)
- process_registry tree test: select()-bounded readline replaces an
  unbounded blocking read (parent wedge now fails THIS test with a clear
  message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s
  (the 1s partition window mid-interpreter-startup is how a child PID
  escaped the live-system guard in CI)

Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors;
all of these complete in ms-to-1s when healthy so the raises cost
nothing on green runs):
- subprocess/thread waits <= 2s raised to 10-15s across mcp_tool,
  mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe,
  mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt,
  voice_cli_integration, docker_environment, session_store_lock_io,
  planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output
  (joins now also assert not is_alive() so stragglers fail loudly)
- wall-clock discrimination ceilings loosened where the guarded hang is
  10x larger: local_background_child_hang 4s->10s, interrupt_cleanup
  setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup
  5s->15s, protocol/gil-starvation fast-handler 0.5s->2s,
  iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s
- narrow assertion windows widened: honcho first-turn wait 0.4..0.65 ->
  0.25..2.0 (property is bounded-not-hung, not an exact wall-clock);
  compression fork-lock TTL 1s->3s (12 refresh chances per lease);
  compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0)
- telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)

* fix(tests): repair indentation from de-flake batch edit

* fix(tests): harden env isolation and replace remaining sleep-sync races

The full 42k-test run and complete npm check surfaced three more classes:

- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
  leaked into Python/TUI tests. Pin the default Honcho host in the
  hermetic fixture, isolate the one fallback test from ~/.honcho, and
  blank SSH_* around terminalSetup tests. This flipped 20 false failures
  back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
  time.sleep globally, then busy-polled with that same mocked sleep. Under
  full-suite load the poller could starve the writer. Each test now waits
  on an Event emitted by the exact flush/retry transition; 30/30 passed
  under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
  not fire before its assertion. A loaded runner descheduled the test for
  >500ms and both chunks arrived. Producer controls now gate second-chunk
  and completion transitions explicitly.

Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.

* refactor(ci): use gh bot pat, better retries

refactor(ci): use retry action for PR label fetch
the retry action now captures stdout as a step output, so it can serve
double duty: retry + output capture for commands like 'gh pr view' whose
result must be consumed by later steps.

Retry action gains:
- 'stdout' output (heredoc-delimited to preserve newlines)
- tee to temp file so stdout still streams to the job log
- step id 'retry' for output reference

Both lint.yml and supply-chain-audit.yml now use the retry action
directly with 'command: gh pr view ...' and read
steps.<id>.outputs.stdout.

ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth

Replace secrets.GITHUB_TOKEN and github.token with
secrets.AUTOFIX_BOT_PAT across all workflows and composite actions
that use the gh CLI or GitHub API. The PAT has consistent permissions
across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate
limit sharing with the default token, and is already used by
js-autofix.yml for the same reasons.

19 sites swapped across 9 files:
- lint.yml (3): label fetch, comment post/edit, comment update
- supply-chain-audit.yml (5): scan, critical comment, unbounded dep
  comment, label fetch, mcp-catalog comment
- lockfile-diff.yml (1): PR comment post/update
- skills-index-freshness.yml (1): issue creation on degraded probe
- skills-index.yml (2): index build, trigger deploy workflow
- upload_to_pypi.yml (2): release view poll, release upload
- ci.yml (1): timings report
- deploy-site.yml (2): skills index crawl
- detect-changes/action.yml (1): compare API call

---------

Co-authored-by: ethernet <arilotter@gmail.com>
2026-07-17 20:55:24 +00:00

907 lines
35 KiB
Python

"""Shared fixtures for the hermes-agent test suite.
Hermetic-test invariants enforced here (see AGENTS.md for rationale):
1. **No credential env vars.** All provider/credential-shaped env vars
(ending in _API_KEY, _TOKEN, _SECRET, _PASSWORD, _CREDENTIALS, etc.)
are unset before every test. Local developer keys cannot leak in.
2. **Isolated HERMES_HOME.** HERMES_HOME points to a per-test tempdir so
code reading ``~/.hermes/*`` via ``get_hermes_home()`` can't see the
real one. (We do NOT also redirect HOME — that broke subprocesses in
CI. Code using ``Path.home() / ".hermes"`` instead of the canonical
``get_hermes_home()`` is a bug to fix at the callsite.)
3. **Deterministic runtime.** TZ=UTC, LANG=C.UTF-8, PYTHONHASHSEED=0.
4. **No HERMES_SESSION_* inheritance** — the agent's current gateway
session must not leak into tests.
These invariants make the local test run match CI closely. Gaps that
remain (CPU count, xdist worker count) are addressed by the canonical
test runner at ``scripts/run_tests.sh``.
"""
import asyncio
import os
import sys
from pathlib import Path
import pytest
# Ensure project root is importable
PROJECT_ROOT = Path(__file__).parent.parent
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
# ── Per-file process isolation ──────────────────────────────────────────────
# Tests run via ``scripts/run_tests_parallel.py``, which spawns a fresh
# ``python -m pytest <file>`` subprocess per test file. Cross-file state
# leakage (module-level dicts, ContextVars, caches) is impossible: each
# file gets a clean Python interpreter. Intra-file ordering is the test
# author's responsibility — if test A in foo.py mutates state that test B
# in foo.py reads, that's a real bug to fix in the file (it would also
# bite anyone running ``pytest tests/foo.py`` directly).
#
# This replaces the historic _reset_module_state autouse fixture (manual
# state clearing) and the brief experiment with subprocess-per-test
# isolation (too slow at ~17k tests).
#
# See ``scripts/run_tests_parallel.py`` for the runner.
# ── Credential env-var filter ──────────────────────────────────────────────
#
# Any env var in the current process matching ONE of these patterns is
# unset for every test. Developers' local keys cannot leak into assertions
# about "auto-detect provider when key present".
_CREDENTIAL_SUFFIXES = (
"_API_KEY",
"_TOKEN",
"_SECRET",
"_PASSWORD",
"_CREDENTIALS",
"_ACCESS_KEY",
"_SECRET_ACCESS_KEY",
"_PRIVATE_KEY",
"_OAUTH_TOKEN",
"_WEBHOOK_SECRET",
"_ENCRYPT_KEY",
"_APP_SECRET",
"_CLIENT_SECRET",
"_CORP_SECRET",
"_AES_KEY",
)
# Explicit names (for ones that don't fit the suffix pattern)
_CREDENTIAL_NAMES = frozenset({
"AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY",
"AWS_SESSION_TOKEN",
"ANTHROPIC_TOKEN",
"FAL_KEY",
"GH_TOKEN",
"GITHUB_TOKEN",
"OPENAI_API_KEY",
"OPENROUTER_API_KEY",
"NOUS_API_KEY",
"GEMINI_API_KEY",
"GOOGLE_API_KEY",
"GROQ_API_KEY",
"XAI_API_KEY",
"MISTRAL_API_KEY",
"DEEPSEEK_API_KEY",
"KIMI_API_KEY",
"MOONSHOT_API_KEY",
"GLM_API_KEY",
"ZAI_API_KEY",
"MINIMAX_API_KEY",
"OLLAMA_API_KEY",
"OPENVIKING_API_KEY",
"COPILOT_API_KEY",
"CLAUDE_CODE_OAUTH_TOKEN",
"BROWSERBASE_API_KEY",
"FIRECRAWL_API_KEY",
"PARALLEL_API_KEY",
"EXA_API_KEY",
"TAVILY_API_KEY",
"WANDB_API_KEY",
"ELEVENLABS_API_KEY",
"HONCHO_API_KEY",
"MEM0_API_KEY",
"SUPERMEMORY_API_KEY",
"RETAINDB_API_KEY",
"HINDSIGHT_API_KEY",
"HINDSIGHT_LLM_API_KEY",
"DAYTONA_API_KEY",
"TWILIO_AUTH_TOKEN",
"TELEGRAM_BOT_TOKEN",
"DISCORD_BOT_TOKEN",
"SLACK_BOT_TOKEN",
"SLACK_APP_TOKEN",
"MATTERMOST_TOKEN",
"MATRIX_ACCESS_TOKEN",
"MATRIX_PASSWORD",
"MATRIX_RECOVERY_KEY",
"HASS_TOKEN",
"EMAIL_PASSWORD",
"BLUEBUBBLES_PASSWORD",
"FEISHU_APP_SECRET",
"FEISHU_ENCRYPT_KEY",
"FEISHU_VERIFICATION_TOKEN",
"DINGTALK_CLIENT_SECRET",
"QQ_CLIENT_SECRET",
"QQ_STT_API_KEY",
"WECOM_SECRET",
"WECOM_CALLBACK_CORP_SECRET",
"WECOM_CALLBACK_TOKEN",
"WECOM_CALLBACK_ENCODING_AES_KEY",
"WEIXIN_TOKEN",
"MODAL_TOKEN_ID",
"MODAL_TOKEN_SECRET",
"TERMINAL_SSH_KEY",
"SUDO_PASSWORD",
"GATEWAY_PROXY_KEY",
"API_SERVER_KEY",
"TOOL_GATEWAY_USER_TOKEN",
"TELEGRAM_WEBHOOK_SECRET",
"WEBHOOK_SECRET",
"VOICE_TOOLS_OPENAI_KEY",
"BROWSER_USE_API_KEY",
"CUSTOM_API_KEY",
"GATEWAY_PROXY_URL",
"GEMINI_BASE_URL",
"OPENAI_BASE_URL",
"OPENROUTER_BASE_URL",
"OLLAMA_BASE_URL",
"GROQ_BASE_URL",
"XAI_BASE_URL",
"ANTHROPIC_BASE_URL",
})
def _looks_like_credential(name: str) -> bool:
"""True if env var name matches a credential-shaped pattern."""
if name in _CREDENTIAL_NAMES:
return True
return any(name.endswith(suf) for suf in _CREDENTIAL_SUFFIXES)
# HERMES_* vars that change test behavior by being set. Unset all of these
# unconditionally — individual tests that need them set do so explicitly.
_HERMES_BEHAVIORAL_VARS = frozenset({
"HERMES_YOLO_MODE",
"HERMES_INTERACTIVE",
"HERMES_QUIET",
"HERMES_TOOL_PROGRESS",
"HERMES_TOOL_PROGRESS_MODE",
"HERMES_MAX_ITERATIONS",
"HERMES_SESSION_PLATFORM",
"HERMES_SESSION_CHAT_ID",
"HERMES_SESSION_CHAT_NAME",
"HERMES_SESSION_THREAD_ID",
"HERMES_SESSION_SOURCE",
"HERMES_SESSION_KEY",
"HERMES_GATEWAY_SESSION",
"HERMES_CRON_SESSION",
"_HERMES_GATEWAY",
"HERMES_PLATFORM",
"HERMES_MODEL",
"HERMES_INFERENCE_MODEL",
"HERMES_INFERENCE_PROVIDER",
"HERMES_TUI_PROVIDER",
"HERMES_MANAGED",
"HERMES_MANAGED_DIR",
"HERMES_DEV",
"HERMES_CONTAINER",
"HERMES_EPHEMERAL_SYSTEM_PROMPT",
"HERMES_TIMEZONE",
"HERMES_REDACT_SECRETS",
"HERMES_BACKGROUND_NOTIFICATIONS",
"HERMES_EXEC_ASK",
"HERMES_HOME_MODE",
"HERMES_AGENT_USE_LEGACY_SESSION_KEYS",
# Kanban path/board pins must never leak from a developer shell or
# dispatched worker into tests; otherwise tests can write fake tasks to
# the real ~/.hermes/kanban.db instead of the per-test HERMES_HOME.
"HERMES_KANBAN_DB",
"HERMES_KANBAN_BOARD",
"HERMES_KANBAN_HOME",
"HERMES_KANBAN_WORKSPACES_ROOT",
"HERMES_KANBAN_LOGS_ROOT",
"HERMES_KANBAN_TASK",
"HERMES_KANBAN_WORKSPACE",
"HERMES_KANBAN_RUN_ID",
"HERMES_KANBAN_CLAIM_LOCK",
"HERMES_KANBAN_DISPATCH_IN_GATEWAY",
"HERMES_TENANT",
# Honcho host selection changes which nested config block wins. A local
# shell override leaked "myhost" into the full suite and flipped 20
# otherwise-unrelated config tests away from the default "hermes" host.
"HERMES_HONCHO_HOST",
# Dashboard OAuth auth gate (PR #30156). When set, the bundled
# dashboard-auth `nous` plugin auto-registers itself on plugin discovery,
# which is triggered by any `/api/status` call. That leaks a provider
# into the dashboard_auth registry across tests in the same worker and
# makes assertions like `auth_providers == []` flaky. CI never sets
# these, so production tests must not see them either.
"HERMES_DASHBOARD_OAUTH_CLIENT_ID",
"HERMES_DASHBOARD_PORTAL_URL",
"TERMINAL_CWD",
"TERMINAL_ENV",
"TERMINAL_CONTAINER_CPU",
"TERMINAL_CONTAINER_DISK",
"TERMINAL_CONTAINER_MEMORY",
"TERMINAL_CONTAINER_PERSISTENT",
"TERMINAL_DOCKER_PERSIST_ACROSS_PROCESSES",
"TERMINAL_DOCKER_ORPHAN_REAPER",
"TERMINAL_DOCKER_RUN_AS_HOST_USER",
"BROWSER_CDP_URL",
"CAMOFOX_URL",
# Platform allowlists — not credentials, but if set from any source
# (user shell, earlier leaky test, CI env), they change gateway auth
# behavior and flake button-authorization tests.
"TELEGRAM_ALLOWED_USERS",
"DISCORD_ALLOWED_USERS",
"WHATSAPP_ALLOWED_USERS",
"SLACK_ALLOWED_USERS",
"SIGNAL_ALLOWED_USERS",
"SIGNAL_GROUP_ALLOWED_USERS",
"EMAIL_ALLOWED_USERS",
"SMS_ALLOWED_USERS",
"MATTERMOST_ALLOWED_USERS",
"MATRIX_ALLOWED_USERS",
"DINGTALK_ALLOWED_USERS",
"FEISHU_ALLOWED_USERS",
"WECOM_ALLOWED_USERS",
"GATEWAY_ALLOWED_USERS",
"GATEWAY_ALLOW_ALL_USERS",
"TELEGRAM_ALLOW_ALL_USERS",
"DISCORD_ALLOW_ALL_USERS",
"WHATSAPP_ALLOW_ALL_USERS",
"SLACK_ALLOW_ALL_USERS",
"SIGNAL_ALLOW_ALL_USERS",
"EMAIL_ALLOW_ALL_USERS",
"SMS_ALLOW_ALL_USERS",
# Gateway home channels are set by /sethome in real profiles. Tests that
# exercise dashboard notification toggles must opt in explicitly or they
# can accidentally subscribe against a developer's real home channel.
"TELEGRAM_HOME_CHANNEL",
"TELEGRAM_HOME_CHANNEL_THREAD_ID",
"TELEGRAM_HOME_CHANNEL_NAME",
"TELEGRAM_CRON_THREAD_ID",
"DISCORD_HOME_CHANNEL",
"DISCORD_HOME_CHANNEL_THREAD_ID",
"DISCORD_HOME_CHANNEL_NAME",
"SLACK_HOME_CHANNEL",
"SLACK_HOME_CHANNEL_THREAD_ID",
"SLACK_HOME_CHANNEL_NAME",
"WHATSAPP_HOME_CHANNEL",
"WHATSAPP_HOME_CHANNEL_THREAD_ID",
"WHATSAPP_HOME_CHANNEL_NAME",
"SIGNAL_HOME_CHANNEL",
"SIGNAL_HOME_CHANNEL_THREAD_ID",
"SIGNAL_HOME_CHANNEL_NAME",
"EMAIL_HOME_CHANNEL",
"EMAIL_HOME_CHANNEL_THREAD_ID",
"EMAIL_HOME_CHANNEL_NAME",
"SMS_HOME_CHANNEL",
"SMS_HOME_CHANNEL_THREAD_ID",
"SMS_HOME_CHANNEL_NAME",
"MATTERMOST_HOME_CHANNEL",
"MATTERMOST_HOME_CHANNEL_THREAD_ID",
"MATTERMOST_HOME_CHANNEL_NAME",
"MATRIX_HOME_CHANNEL",
"MATRIX_HOME_CHANNEL_THREAD_ID",
"MATRIX_HOME_CHANNEL_NAME",
"DINGTALK_HOME_CHANNEL",
"DINGTALK_HOME_CHANNEL_THREAD_ID",
"DINGTALK_HOME_CHANNEL_NAME",
"FEISHU_HOME_CHANNEL",
"FEISHU_HOME_CHANNEL_THREAD_ID",
"FEISHU_HOME_CHANNEL_NAME",
"WECOM_HOME_CHANNEL",
"WECOM_HOME_CHANNEL_THREAD_ID",
"WECOM_HOME_CHANNEL_NAME",
# API server bind/auth settings are common in local gateway profiles and
# change adapter defaults plus load_gateway_config() enablement. Tests that
# need them set opt in explicitly with monkeypatch.
"API_SERVER_ENABLED",
"API_SERVER_HOST",
"API_SERVER_PORT",
"API_SERVER_KEY",
"API_SERVER_CORS_ORIGINS",
"API_SERVER_MODEL_NAME",
# Platform gating — set by load_gateway_config() as a side effect when
# a config.yaml is present, so individual test bodies that call the
# loader leak these values into later tests in the same process.
# Force-clear on every test setup so the leak can't happen.
"SLACK_REQUIRE_MENTION",
"SLACK_STRICT_MENTION",
"SLACK_FREE_RESPONSE_CHANNELS",
"SLACK_ALLOW_BOTS",
"SLACK_REACTIONS",
"DISCORD_REQUIRE_MENTION",
"DISCORD_FREE_RESPONSE_CHANNELS",
"TELEGRAM_REQUIRE_MENTION",
"WHATSAPP_REQUIRE_MENTION",
"DINGTALK_REQUIRE_MENTION",
"MATRIX_REQUIRE_MENTION",
})
@pytest.fixture(autouse=True)
def _hermetic_environment(tmp_path, monkeypatch):
"""Blank out all credential/behavioral env vars so local and CI match.
Also redirects HOME and HERMES_HOME to per-test tempdirs so code that
reads ``~/.hermes/*`` can't touch the real one, and pins TZ/LANG so
datetime/locale-sensitive tests are deterministic.
"""
# 1. Blank every credential-shaped env var that's currently set.
for name in list(os.environ.keys()):
if _looks_like_credential(name):
monkeypatch.delenv(name, raising=False)
# 2. Blank behavioral HERMES_* vars that could change test semantics.
for name in _HERMES_BEHAVIORAL_VARS:
monkeypatch.delenv(name, raising=False)
# Honcho's fallback host/config resolution legitimately reads the user's
# global ~/.honcho/config.json. Keep HOME stable (subprocess tests depend
# on it), but pin the host so ordinary tests cannot inherit a developer's
# defaultHost and silently select the wrong nested config block. Tests of
# custom host resolution override/delete this explicitly.
monkeypatch.setenv("HERMES_HONCHO_HOST", "hermes")
# 3. Redirect HERMES_HOME to a per-test tempdir. Code that reads
# ``~/.hermes/*`` via ``get_hermes_home()`` now gets the tempdir.
#
# NOTE: We do NOT also redirect HOME. Doing so broke CI because
# some tests (and their transitive deps) spawn subprocesses that
# inherit HOME and expect it to be stable. If a test genuinely
# needs HOME isolated, it should set it explicitly in its own
# fixture. Any code in the codebase reading ``~/.hermes/*`` via
# ``Path.home() / ".hermes"`` instead of ``get_hermes_home()``
# is a bug to fix at the callsite.
fake_hermes_home = tmp_path / "hermes_test"
fake_hermes_home.mkdir()
(fake_hermes_home / "sessions").mkdir()
(fake_hermes_home / "cron").mkdir()
(fake_hermes_home / "memories").mkdir()
(fake_hermes_home / "skills").mkdir()
monkeypatch.setenv("HERMES_HOME", str(fake_hermes_home))
# 4. Deterministic locale / timezone / hashseed. CI runs in UTC with
# C.UTF-8 locale; local dev often doesn't. Pin everything.
monkeypatch.setenv("TZ", "UTC")
monkeypatch.setenv("LANG", "C.UTF-8")
monkeypatch.setenv("LC_ALL", "C.UTF-8")
monkeypatch.setenv("PYTHONHASHSEED", "0")
# 4b. Disable AWS IMDS lookups. Without this, any test that ends up
# calling has_aws_credentials() / resolve_aws_auth_env_var()
# (e.g. provider auto-detect, status command, cron run_job) burns
# ~2s waiting for the metadata service at 169.254.169.254 to time
# out. Tests don't run on EC2 — IMDS is always unreachable here.
monkeypatch.setenv("AWS_EC2_METADATA_DISABLED", "true")
monkeypatch.setenv("AWS_METADATA_SERVICE_TIMEOUT", "1")
monkeypatch.setenv("AWS_METADATA_SERVICE_NUM_ATTEMPTS", "1")
# Tirith auto-installs from GitHub when enabled and missing. Unit tests
# should never perform that implicit network/bootstrap path; Tirith-specific
# tests opt back in by patching the security config directly.
monkeypatch.setenv("TIRITH_ENABLED", "false")
# 5. Reset plugin singleton so tests don't leak plugins from
# ~/.hermes/plugins/ (which, per step 3, is now empty — but the
# singleton might still be cached from a previous test).
try:
import hermes_cli.plugins as _plugins_mod
monkeypatch.setattr(_plugins_mod, "_plugin_manager", None)
except Exception:
pass
# Explicitly clear provider-specific base URL overrides that don't match
# the generic credential-shaped env-var filter above.
monkeypatch.delenv("GMI_API_KEY", raising=False)
monkeypatch.delenv("GMI_BASE_URL", raising=False)
# Backward-compat alias — old tests reference this fixture name. Keep it
# as a no-op wrapper so imports don't break.
@pytest.fixture(autouse=True)
def _isolate_hermes_home(_hermetic_environment):
"""Alias preserved for any test that yields this name explicitly."""
return None
# ── Module-level state reset — replaced by per-file process isolation ──────
#
# Each test FILE runs in a freshly-spawned ``python -m pytest <file>``
# subprocess via ``scripts/run_tests_parallel.py``, so module-level dicts /
# sets / ContextVars from tests in one file cannot leak into tests in
# another file. No manual per-module clearing needed.
#
# Within a single file, ordering is the author's responsibility. If your
# tests in the same file share mutable state, either reset it explicitly
# in a fixture or split them across files.
#
# The skill ``test-suite-cascade-diagnosis`` documents the cascade patterns
# this replaces; the running example was ``test_command_guards`` failing
# 12/15 CI runs because ``tools.approval._session_approved`` carried
# approvals from one test's session into another's.
@pytest.fixture()
def tmp_dir(tmp_path):
"""Provide a temporary directory that is cleaned up automatically."""
return tmp_path
@pytest.fixture()
def mock_config():
"""Return a minimal hermes config dict suitable for unit tests."""
return {
"model": "test/mock-model",
"toolsets": ["terminal", "file"],
"max_turns": 10,
"terminal": {
"backend": "local",
"cwd": "/tmp",
"timeout": 30,
},
"compression": {"enabled": False},
"memory": {"memory_enabled": False, "user_profile_enabled": False},
"command_allowlist": [],
}
# ── Per-test timeout — handled by the isolation plugin ─────────────────────
#
# The subprocess-per-test plugin enforces the configured ``isolate_timeout``
# ini key by terminating the child if it overruns. The old SIGALRM-based
# fixture (POSIX-only, didn't work on Windows) is gone.
@pytest.fixture(autouse=True)
def _ensure_current_event_loop(request):
"""Provide a default event loop for sync tests that call get_event_loop().
Python 3.11+ no longer guarantees a current loop for plain synchronous tests.
A number of gateway tests still use asyncio.get_event_loop().run_until_complete(...).
Ensure they always have a usable loop without interfering with pytest-asyncio's
own loop management for @pytest.mark.asyncio tests.
On Python 3.12+, ``asyncio.get_event_loop_policy().get_event_loop()`` with no
*running* loop emits DeprecationWarning; skip that path and install a fresh
loop via ``new_event_loop()`` instead.
"""
if request.node.get_closest_marker("asyncio") is not None:
yield
return
loop = None
try:
loop = asyncio.get_running_loop()
except RuntimeError:
pass
if loop is None and sys.version_info < (3, 12):
try:
loop = asyncio.get_event_loop_policy().get_event_loop()
except RuntimeError:
loop = None
created = loop is None or loop.is_closed()
if created:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
yield
finally:
if created and loop is not None:
try:
loop.close()
finally:
asyncio.set_event_loop(None)
# ── Live-system guard ──────────────────────────────────────────────────────
#
# Several test files exercise the gateway-restart / kill code paths
# (``cmd_update``, ``kill_gateway_processes``, ``stop_profile_gateway``).
# When a single test forgets to mock either ``os.kill`` or the global
# ``find_gateway_pids`` helper, the real call leaks out of the hermetic
# environment and finds the developer's live ``hermes-gateway`` process
# via ``psutil`` — sending it SIGTERM mid-test. The shutdown forensics in
# PR #23285 caught this happening 5+ times in 3 days, every time
# correlated with a ``tests/hermes_cli/`` pytest run starting up.
#
# This fixture makes the leak impossible by intercepting the two
# primitives that actually do damage:
#
# • ``os.kill`` rejects any PID outside the test process subtree with
# a hard ``RuntimeError`` so the offending test gets a stack trace
# instead of silently murdering the real gateway.
# • ``subprocess.run`` / ``subprocess.Popen`` / ``call`` / ``check_call`` /
# ``check_output`` reject any ``systemctl ... <verb> hermes-gateway``
# invocation that would mutate the live unit. Read-only systemctl
# calls (``status``, ``show``, ``list-units``) still pass through.
#
# We intentionally do NOT stub ``find_gateway_pids`` / ``_scan_gateway_pids``
# here — tests of those functions themselves need the real implementation.
# Even if a test gets the live gateway PID back from a real scan, the
# ``os.kill`` guard above catches the actual signal call, and the
# ``systemctl`` guard catches the systemd path. Discovery without
# delivery is harmless.
_LIVE_SYSTEM_GUARD_BYPASS_MARK = "live_system_guard_bypass"
def pytest_configure(config): # noqa: D401 — pytest hook
"""Register markers used by hermetic conftest."""
config.addinivalue_line(
"markers",
f"{_LIVE_SYSTEM_GUARD_BYPASS_MARK}: bypass the live-system guard "
"(only for tests that genuinely need real os.kill / subprocess "
"behaviour — e.g. PTY tests that signal their own child).",
)
# The pyproject addopts pin ``--timeout-method=signal`` relies on
# ``signal.SIGALRM``, which does not exist on Windows — pytest-timeout
# raises AttributeError at timer setup and the whole run aborts before any
# test executes. Fall back to the thread-based timer on Windows so the
# suite runs natively there (POSIX keeps the more reliable signal method).
if sys.platform == "win32" and getattr(config.option, "timeout_method", None) == "signal":
config.option.timeout_method = "thread"
@pytest.fixture(autouse=True)
def _live_system_guard(request, monkeypatch):
"""Block real os.kill / systemctl / gateway-pid scans during tests.
See block comment above for the why. Tests that genuinely need
real signal delivery (e.g. PTY tests that SIGINT their own child)
can opt out with ``@pytest.mark.live_system_guard_bypass``.
Coverage (every primitive that can deliver a signal to or otherwise
terminate a foreign process):
• os.kill, os.killpg (POSIX)
• subprocess.run / Popen / call / check_call / check_output
• subprocess.getoutput / getstatusoutput
• os.system / os.popen
• pty.spawn
• asyncio.create_subprocess_exec / create_subprocess_shell
Subprocess inspection looks at the WHOLE command string (not just
tokens[0]), so ``bash -c "systemctl restart hermes-gateway"``,
``sudo systemctl ...``, ``env systemctl ...``, ``setsid systemctl ...``
are all caught. ``pkill``/``killall``/``taskkill`` invocations
targeting hermes/python patterns are also blocked.
"""
if request.node.get_closest_marker(_LIVE_SYSTEM_GUARD_BYPASS_MARK):
yield
return
import os as _os
import shlex as _shlex
import subprocess as _subprocess
test_pid = _os.getpid()
# Capture the test process's existing children at fixture start —
# any *new* children spawned by the test are also allowlisted via
# the live psutil walk below. Static set keeps the fast path cheap.
try:
import psutil as _psutil
_initial_children = {
c.pid for c in _psutil.Process(test_pid).children(recursive=True)
}
except Exception:
_psutil = None
_initial_children = set()
def _is_own_subtree(pid: int) -> bool:
# PID 0 means "our own process group"; -1 means "every process we
# can signal". Both are dangerous when paired with SIGTERM/SIGKILL,
# but pid 0 is technically scoped to our group so allow it; pid -1
# is treated as foreign (refuse).
if pid == 0:
return True
if pid < 0:
return False
if pid == test_pid or pid in _initial_children:
return True
if _psutil is None:
return False
try:
walker = _psutil.Process(pid)
except Exception:
# Stale PID — kill would be a no-op anyway, allow it.
return True
try:
for parent in walker.parents():
if parent.pid == test_pid:
return True
except Exception:
return False
return False
real_kill = _os.kill
def _guarded_kill(pid, sig, *args, **kwargs):
# Signal 0 is a pure liveness probe — it cannot terminate anything.
# psutil.pid_exists() uses os.kill(pid, 0) on POSIX, and probing a
# just-killed grandchild that was reparented to init (zombie with a
# foreign parent chain) must not trip the guard. Flaked in CI on
# test_entire_tree_is_sigkilled_not_just_parent.
if int(sig) == 0:
return real_kill(pid, sig, *args, **kwargs)
if _is_own_subtree(int(pid)):
return real_kill(pid, sig, *args, **kwargs)
raise RuntimeError(
f"tests/conftest.py live-system guard: blocked os.kill("
f"{pid}, {sig}) — PID is outside the test process subtree. "
"If this fired in CI it means the test reached a real "
"kill_gateway_processes / stop_profile_gateway / cmd_update "
"code path without mocking find_gateway_pids and os.kill. "
"Mock both, or mark the test with "
"@pytest.mark.live_system_guard_bypass if real signal "
"delivery is genuinely required."
)
monkeypatch.setattr(_os, "kill", _guarded_kill)
# ``os.killpg`` is the same risk class — sends a signal to every
# process in a group. The gateway is a session leader (its own
# PGID == its PID), so killpg(gateway_pid, SIGTERM) is a one-shot
# kill of the live process. Allow it only when the target PGID is
# the test process's own group.
if hasattr(_os, "killpg"):
real_killpg = _os.killpg
own_pgid = _os.getpgrp()
def _guarded_killpg(pgid, sig, *args, **kwargs):
# Signal 0 is a pure liveness probe — never destructive.
if int(sig) == 0:
return real_killpg(pgid, sig, *args, **kwargs)
if int(pgid) == own_pgid or _is_own_subtree(int(pgid)):
return real_killpg(pgid, sig, *args, **kwargs)
raise RuntimeError(
f"tests/conftest.py live-system guard: blocked "
f"os.killpg({pgid}, {sig}) — PGID is outside the test "
"process group. See _live_system_guard for the why."
)
monkeypatch.setattr(_os, "killpg", _guarded_killpg)
# ── Subprocess command-string inspection (whole-line) ──────────
_HERMES_TOKENS = (
"hermes-gateway",
"hermes.service",
"hermes_cli.main gateway",
"hermes_cli/main.py gateway",
"gateway/run.py",
"hermes gateway",
)
_MUTATING_VERBS = (
"restart", "start", "stop", "kill", "reload",
"reset-failed", "enable", "disable", "mask", "unmask",
"daemon-reload", "try-restart", "reload-or-restart",
)
_PROCESS_KILLERS = ("pkill", "killall", "taskkill", "skill", "fuser")
def _cmd_to_string(cmd) -> str:
if cmd is None:
return ""
if isinstance(cmd, (bytes, bytearray)):
try:
return bytes(cmd).decode(errors="replace")
except Exception:
return ""
if isinstance(cmd, str):
return cmd
if isinstance(cmd, (list, tuple)):
try:
return " ".join(str(t) for t in cmd)
except Exception:
return ""
return str(cmd)
def _matches_hermes_gateway(cmd_str: str) -> bool:
low = cmd_str.lower()
return any(tok in low for tok in _HERMES_TOKENS)
def _is_blocked_systemctl(cmd) -> bool:
cmd_str = _cmd_to_string(cmd)
if "systemctl" not in cmd_str:
return False
if not _matches_hermes_gateway(cmd_str):
return False
try:
tokens = _shlex.split(cmd_str)
except ValueError:
tokens = cmd_str.split()
return any(verb in tokens for verb in _MUTATING_VERBS)
def _is_process_killer(cmd) -> bool:
cmd_str = _cmd_to_string(cmd)
try:
tokens = _shlex.split(cmd_str)
except ValueError:
tokens = cmd_str.split()
if not tokens:
return False
for tok in tokens:
head = tok.rsplit("/", 1)[-1].rsplit("\\", 1)[-1]
if head in _PROCESS_KILLERS:
low = cmd_str.lower()
# pkill -f pattern: catch hermes-themed patterns + a
# plain "python" -f which would catch the live gateway
# whose cmdline contains "python -m hermes_cli.main".
if (
"hermes" in low
or "gateway" in low
or ("python" in low and "-f" in tokens)
):
return True
return False
def _check_subprocess_cmd(name, cmd):
if _is_blocked_systemctl(cmd):
raise RuntimeError(
f"tests/conftest.py live-system guard: blocked "
f"subprocess.{name}({cmd!r}) — would mutate the "
"live hermes-gateway systemd unit. Mock "
"subprocess.run / _run_systemctl in the test, or "
"mark with @pytest.mark.live_system_guard_bypass."
)
if _is_process_killer(cmd):
raise RuntimeError(
f"tests/conftest.py live-system guard: blocked "
f"subprocess.{name}({cmd!r}) — process-killer command "
"targeting hermes/python could hit the live gateway. "
"Mark with @pytest.mark.live_system_guard_bypass if "
"intentional."
)
# Block any subprocess that would run `hermes update` (or the
# equivalent `python -m hermes_cli.main update`). These commands
# run `git fetch origin + git pull` against the REAL checkout,
# overwriting files like pyproject.toml mid-test-run and corrupting
# every subsequent subprocess that reads them. The corruption is
# especially insidious because the spawned process uses setsid/
# start_new_session=True, making it invisible to pytest's process
# tree (PPid=1) and nearly impossible to trace without explicit
# inotify/SHA watchdogs. Any test that legitimately needs to exercise
# the update-spawn path must mock subprocess.Popen explicitly.
cmd_str = _cmd_to_string(cmd)
low = cmd_str.lower()
if "update" in low and (
# hermes update / hermes update --gateway / setsid bash -c ... hermes update
("hermes" in low and "update" in low.split())
or
# python -m hermes_cli.main update --gateway
("hermes_cli" in low and "update" in low.split())
or
# venv/bin/hermes update (absolute path variant used in tests)
(".venv/bin/hermes" in low and "update" in low)
):
raise RuntimeError(
f"tests/conftest.py live-system guard: blocked "
f"subprocess.{name}({cmd!r}) — this command would run "
"`hermes update` against the real checkout, fetching "
"from origin and overwriting repo files (e.g. "
"pyproject.toml) mid-test-run. This corrupts every "
"subsequent subprocess in the same runner. "
"Mock subprocess.Popen (and subprocess.run if used) "
"in the test instead, or mark with "
"@pytest.mark.live_system_guard_bypass if genuinely "
"needed (e.g. an integration test testing the update "
"flow against a dedicated throwaway repo)."
)
def _wrap_subprocess(name, real):
def _guarded(cmd, *args, **kwargs):
_check_subprocess_cmd(name, cmd)
return real(cmd, *args, **kwargs)
_guarded.__name__ = f"_guarded_{name}"
# Make the wrapper subscriptable like the wrapped callable when
# the wrapped object is. ``subprocess.Popen[bytes]`` is used as
# a type annotation in third-party packages (mcp, etc.); replacing
# ``Popen`` with a plain function breaks ``Popen[bytes]`` at
# import time. Defer ``__class_getitem__`` to the original.
if hasattr(real, "__class_getitem__"):
_guarded.__class_getitem__ = real.__class_getitem__
return _guarded
def _wrap_popen():
"""Subclass Popen so isinstance checks AND Popen[bytes] still work."""
real = _subprocess.Popen
class _GuardedPopen(real): # type: ignore[misc, valid-type]
def __init__(self, cmd, *args, **kwargs):
_check_subprocess_cmd("Popen", cmd)
super().__init__(cmd, *args, **kwargs)
_GuardedPopen.__name__ = "Popen"
_GuardedPopen.__qualname__ = "Popen"
return _GuardedPopen
real_run = _subprocess.run
real_popen = _subprocess.Popen
real_call = _subprocess.call
real_check_call = _subprocess.check_call
real_check_output = _subprocess.check_output
real_getoutput = _subprocess.getoutput
real_getstatusoutput = _subprocess.getstatusoutput
monkeypatch.setattr(_subprocess, "run", _wrap_subprocess("run", real_run))
monkeypatch.setattr(_subprocess, "Popen", _wrap_popen())
monkeypatch.setattr(_subprocess, "call", _wrap_subprocess("call", real_call))
monkeypatch.setattr(
_subprocess, "check_call", _wrap_subprocess("check_call", real_check_call)
)
monkeypatch.setattr(
_subprocess,
"check_output",
_wrap_subprocess("check_output", real_check_output),
)
monkeypatch.setattr(
_subprocess, "getoutput", _wrap_subprocess("getoutput", real_getoutput)
)
monkeypatch.setattr(
_subprocess,
"getstatusoutput",
_wrap_subprocess("getstatusoutput", real_getstatusoutput),
)
# os.system / os.popen — same risk class, completely unwrapped before.
real_os_system = _os.system
real_os_popen = _os.popen
def _guarded_os_system(command):
_check_subprocess_cmd("os.system", command)
return real_os_system(command)
def _guarded_os_popen(cmd, *args, **kwargs):
_check_subprocess_cmd("os.popen", cmd)
return real_os_popen(cmd, *args, **kwargs)
monkeypatch.setattr(_os, "system", _guarded_os_system)
monkeypatch.setattr(_os, "popen", _guarded_os_popen)
# pty.spawn — POSIX-only.
try:
import pty as _pty
if hasattr(_pty, "spawn"):
real_pty_spawn = _pty.spawn
def _guarded_pty_spawn(argv, *args, **kwargs):
_check_subprocess_cmd("pty.spawn", argv)
return real_pty_spawn(argv, *args, **kwargs)
monkeypatch.setattr(_pty, "spawn", _guarded_pty_spawn)
except Exception:
pass
# asyncio.create_subprocess_* — bypasses subprocess module entirely.
try:
import asyncio as _asyncio
real_async_exec = _asyncio.create_subprocess_exec
real_async_shell = _asyncio.create_subprocess_shell
async def _guarded_async_exec(program, *args, **kwargs):
_check_subprocess_cmd(
"asyncio.create_subprocess_exec", [program, *args]
)
return await real_async_exec(program, *args, **kwargs)
async def _guarded_async_shell(cmd, *args, **kwargs):
_check_subprocess_cmd("asyncio.create_subprocess_shell", cmd)
return await real_async_shell(cmd, *args, **kwargs)
monkeypatch.setattr(_asyncio, "create_subprocess_exec", _guarded_async_exec)
monkeypatch.setattr(
_asyncio, "create_subprocess_shell", _guarded_async_shell
)
except Exception:
pass
yield