fix(tests): fix several failing/flaky tests on main (#6777)

* fix(tests): mock is_safe_url in tests that use example.com

Tests using example.com URLs were failing because is_safe_url does a real DNS lookup which fails in environments where example.com doesn't resolve, causing the request to be blocked before reaching the already-mocked HTTP client. This should fix around 17 failing tests.

These tests test logic, caching, etc. so mocking this method should not modify them in any way. TestMattermostSendUrlAsFile was already doing this so we follow the same pattern.

* fix(test): use case-insensitive lookup for model context length check

DEFAULT_CONTEXT_LENGTHS uses inconsistent casing (MiniMax keys are lowercase, Qwen keys are mixed-case) so the test was broken in some cases since it couldn't find the model.

* fix(test): patch is_linux in systemd gateway restart test

The test only patched is_macos to False but didn't patch is_linux to True. On macOS hosts, is_linux() returns False and the systemd restart code path is skipped entirely, making the assertion fail.

* fix(test): use non-blocklisted env var in docker forward_env tests

GITHUB_TOKEN is in api_key_env_vars and thus in _HERMES_PROVIDER_ENV_BLOCKLIST so the env var is silently dropped, we replace it with a non-blocked one like DATABASE_URL so the tests actually work.

* fix(test): fully isolate _has_any_provider_configured from host env

_has_any_provider_configured() checks all env vars from PROVIDER_REGISTRY (not just the 5 the tests were clearing) and also calls get_auth_status() which detects gh auth token for Copilot. On machines with any of these set, the function returns True before reaching the code path under test.

Clear all registry vars and mock get_auth_status so host credentials don't interfere.

* fix(test): correct path to hermes_base_env.py in tool parser tests

Path(__file__).parent.parent resolved to tests/, not the project root.
The file lives at environments/hermes_base_env.py so we need one more parent level.

* fix(test): accept optional HTML fields in Matrix send payload

_send_matrix sometimes adds format and formatted_body when the markdown library is installed. The test was doing an exact dict equality check which broke. Check required fields instead.

* fix(test): add config.yaml to codex vision requirements test

The test only wrote auth.json but not config.yaml, so _read_main_provider() returned empty and vision auto-detect never tried the codex provider. Add a config.yaml pointing at openai-codex so the fallback path actually resolves the client.

* fix(test): clear OPENROUTER_API_KEY in _isolate_hermes_home

run_agent.py calls load_hermes_dotenv() at import time, which injects API keys from ~/.hermes/.env into os.environ before any test fixture runs. This caused test_agent_loop_tool_calling to make real API calls instead of skipping, which ends up making some tests fail.

* fix(test): add get_rate_limit_state to agent mock in usage report tests

_show_usage now calls agent.get_rate_limit_state() for rate limit
  display. The SimpleNamespace mock was missing this method.

* fix(test): update expected Camofox config version from 12 to 13

* fix(test): mock _get_enabled_platforms in nous managed defaults test

Importing gateway.run leaks DISCORD_BOT_TOKEN into os.environ, which makes _get_enabled_platforms() return ["cli", "discord"] instead of just ["cli"]. tools_command loops per platform, so apply_nous_managed_defaults
  runs twice: the first call sets config values, the second sees them as
  already configured and returns an empty set, causing the assertion to
  fail.
This commit is contained in:
Dylan Socolobsky 2026-04-09 17:17:06 -03:00 committed by GitHub
parent 3eade90b39
commit c6dba918b3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 85 additions and 33 deletions

View file

@ -41,6 +41,7 @@ def _attach_agent(
session_completion_tokens=completion_tokens,
session_total_tokens=total_tokens,
session_api_calls=api_calls,
get_rate_limit_state=lambda: None,
context_compressor=SimpleNamespace(
last_prompt_tokens=context_tokens,
context_length=context_length,

View file

@ -38,6 +38,8 @@ def _isolate_hermes_home(tmp_path, monkeypatch):
monkeypatch.delenv("HERMES_SESSION_CHAT_ID", raising=False)
monkeypatch.delenv("HERMES_SESSION_CHAT_NAME", raising=False)
monkeypatch.delenv("HERMES_GATEWAY_SESSION", raising=False)
# Avoid making real calls during tests if this key is set in the env files
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
@pytest.fixture()

View file

@ -38,10 +38,11 @@ def _make_timeout_error() -> httpx.TimeoutException:
# cache_image_from_url (base.py)
# ---------------------------------------------------------------------------
@patch("tools.url_safety.is_safe_url", return_value=True)
class TestCacheImageFromUrl:
"""Tests for gateway.platforms.base.cache_image_from_url"""
def test_success_on_first_attempt(self, tmp_path, monkeypatch):
def test_success_on_first_attempt(self, _mock_safe, tmp_path, monkeypatch):
"""A clean 200 response caches the image and returns a path."""
monkeypatch.setattr("gateway.platforms.base.IMAGE_CACHE_DIR", tmp_path / "img")
@ -65,7 +66,7 @@ class TestCacheImageFromUrl:
assert path.endswith(".jpg")
mock_client.get.assert_called_once()
def test_retries_on_timeout_then_succeeds(self, tmp_path, monkeypatch):
def test_retries_on_timeout_then_succeeds(self, _mock_safe, tmp_path, monkeypatch):
"""A timeout on the first attempt is retried; second attempt succeeds."""
monkeypatch.setattr("gateway.platforms.base.IMAGE_CACHE_DIR", tmp_path / "img")
@ -95,7 +96,7 @@ class TestCacheImageFromUrl:
assert mock_client.get.call_count == 2
mock_sleep.assert_called_once()
def test_retries_on_429_then_succeeds(self, tmp_path, monkeypatch):
def test_retries_on_429_then_succeeds(self, _mock_safe, tmp_path, monkeypatch):
"""A 429 response on the first attempt is retried; second attempt succeeds."""
monkeypatch.setattr("gateway.platforms.base.IMAGE_CACHE_DIR", tmp_path / "img")
@ -122,7 +123,7 @@ class TestCacheImageFromUrl:
assert path.endswith(".jpg")
assert mock_client.get.call_count == 2
def test_raises_after_max_retries_exhausted(self, tmp_path, monkeypatch):
def test_raises_after_max_retries_exhausted(self, _mock_safe, tmp_path, monkeypatch):
"""Timeout on every attempt raises after all retries are consumed."""
monkeypatch.setattr("gateway.platforms.base.IMAGE_CACHE_DIR", tmp_path / "img")
@ -145,7 +146,7 @@ class TestCacheImageFromUrl:
# 3 total calls: initial + 2 retries
assert mock_client.get.call_count == 3
def test_non_retryable_4xx_raises_immediately(self, tmp_path, monkeypatch):
def test_non_retryable_4xx_raises_immediately(self, _mock_safe, tmp_path, monkeypatch):
"""A 404 (non-retryable) is raised immediately without any retry."""
monkeypatch.setattr("gateway.platforms.base.IMAGE_CACHE_DIR", tmp_path / "img")
@ -175,10 +176,11 @@ class TestCacheImageFromUrl:
# cache_audio_from_url (base.py)
# ---------------------------------------------------------------------------
@patch("tools.url_safety.is_safe_url", return_value=True)
class TestCacheAudioFromUrl:
"""Tests for gateway.platforms.base.cache_audio_from_url"""
def test_success_on_first_attempt(self, tmp_path, monkeypatch):
def test_success_on_first_attempt(self, _mock_safe, tmp_path, monkeypatch):
"""A clean 200 response caches the audio and returns a path."""
monkeypatch.setattr("gateway.platforms.base.AUDIO_CACHE_DIR", tmp_path / "audio")
@ -202,7 +204,7 @@ class TestCacheAudioFromUrl:
assert path.endswith(".ogg")
mock_client.get.assert_called_once()
def test_retries_on_timeout_then_succeeds(self, tmp_path, monkeypatch):
def test_retries_on_timeout_then_succeeds(self, _mock_safe, tmp_path, monkeypatch):
"""A timeout on the first attempt is retried; second attempt succeeds."""
monkeypatch.setattr("gateway.platforms.base.AUDIO_CACHE_DIR", tmp_path / "audio")
@ -232,7 +234,7 @@ class TestCacheAudioFromUrl:
assert mock_client.get.call_count == 2
mock_sleep.assert_called_once()
def test_retries_on_429_then_succeeds(self, tmp_path, monkeypatch):
def test_retries_on_429_then_succeeds(self, _mock_safe, tmp_path, monkeypatch):
"""A 429 response on the first attempt is retried; second attempt succeeds."""
monkeypatch.setattr("gateway.platforms.base.AUDIO_CACHE_DIR", tmp_path / "audio")
@ -259,7 +261,7 @@ class TestCacheAudioFromUrl:
assert path.endswith(".ogg")
assert mock_client.get.call_count == 2
def test_retries_on_500_then_succeeds(self, tmp_path, monkeypatch):
def test_retries_on_500_then_succeeds(self, _mock_safe, tmp_path, monkeypatch):
"""A 500 response on the first attempt is retried; second attempt succeeds."""
monkeypatch.setattr("gateway.platforms.base.AUDIO_CACHE_DIR", tmp_path / "audio")
@ -286,7 +288,7 @@ class TestCacheAudioFromUrl:
assert path.endswith(".ogg")
assert mock_client.get.call_count == 2
def test_raises_after_max_retries_exhausted(self, tmp_path, monkeypatch):
def test_raises_after_max_retries_exhausted(self, _mock_safe, tmp_path, monkeypatch):
"""Timeout on every attempt raises after all retries are consumed."""
monkeypatch.setattr("gateway.platforms.base.AUDIO_CACHE_DIR", tmp_path / "audio")
@ -309,7 +311,7 @@ class TestCacheAudioFromUrl:
# 3 total calls: initial + 2 retries
assert mock_client.get.call_count == 3
def test_non_retryable_4xx_raises_immediately(self, tmp_path, monkeypatch):
def test_non_retryable_4xx_raises_immediately(self, _mock_safe, tmp_path, monkeypatch):
"""A 404 (non-retryable) is raised immediately without any retry."""
monkeypatch.setattr("gateway.platforms.base.AUDIO_CACHE_DIR", tmp_path / "audio")

View file

@ -4,7 +4,7 @@ import base64
import os
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock
from unittest.mock import AsyncMock, patch
import pytest
@ -355,7 +355,8 @@ class TestMediaUpload:
assert calls[3][1]["chunk_index"] == 2
@pytest.mark.asyncio
async def test_download_remote_bytes_rejects_large_content_length(self):
@patch("tools.url_safety.is_safe_url", return_value=True)
async def test_download_remote_bytes_rejects_large_content_length(self, _mock_safe):
from gateway.platforms.wecom import WeComAdapter
class FakeResponse:

View file

@ -628,14 +628,21 @@ class TestHasAnyProviderConfigured:
def test_claude_code_creds_ignored_on_fresh_install(self, monkeypatch, tmp_path):
"""Claude Code credentials should NOT skip the wizard when Hermes is unconfigured."""
from hermes_cli import config as config_module
from hermes_cli.auth import PROVIDER_REGISTRY
hermes_home = tmp_path / ".hermes"
hermes_home.mkdir()
monkeypatch.setattr(config_module, "get_env_path", lambda: hermes_home / ".env")
monkeypatch.setattr(config_module, "get_hermes_home", lambda: hermes_home)
# Clear all provider env vars so earlier checks don't short-circuit
for var in ("OPENROUTER_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY",
"ANTHROPIC_TOKEN", "OPENAI_BASE_URL"):
_all_vars = {"OPENROUTER_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY",
"ANTHROPIC_TOKEN", "OPENAI_BASE_URL"}
for pconfig in PROVIDER_REGISTRY.values():
if pconfig.auth_type == "api_key":
_all_vars.update(pconfig.api_key_env_vars)
for var in _all_vars:
monkeypatch.delenv(var, raising=False)
# Prevent gh-cli / copilot auth fallback from leaking in
monkeypatch.setattr("hermes_cli.auth.get_auth_status", lambda _pid: {})
# Simulate valid Claude Code credentials
monkeypatch.setattr(
"agent.anthropic_adapter.read_claude_code_credentials",
@ -710,6 +717,7 @@ class TestHasAnyProviderConfigured:
"""config.yaml model dict with empty default and no creds stays false."""
import yaml
from hermes_cli import config as config_module
from hermes_cli.auth import PROVIDER_REGISTRY
hermes_home = tmp_path / ".hermes"
hermes_home.mkdir()
config_file = hermes_home / "config.yaml"
@ -719,9 +727,15 @@ class TestHasAnyProviderConfigured:
monkeypatch.setattr(config_module, "get_env_path", lambda: hermes_home / ".env")
monkeypatch.setattr(config_module, "get_hermes_home", lambda: hermes_home)
monkeypatch.setenv("HERMES_HOME", str(hermes_home))
for var in ("OPENROUTER_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY",
"ANTHROPIC_TOKEN", "OPENAI_BASE_URL"):
_all_vars = {"OPENROUTER_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY",
"ANTHROPIC_TOKEN", "OPENAI_BASE_URL"}
for pconfig in PROVIDER_REGISTRY.values():
if pconfig.auth_type == "api_key":
_all_vars.update(pconfig.api_key_env_vars)
for var in _all_vars:
monkeypatch.delenv(var, raising=False)
# Prevent gh-cli / copilot auth fallback from leaking in
monkeypatch.setattr("hermes_cli.auth.get_auth_status", lambda _pid: {})
from hermes_cli.main import _has_any_provider_configured
assert _has_any_provider_configured() is False
@ -941,9 +955,10 @@ class TestHuggingFaceModels:
"""Every HF model should have a context length entry."""
from hermes_cli.models import _PROVIDER_MODELS
from agent.model_metadata import DEFAULT_CONTEXT_LENGTHS
lower_keys = {k.lower() for k in DEFAULT_CONTEXT_LENGTHS}
hf_models = _PROVIDER_MODELS["huggingface"]
for model in hf_models:
assert model in DEFAULT_CONTEXT_LENGTHS, (
assert model.lower() in lower_keys, (
f"HF model {model!r} missing from DEFAULT_CONTEXT_LENGTHS"
)

View file

@ -354,6 +354,14 @@ def test_first_install_nous_auto_configures_managed_defaults(monkeypatch):
lambda *args, **kwargs: {"web", "image_gen", "tts", "browser"},
)
monkeypatch.setattr("hermes_cli.tools_config.save_config", lambda config: None)
# Prevent leaked platform tokens (e.g. DISCORD_BOT_TOKEN from gateway.run
# import) from adding extra platforms. The loop in tools_command runs
# apply_nous_managed_defaults per platform; a second iteration sees values
# set by the first as "explicit" and skips them.
monkeypatch.setattr(
"hermes_cli.tools_config._get_enabled_platforms",
lambda: ["cli"],
)
monkeypatch.setattr(
"hermes_cli.nous_subscription.get_nous_auth_status",
lambda: {"logged_in": True},

View file

@ -368,6 +368,9 @@ class TestCmdUpdateLaunchdRestart:
monkeypatch.setattr(
gateway_cli, "is_macos", lambda: False,
)
monkeypatch.setattr(
gateway_cli, "is_linux", lambda: True,
)
mock_run.side_effect = _make_run_side_effect(
commit_count="3",

View file

@ -63,4 +63,4 @@ class TestCamofoxConfigDefaults:
from hermes_cli.config import DEFAULT_CONFIG
# managed_persistence is auto-merged by _deep_merge, no version bump needed
assert DEFAULT_CONFIG["_config_version"] == 12
assert DEFAULT_CONFIG["_config_version"] == 13

View file

@ -258,28 +258,30 @@ def _make_execute_only_env(forward_env=None):
def test_init_env_args_uses_hermes_dotenv_for_allowlisted_env(monkeypatch):
"""_build_init_env_args picks up forwarded env vars from .env file at init time."""
env = _make_execute_only_env(["GITHUB_TOKEN"])
# Use a var that is NOT in _HERMES_PROVIDER_ENV_BLOCKLIST (GITHUB_TOKEN
# is in the copilot provider's api_key_env_vars and gets stripped).
env = _make_execute_only_env(["DATABASE_URL"])
monkeypatch.delenv("GITHUB_TOKEN", raising=False)
monkeypatch.setattr(docker_env, "_load_hermes_env_vars", lambda: {"GITHUB_TOKEN": "value_from_dotenv"})
monkeypatch.delenv("DATABASE_URL", raising=False)
monkeypatch.setattr(docker_env, "_load_hermes_env_vars", lambda: {"DATABASE_URL": "value_from_dotenv"})
args = env._build_init_env_args()
args_str = " ".join(args)
assert "GITHUB_TOKEN=value_from_dotenv" in args_str
assert "DATABASE_URL=value_from_dotenv" in args_str
def test_init_env_args_prefers_shell_env_over_hermes_dotenv(monkeypatch):
"""Shell env vars take priority over .env file values in init env args."""
env = _make_execute_only_env(["GITHUB_TOKEN"])
env = _make_execute_only_env(["DATABASE_URL"])
monkeypatch.setenv("GITHUB_TOKEN", "value_from_shell")
monkeypatch.setattr(docker_env, "_load_hermes_env_vars", lambda: {"GITHUB_TOKEN": "value_from_dotenv"})
monkeypatch.setenv("DATABASE_URL", "value_from_shell")
monkeypatch.setattr(docker_env, "_load_hermes_env_vars", lambda: {"DATABASE_URL": "value_from_dotenv"})
args = env._build_init_env_args()
args_str = " ".join(args)
assert "GITHUB_TOKEN=value_from_shell" in args_str
assert "DATABASE_URL=value_from_shell" in args_str
assert "value_from_dotenv" not in args_str

View file

@ -147,7 +147,7 @@ class TestBaseEnvCompatibility:
"""Hermes wires parser selection through ServerManager.tool_parser."""
import ast
base_env_path = Path(__file__).parent.parent / "environments" / "hermes_base_env.py"
base_env_path = Path(__file__).parent.parent.parent / "environments" / "hermes_base_env.py"
source = base_env_path.read_text()
tree = ast.parse(source)
@ -171,7 +171,7 @@ class TestBaseEnvCompatibility:
def test_hermes_base_env_uses_config_tool_call_parser(self):
"""Verify hermes_base_env uses the config field rather than a local parser instance."""
base_env_path = Path(__file__).parent.parent / "environments" / "hermes_base_env.py"
base_env_path = Path(__file__).parent.parent.parent / "environments" / "hermes_base_env.py"
source = base_env_path.read_text()
assert 'tool_call_parser: str = Field(' in source

View file

@ -125,7 +125,9 @@ class TestSendMatrix:
url = call_kwargs[0][0]
assert url.startswith("https://matrix.example.com/_matrix/client/v3/rooms/!room:example.com/send/m.room.message/")
assert call_kwargs[1]["headers"]["Authorization"] == "Bearer syt_tok"
assert call_kwargs[1]["json"] == {"msgtype": "m.text", "body": "hello matrix"}
payload = call_kwargs[1]["json"]
assert payload["msgtype"] == "m.text"
assert payload["body"] == "hello matrix"
def test_http_error(self):
resp = _make_aiohttp_resp(403, text_data="Forbidden")

View file

@ -30,7 +30,10 @@ class TestValidateImageUrl:
"""Tests for URL validation, including urlparse-based netloc check."""
def test_valid_https_url(self):
assert _validate_image_url("https://example.com/image.jpg") is True
with patch("tools.url_safety.socket.getaddrinfo", return_value=[
(2, 1, 6, "", ("93.184.216.34", 0)),
]):
assert _validate_image_url("https://example.com/image.jpg") is True
def test_valid_http_url(self):
with patch("tools.url_safety.socket.getaddrinfo", return_value=[
@ -56,10 +59,16 @@ class TestValidateImageUrl:
assert _validate_image_url("http://localhost:8080/image.png") is False
def test_valid_url_with_port(self):
assert _validate_image_url("http://example.com:8080/image.png") is True
with patch("tools.url_safety.socket.getaddrinfo", return_value=[
(2, 1, 6, "", ("93.184.216.34", 0)),
]):
assert _validate_image_url("http://example.com:8080/image.png") is True
def test_valid_url_with_path_only(self):
assert _validate_image_url("https://example.com/") is True
with patch("tools.url_safety.socket.getaddrinfo", return_value=[
(2, 1, 6, "", ("93.184.216.34", 0)),
]):
assert _validate_image_url("https://example.com/") is True
def test_rejects_empty_string(self):
assert _validate_image_url("") is False
@ -441,6 +450,11 @@ class TestVisionRequirements:
(tmp_path / "auth.json").write_text(
'{"active_provider":"openai-codex","providers":{"openai-codex":{"tokens":{"access_token":"codex-access-token","refresh_token":"codex-refresh-token"}}}}'
)
# config.yaml must reference the codex provider so vision auto-detect
# falls back to the active provider via _read_main_provider().
(tmp_path / "config.yaml").write_text(
'model:\n default: gpt-4o\n provider: openai-codex\n'
)
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
monkeypatch.delenv("OPENAI_BASE_URL", raising=False)
monkeypatch.delenv("OPENAI_API_KEY", raising=False)

View file

@ -225,6 +225,7 @@ class TestWebCrawlTavily:
patch.dict(os.environ, {"TAVILY_API_KEY": "tvly-test"}), \
patch("tools.web_tools.httpx.post", return_value=mock_response), \
patch("tools.web_tools.check_website_access", return_value=None), \
patch("tools.web_tools.is_safe_url", return_value=True), \
patch("tools.interrupt.is_interrupted", return_value=False):
from tools.web_tools import web_crawl_tool
result = json.loads(asyncio.get_event_loop().run_until_complete(
@ -244,6 +245,7 @@ class TestWebCrawlTavily:
patch.dict(os.environ, {"TAVILY_API_KEY": "tvly-test"}), \
patch("tools.web_tools.httpx.post", return_value=mock_response) as mock_post, \
patch("tools.web_tools.check_website_access", return_value=None), \
patch("tools.web_tools.is_safe_url", return_value=True), \
patch("tools.interrupt.is_interrupted", return_value=False):
from tools.web_tools import web_crawl_tool
asyncio.get_event_loop().run_until_complete(