mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-25 17:18:11 +00:00
fix(tests): gate WAL-dependent tests on the linked SQLite's real capability
Some checks are pending
CI / Detect affected areas (push) Waiting to run
CI / Python tests (push) Blocked by required conditions
CI / Python lints (push) Blocked by required conditions
CI / JS & TS checks (push) Blocked by required conditions
CI / Desktop E2E (push) Blocked by required conditions
CI / Docs Site (push) Blocked by required conditions
CI / Deny unrelated histories (push) Blocked by required conditions
CI / Check contributors (push) Blocked by required conditions
CI / Check uv.lock (push) Blocked by required conditions
CI / package-lock.json diff (push) Blocked by required conditions
CI / Lint Docker scripts (push) Blocked by required conditions
CI / Build&Test Docker image (push) Blocked by required conditions
CI / Supply-chain scan (push) Blocked by required conditions
CI / Review label gate (push) Blocked by required conditions
CI / OSV scan (push) Waiting to run
CI / CI review comment (live) (push) Blocked by required conditions
CI / All required checks pass (push) Blocked by required conditions
CI / CI timing report (push) Blocked by required conditions
Deploy Site / deploy-vercel (push) Waiting to run
Deploy Site / deploy-docs (push) Waiting to run
Docker Build, Test, and Publish / build (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Waiting to run
Docker Build, Test, and Publish / build (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Docker Build, Test, and Publish / publish (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Blocked by required conditions
Docker Build, Test, and Publish / publish (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
Docker Build, Test, and Publish / merge (push) Blocked by required conditions
auto-fix lint issues & formatting / Generate eslint --fix patch (push) Waiting to run
auto-fix lint issues & formatting / Apply patch (push) Blocked by required conditions
Some checks are pending
CI / Detect affected areas (push) Waiting to run
CI / Python tests (push) Blocked by required conditions
CI / Python lints (push) Blocked by required conditions
CI / JS & TS checks (push) Blocked by required conditions
CI / Desktop E2E (push) Blocked by required conditions
CI / Docs Site (push) Blocked by required conditions
CI / Deny unrelated histories (push) Blocked by required conditions
CI / Check contributors (push) Blocked by required conditions
CI / Check uv.lock (push) Blocked by required conditions
CI / package-lock.json diff (push) Blocked by required conditions
CI / Lint Docker scripts (push) Blocked by required conditions
CI / Build&Test Docker image (push) Blocked by required conditions
CI / Supply-chain scan (push) Blocked by required conditions
CI / Review label gate (push) Blocked by required conditions
CI / OSV scan (push) Waiting to run
CI / CI review comment (live) (push) Blocked by required conditions
CI / All required checks pass (push) Blocked by required conditions
CI / CI timing report (push) Blocked by required conditions
Deploy Site / deploy-vercel (push) Waiting to run
Deploy Site / deploy-docs (push) Waiting to run
Docker Build, Test, and Publish / build (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Waiting to run
Docker Build, Test, and Publish / build (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Waiting to run
Docker Build, Test, and Publish / publish (amd64, type=gha,scope=docker-amd64, type=gha,mode=max,scope=docker-amd64, linux/amd64, ubuntu-latest) (push) Blocked by required conditions
Docker Build, Test, and Publish / publish (arm64, type=gha,scope=docker-arm64, type=gha,mode=max,scope=docker-arm64, linux/arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
Docker Build, Test, and Publish / merge (push) Blocked by required conditions
auto-fix lint issues & formatting / Generate eslint --fix patch (push) Waiting to run
auto-fix lint issues & formatting / Apply patch (push) Blocked by required conditions
Two tests fail deterministically on main depending only on which SQLite the
test interpreter links — nothing about the code under test. Both are green in
isolation and red in the suite / on an older library, the worst diagnostic
shape.
Root cause A — WAL is not always WAL. Hermes refuses journal_mode=WAL on
SQLite builds carrying the upstream WAL-reset corruption bug (3.7.0–3.51.2,
excluding backports 3.50.7 / 3.44.6) and falls back to DELETE. On such a
build NO -wal sidecar is ever created, so
test_wal_checkpoint_truncates_wal_file asserts on a file that cannot exist.
Invisible locally when the repo .venv and the Hermes managed runtime link
different versions (observed: .venv 3.50.4 → DELETE, runtime 3.53.1 → WAL),
so the same test passes for one interpreter and fails for the other.
- tests/conftest.py: add a `requires_wal` marker plus a
pytest_collection_modifyitems hook that skips such tests when the linked
library will fall back to DELETE. The skip reason names the actual
version so it is diagnosable rather than mysterious.
- pyproject.toml: register the marker.
- test_kanban_db_repair.py: mark the -wal-sidecar test.
Root cause B — process-global warn-once dedup. The WAL-fallback warning is
emitted at most once per (process, db_label). Any earlier test in
test_kanban_db.py that opens a kanban.db consumes that one-shot, so
test_connect_falls_back_to_delete_on_locking_protocol sees zero warnings and
fails — but only as part of the file, never alone.
- test_kanban_db.py: clear both dedup sets in the test that asserts on the
warning, with a comment explaining the isolation trap.
The gate deliberately does NOT import hermes_state. That module computes
DEFAULT_DB_PATH from get_hermes_home() at import time, so importing it during
collection — before the per-test _isolate_hermes_home fixture redirects
HERMES_HOME — permanently caches the developer's REAL ~/.hermes/state.db for
the whole session. The first version of this change did exactly that and made
tests read a live 31,881-session production database (test_console_engine
asserted "Total sessions: 2" and got 31881). The version predicate is
duplicated instead, and tests/test_conftest_wal_gate.py pins the two
implementations in agreement across every documented upstream boundary plus
guards against the import coming back.
Verified: on SQLite 3.50.4 the sidecar test SKIPS naming the version; on
3.53.1 it RUNS and passes, so coverage is not lost where WAL works. Clean
main fails exactly these 2 tests under
`scripts/run_tests.sh tests/hermes_cli/ tests/test_hermes_state.py`
(9926 passed, 2 failed); with this change the same scope is green.
Tests: 726 passed across the two kanban files, test_hermes_state.py, and the
new gate tests.
This commit is contained in:
parent
35b1e57862
commit
07e97d2f5d
5 changed files with 155 additions and 0 deletions
|
|
@ -332,6 +332,7 @@ markers = [
|
|||
"integration: marks tests requiring external services (API keys, Modal, etc.)",
|
||||
"real_concurrent_gate: opt out of the autouse stub that disables _detect_concurrent_hermes_instances",
|
||||
"real_agent_prewarm: opt out of the autouse stub that disables the tui_gateway deferred agent pre-warm timer",
|
||||
"requires_wal: needs the runtime to actually enable SQLite WAL mode (skipped where Hermes falls back to journal_mode=DELETE)",
|
||||
]
|
||||
# integration tests take way too long to run in the normal CI environments
|
||||
addopts = "-m 'not integration'"
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ test runner at ``scripts/run_tests.sh``.
|
|||
|
||||
import asyncio
|
||||
import os
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
|
@ -541,6 +542,44 @@ def _ensure_current_event_loop(request):
|
|||
# delivery is harmless.
|
||||
|
||||
_LIVE_SYSTEM_GUARD_BYPASS_MARK = "live_system_guard_bypass"
|
||||
_REQUIRES_WAL_MARK = "requires_wal"
|
||||
|
||||
|
||||
def _wal_is_usable() -> bool:
|
||||
"""True when Hermes will actually put a database into WAL mode here.
|
||||
|
||||
Hermes refuses journal_mode=WAL on SQLite builds carrying the upstream
|
||||
WAL-reset corruption bug (3.7.0–3.51.2, excluding backports 3.50.7 /
|
||||
3.44.6) and falls back to DELETE. On such a build NO ``-wal`` sidecar is
|
||||
ever created, so a test asserting on WAL frames, ``-wal`` file size, or
|
||||
checkpoint behaviour cannot pass — it is testing a mode the runtime
|
||||
declined to enable, not a regression.
|
||||
|
||||
This matters because the interpreter running the tests and the interpreter
|
||||
running Hermes can link DIFFERENT SQLite versions: a repo ``.venv`` on
|
||||
3.50.4 (vulnerable → DELETE) alongside a Hermes managed runtime on 3.53.1
|
||||
(fixed → WAL). The same test then passes in one and fails in the other.
|
||||
|
||||
IMPORTANT: this must NOT import ``hermes_state``. That module computes
|
||||
``DEFAULT_DB_PATH`` from ``get_hermes_home()`` at import time, so importing
|
||||
it during collection — before the per-test ``_isolate_hermes_home`` fixture
|
||||
redirects ``HERMES_HOME`` — permanently caches the DEVELOPER'S REAL
|
||||
``~/.hermes/state.db`` for the whole session. Tests then read live
|
||||
production sessions instead of a tempdir. The version predicate is
|
||||
duplicated from ``hermes_state._is_sqlite_wal_reset_vulnerable`` (upstream
|
||||
fixed ranges, stable) rather than imported, and
|
||||
``test_conftest_wal_gate.py`` pins the two implementations in agreement.
|
||||
"""
|
||||
info = sqlite3.sqlite_version_info
|
||||
if info < (3, 7, 0):
|
||||
return True # pre-WAL library: cannot hit the race
|
||||
if info >= (3, 51, 3):
|
||||
return True # fixed upstream
|
||||
if (3, 50, 7) <= info < (3, 51, 0):
|
||||
return True # 3.50.x backport
|
||||
if (3, 44, 6) <= info < (3, 45, 0):
|
||||
return True # 3.44.x backport
|
||||
return False
|
||||
|
||||
|
||||
def pytest_configure(config): # noqa: D401 — pytest hook
|
||||
|
|
@ -551,6 +590,12 @@ def pytest_configure(config): # noqa: D401 — pytest hook
|
|||
"(only for tests that genuinely need real os.kill / subprocess "
|
||||
"behaviour — e.g. PTY tests that signal their own child).",
|
||||
)
|
||||
config.addinivalue_line(
|
||||
"markers",
|
||||
f"{_REQUIRES_WAL_MARK}: test needs the runtime to actually enable "
|
||||
"SQLite WAL mode; skipped on builds where Hermes falls back to "
|
||||
"journal_mode=DELETE for the WAL-reset bug.",
|
||||
)
|
||||
|
||||
# The pyproject addopts pin ``--timeout-method=signal`` relies on
|
||||
# ``signal.SIGALRM``, which does not exist on Windows — pytest-timeout
|
||||
|
|
@ -561,6 +606,26 @@ def pytest_configure(config): # noqa: D401 — pytest hook
|
|||
config.option.timeout_method = "thread"
|
||||
|
||||
|
||||
def pytest_collection_modifyitems(config, items): # noqa: D401 — pytest hook
|
||||
"""Skip ``requires_wal`` tests when the linked SQLite can't use WAL.
|
||||
|
||||
Cheaper and more honest than each test hand-rolling a version check: the
|
||||
reason string names the actual linked version so the skip is diagnosable
|
||||
rather than mysterious.
|
||||
"""
|
||||
if _wal_is_usable():
|
||||
return
|
||||
|
||||
reason = (
|
||||
f"SQLite {sqlite3.sqlite_version} has the WAL-reset bug — Hermes uses "
|
||||
"journal_mode=DELETE here, so no -wal sidecar exists to assert on"
|
||||
)
|
||||
skip_marker = pytest.mark.skip(reason=reason)
|
||||
for item in items:
|
||||
if item.get_closest_marker(_REQUIRES_WAL_MARK) is not None:
|
||||
item.add_marker(skip_marker)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _live_system_guard(request, monkeypatch):
|
||||
"""Block real os.kill / systemctl / gateway-pid scans during tests.
|
||||
|
|
|
|||
|
|
@ -3248,6 +3248,17 @@ def test_connect_falls_back_to_delete_on_locking_protocol(tmp_path, monkeypatch,
|
|||
import sqlite3 as _sqlite3
|
||||
from unittest.mock import patch as _patch
|
||||
|
||||
import hermes_state as _hs
|
||||
|
||||
# The fallback warning is deduped process-globally ("once per process per
|
||||
# database" — _log_wal_fallback_once / _log_wal_reset_bug_once). Any earlier
|
||||
# test in this file that opened a kanban.db already consumed the one-shot
|
||||
# for that label, so without clearing it this test sees zero warnings and
|
||||
# fails only when run as part of the file (it passes in isolation). Clear
|
||||
# both dedup sets so the warning is emitted for this connect().
|
||||
_hs._wal_fallback_warned_paths.clear()
|
||||
_hs._wal_reset_bug_warned_paths.clear()
|
||||
|
||||
home = tmp_path / ".hermes"
|
||||
home.mkdir()
|
||||
monkeypatch.setenv("HERMES_HOME", str(home))
|
||||
|
|
|
|||
|
|
@ -356,6 +356,7 @@ def test_wal_checkpoint_failure_never_fails_the_tick(tmp_path, monkeypatch):
|
|||
conn.close()
|
||||
|
||||
|
||||
@pytest.mark.requires_wal
|
||||
def test_wal_checkpoint_truncates_wal_file(tmp_path, monkeypatch):
|
||||
"""End-to-end: the checkpoint actually truncates the -wal sidecar."""
|
||||
db_path = tmp_path / "kanban.db"
|
||||
|
|
|
|||
77
tests/test_conftest_wal_gate.py
Normal file
77
tests/test_conftest_wal_gate.py
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
"""The conftest WAL gate must agree with hermes_state, and must not import it.
|
||||
|
||||
``tests/conftest.py::_wal_is_usable`` duplicates the SQLite WAL-reset version
|
||||
predicate instead of importing ``hermes_state``. That is deliberate: importing
|
||||
``hermes_state`` during collection caches ``DEFAULT_DB_PATH`` from the real
|
||||
``~/.hermes`` before the per-test ``HERMES_HOME`` redirect, which makes tests
|
||||
read the developer's live production database.
|
||||
|
||||
Duplication needs a guard, so these tests pin the two implementations in
|
||||
agreement across the documented upstream boundaries.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
|
||||
from hermes_state import is_sqlite_wal_reset_vulnerable
|
||||
from tests.conftest import _wal_is_usable
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"version_info",
|
||||
[
|
||||
(3, 6, 23), # pre-WAL
|
||||
(3, 7, 0), # first WAL release — vulnerable
|
||||
(3, 44, 5), # below the 3.44 backport
|
||||
(3, 44, 6), # 3.44 backport — fixed
|
||||
(3, 45, 0), # next minor, back to vulnerable
|
||||
(3, 50, 4), # the version that exposed this (repo .venv)
|
||||
(3, 50, 6), # below the 3.50 backport
|
||||
(3, 50, 7), # 3.50 backport — fixed
|
||||
(3, 51, 2), # last vulnerable
|
||||
(3, 51, 3), # fixed upstream
|
||||
(3, 53, 1), # the managed runtime
|
||||
],
|
||||
)
|
||||
def test_conftest_gate_agrees_with_hermes_state(version_info, monkeypatch):
|
||||
"""``_wai_is_usable`` must be the exact inverse of the canonical predicate."""
|
||||
monkeypatch.setattr(sqlite3, "sqlite_version_info", version_info)
|
||||
assert _wal_is_usable() is not is_sqlite_wal_reset_vulnerable(version_info), (
|
||||
f"conftest gate and hermes_state disagree for SQLite {version_info}"
|
||||
)
|
||||
|
||||
|
||||
def test_conftest_does_not_import_hermes_state_at_collection():
|
||||
"""The gate must stay import-free of hermes_state.
|
||||
|
||||
Importing it during collection caches DEFAULT_DB_PATH from the real
|
||||
~/.hermes, so tests read live production sessions instead of a tempdir.
|
||||
Reading the source is not an option here (banned), so assert on behavior:
|
||||
the gate must work with ``hermes_state`` absent from ``sys.modules`` and
|
||||
blocked from being imported.
|
||||
"""
|
||||
import builtins
|
||||
import sys
|
||||
|
||||
real_import = builtins.__import__
|
||||
blocked: list[str] = []
|
||||
|
||||
def guard(name, *args, **kwargs):
|
||||
if name == "hermes_state" or name.startswith("hermes_state."):
|
||||
blocked.append(name)
|
||||
raise AssertionError(
|
||||
"conftest._wal_is_usable imported hermes_state — this caches "
|
||||
"DEFAULT_DB_PATH from the real ~/.hermes during collection"
|
||||
)
|
||||
return real_import(name, *args, **kwargs)
|
||||
|
||||
saved = sys.modules.pop("hermes_state", None)
|
||||
builtins.__import__ = guard
|
||||
try:
|
||||
_wal_is_usable() # must not raise
|
||||
finally:
|
||||
builtins.__import__ = real_import
|
||||
if saved is not None:
|
||||
sys.modules["hermes_state"] = saved
|
||||
assert not blocked
|
||||
Loading…
Add table
Add a link
Reference in a new issue