_fresh_modules() in test_vision_routing_31179.py deleted agent.image_routing
(+siblings) from sys.modules and never restored the ORIGINAL modules — the
reloaded copies leaked. Any later same-process test holding function refs to
the original module (test_image_routing.py) then patched the reloaded copy
via mock.patch string targets, making patches invisible: order-dependent
failures (repro: 31179 file first → 2 failures; reversed → green).
Autouse fixture now snapshots the affected sys.modules entries and restores
them on teardown. Both orders + solo runs verified green.
Masked by the per-file-isolated canonical runner; bites anyone running
pytest tests/agent/ directly.
Companion to the cherry-picked #74158 fix (non-streaming path):
- gateway/run.py: remove the identical history-dedup filter from
_deliver_media_from_response — the post-stream rescan is explicit-only
by design (#20834), so every MEDIA tag it finds is a deliberate
attachment request; drop the now-unused history_media_paths parameter
and its call-site plumbing.
- gateway/platforms/base.py: log suppressed bare local file paths on the
surviving local_files history dedup (#73771 observability ask).
- tests: focused regression file covering explicit resend delivery on
both lanes, current-turn tool-echo poisoning, surviving bare-path
dedup + its log line, and the upstream auto-append dedup invariant.
Fixes#73771
Follow-up to the salvaged success-path removal: installs that already
repaired (or predate the cleanup) still carry leaked ~1 GB parked venvs.
When the runtime probes safe, reclaim aged (>1h) stale markers next to
the live venv — age-gated to avoid racing an in-flight sibling repair,
boundary-checked via _remove_tree so symlinked names can't escape the
checkout. Also drop the now-stale 'before removing the parked venv'
user guidance in update_cmd.
Tests: success-path removal, safe-path sweep (aged removed, fresh kept).
- apply_database_pragmas: journal_mode ownership stays with
apply_wal_with_fallback/resolve_journal_mode (single guarded owner);
the helper now only applies wal_autocheckpoint / journal_size_limit,
via load_config_readonly (hot-path safe).
- Silent-refusal WAL success path re-applies the macOS
checkpoint_fullfsync barrier and synchronous=FULL enforcement.
- Test doubles updated for connect_tracked's factory kwarg and the
WAL-reset vulnerability gate (fixed-SQLite assumption made explicit).
Config-driven `journal_mode`, `wal_autocheckpoint`, and `journal_size_limit`
are now honored in `SessionDB` init. Users running SQLite on NFS/SMB or
with custom tuning had no supported config surface; values were hardcoded
at connection time.
What
- New `apply_database_pragmas()` in `hermes_state.py`
- Reads nested `database:` keys from `config.yaml` via existing `cfg_get`/`load_config`
- Called after `apply_wal_with_fallback()` in `SessionDB._connect_and_init()`
Fix
- Adds optional PRAGMA switches for journal_mode, wal_autocheckpoint, journal_size_limit
- On Darwin; Windows keeps DELETE unless config explicitly requests WAL
Runtime Proof
$ /opt/homebrew/bin/pytest tests/test_hermes_state.py::TestApplyDatabasePragmas -q
3 passed in 0.72s
Regression Checks
- Full `tests/test_hermes_state.py`: 306 passed in 11.32s
Salvages #55322 (ZFS 'disk i/o error' marker) without regressing the
Bug D transient-EIO protection from 5c49cd0ed0. 'disk i/o error' is
ambiguous: deterministic on ZFS/APFS-CoW SHM corruption (#55305,
#71498) but often a one-shot transient (page-cache pressure, lock
contention). A blind marker match re-introduced the mixed-journal-mode
corruption pattern; a blind re-raise wedged state.db on ZFS.
Disambiguate: retry the WAL pragma twice with a short backoff. A
transient EIO clears and WAL proceeds; a deterministic failure keeps
raising and falls through to the guarded DELETE fallback (still
refusing to downgrade a DB whose on-disk header reports WAL).
Tests: transient-EIO recovery, persistent-EIO fallback, and the
never-downgrade-WAL-on-disk guard.
The WAL→DELETE fallback on WAL-incompatible filesystems (NFS / SMB / FUSE /
the AgentFS NFS overlay) was logged at WARNING, treating a real loss of
concurrency — under the kanban dispatcher + workers a write blocks readers,
surfacing as SQLITE_BUSY — as if it were cosmetic. Escalate the deduplicated
fallback log to ERROR so the degradation is observable, not silent.
Add an opt-in require_wal=True to apply_wal_with_fallback that raises a typed
WalUnsupportedError (subclass of sqlite3.OperationalError, so existing DB-init
handlers still catch it) instead of degrading to DELETE, for callers that
mandate WAL concurrency. All four current callers keep the default
require_wal=False so NFS-homed installs keep working unchanged.
Tests: 4 new require_wal cases; WARNING→ERROR assertion updates in both
test_hermes_state_wal_fallback.py and test_kanban_db.py.
apply_wal_with_fallback() only detected WAL-incompatible filesystems via a
RAISED OperationalError matched against _WAL_INCOMPAT_MARKERS. But macOS NFS,
SMB/CIFS, and overlay filesystems (e.g. AgentFS's NFS-backed mount) refuse the
WAL switch WITHOUT raising: `PRAGMA journal_mode=WAL` returns the still-effective
mode ('delete') and no exception. The code then ran `return "wal"`
unconditionally, so it:
1. returned a false "wal" while the DB was actually in DELETE mode, and
2. never called _log_wal_fallback_once, so the operator got ZERO signal that
concurrency had silently degraded (reader-blocks-writer).
state.db and kanban.db share this path, so a session/kanban board DB on a
network or overlay filesystem ran in DELETE with no diagnostic.
Fix: read the row `PRAGMA journal_mode=WAL` returns and verify it is actually
'wal' instead of assuming success; on a silent no-op, emit the existing
fallback WARNING and return the true mode. The raise-based path is unchanged.
Reproduced on a real AgentFS NFS overlay (PRAGMA journal_mode=WAL returned
('delete',) with no OperationalError). Adds a regression test for the
silent-no-op shape; the 16 existing WAL-fallback tests are unchanged.
Use database.journal_mode as the sole non-secret operator setting, preserve the vulnerable-SQLite safety gate and existing WAL databases, validate explicit DELETE results, document the active config path, and cover real SQLite openers with behavioral tests.
Add HERMES_JOURNAL_MODE env / database.journal_mode config for
virtiofs/NFS/SMB where WAL is not crash-safe. Route 5 bypass openers
through apply_wal_with_fallback so a single setting covers every .db
(#68545).
Two relay-lane bugs from live Discord staging testing (2026-07-29):
1. TTS audio never attached over relay (any platform, any lane).
_history_media_paths_for_session excluded only the trailing assistant
entry from the persisted transcript when building the delivered-media
dedup set. The agent persists rows as it produces them, so THIS turn's
text_to_speech tool result (media_tag JSON) was already in the
transcript at delivery time — the fresh TTS path deduped against
itself and extract_media's attachment was silently stripped
(response_delivery_dropped for a MEDIA-tag-only reply; fly logs show
the exact signature). Fix: exclude everything from the last USER
message onward (the current turn); prior-turn dedup unchanged.
Affects every platform adapter (native + relay) on the non-streaming
delivery path — the streaming path passes explicit history and was
unaffected.
2. Connector-auto-created threads never got the LLM session-title
rename. The title fires on the FIRST exchange, whose source is the
PARENT channel event — the thread didn't exist at ingest, so the
Phase 4 auto-thread markers can't be present and
_is_discord_auto_thread_lane never matches on the relay title turn
(initial titles worked; semantic renames never happened; staging
telemetry shows zero thread_rename ops ever sent). Fix: consume the
connector's new send-result feedback (paired gateway-gateway PR —
contract §SendResult thread_id/auto_thread_name, additive):
RelayAdapter.send() caches (thread_id, initial_name) per chat
(bounded 256), run.py's title-callback registration + rename lane
read it back and pass initial_name as only_if_current_name so the
human-rename-wins guard holds on the relay lane too. Native marker
path unchanged; connectors that don't stamp the fields degrade to
exactly the old behavior.
Tests: 3 new (send-result feedback capture, absence, bound) in
test_relay_threads.py; 3 new in test_history_media_current_turn.py
(current-turn TTS not deduped, prior-turn still deduped, no-user-row
fallback). Relay suite 144 passed.
A shared state.db is legitimately held for multi-second stretches by
sibling Hermes processes: VACUUM after auto-prune, the TRUNCATE WAL
checkpoint at close on a large WAL, offline recovery, or an older
still-running process whose FTS maintenance predates the bounded-merge
protocol (every `hermes update` leaves mixed-version processes sharing
the DB until the old ones exit).
The old retry budget was attempt-counted: 15 attempts x 20-150ms jitter
gives up after ~1.3s of waiting. Any hold longer than that surfaced as:
- append_message failing -> the conversation loop aborts the turn as
session_persistence_failed ('No reply: the turn was stopped because
session storage could not be written') on a perfectly healthy store;
- SessionDB() open failing -> the CLI disables persistence for the
entire run ('Failed to initialize SessionDB ... database is locked').
Both observed in production logs on 2026-07-29 (10.8 GB state.db, 9
concurrent hermes processes, three of them pre-dating the bounded-merge
fix pull).
Changes:
- _execute_write patience is now TIME-based with two budgets: routine
writes wait up to 20s; transcript-critical writes (append_message,
session-row creation — the ones whose failure aborts a user turn)
wait up to 60s. Jitter stays 20-150ms for the first 2s, then backs
off to 250ms-1s so a long hold isn't hammered with BEGIN IMMEDIATE.
- Exhausted patience raises an error that names the actual cause
(another process held the write lock; the database is healthy)
instead of a bare 'database is locked' that reads like disk damage —
and the turn-abort explainer inherits that clarity.
- SessionDB open now applies the same jittered patience to the
locked/busy class around connect+schema-init, instead of failing the
whole open (and disabling persistence for the run) on the first 1s
timeout. Non-lock errors, including the malformed-schema repair
class, propagate immediately as before.
Fixes#74478
Telegram clients split messages above 4096 chars into multiple updates. A
long '/queue <prompt>' paste arrives as a COMMAND chunk near the limit plus
plain TEXT continuation chunk(s). _handle_command dispatched the command
chunk immediately, so the continuation landed as a separate plain message
that interrupted the running agent instead of being queued.
Near-limit (>= _SPLIT_THRESHOLD) command chunks now route through the same
text-batching pipeline used for split plain-text messages, merging the
continuation before dispatch. Short commands (/stop, /approve, ...) keep the
immediate path and are never delayed.
repair_vulnerable_runtime() hardcoded <checkout>/venv as the live venv,
so uv-default/dev checkouts installed into .venv got 'not-applicable' on
every hermes update — no repair path ever fired, leaving state.db-class
DBs on journal_mode=DELETE forever (measured 26 ms + ~5.5 fsyncs per
append vs ~0.01 ms under WAL, ~2,600x) while the WAL fallback warning
falsely promised hermes update would repair the runtime.
- _default_live_venv(): target venv/ when it has an interpreter (managed
layout precedence), fall back to .venv/, keep not-applicable when
neither exists. Explicit venv_dir arg unchanged; all staging/smoke/
cutover/rollback machinery untouched.
- Rebuilt against the pruned test suite (main's test-prune waves 1+2
rewrote test_managed_uv.py, so this reapplies cleanly): 3 new
TestDefaultLiveVenv tests + repair neutralized in the 6 unit tests
whose subject is uv install/self-update mechanics — with .venv now
probed for real, CI's own vulnerable .venv made the unmocked repair
hook fire inside those tests and re-invoke _install_uv.
33/33 tests green on the pruned suite.
The poll this page's pairing block rode was retired hours earlier by
f8e07a332, which moved Messaging onto platforms.changed. That signal is the
mtime of gateway_state.json — where the gateway persists connect/disconnect
health — and a new pairing request moves none of it. So on an event-capable
backend a pending row stayed invisible until something unrelated
reconnected, and the count badge with it.
Adds pairing.changed to the change watcher, signed off the pending/approved
ledgers across the global store and every profile's own. _rate_limits.json
is deliberately excluded: it moves on every unauthorized DM, including ones
that produce no new row, so signalling on it would refetch for nothing.
The page now refreshes platforms and pairing on their own signals rather
than one combined call, and the legacy visible-tab poll (older backends)
covers both.
The gateway keeps one PairingStore per served profile, but every
`/api/pairing` endpoint built the global one. An operator managing a named
profile saw the wrong pending list, and approving wrote a grant into a
whitelist their running gateway never consults — the user stays locked out
while the UI shows them as approved.
`_pairing_store(profile)` now resolves per profile and validates the name
(400/404 on an unknown one). No `_profile_scope` needed: PairingStore
resolves the profile's home itself, so nothing process-global is swapped
across an await.
Both GUIs had to change to match. The listing rides the query param — for
the dashboard that meant deleting `pairing` from the "machine-global, must
NOT be rewritten" exclusion list, a comment this change makes false. The
mutating endpoints read the profile off the BODY, which no query-param
rewrite reaches, so approve/revoke send it explicitly on both surfaces.
The per-profile pairing isolation added self._dir and scoped every
per-file path helper (_pending_path, _approved_path) to it, but
_all_platforms still enumerated the module-global PAIRING_DIR. For a
profile-scoped PairingStore, list_approved/list_pending/clear_pending
therefore operated on the GLOBAL platform set while loading each
platform's file from the PROFILE dir — so list_approved() returned []
for a user that is_approved() confirmed as approved, a silent divergence
between the authz surface and the list/inspect/clear surface.
Route discovery through self._dir. Byte-identical for the global store
(self._dir == PAIRING_DIR when no profile is set); only the buggy
profile-scoped case changes. self._dir is guaranteed to exist (__init__
mkdirs it).
- hermes_cli/web_routers/sessions.py: 14 routes across 3 routers
(list_router, search_router, manage_router) mounted at the three original
registration points so global route order is preserved exactly.
- hermes_cli/web_routers/mcp.py: 11 routes; OAuth flow registry
(_mcp_oauth_flows/lock/cap) stays in web_server, reached via new
web_deps.LateState live proxies so tests mutating web_server._mcp_oauth_flows
keep working.
- hermes_cli/web_routers/skills.py: 12 routes across hub_router + router
(two original registration points straddle the profiles router include).
- hermes_cli/web_routers/tools.py: 12 routes; toolset/terminal catalogs stay
in web_server (some are defined after the mount point), reached via LateState.
- web_deps.py: add LateState — operation-time proxy for web_server-owned
module state (getattr/item/iter/len/contains/context-manager/comparisons).
- Handler bodies byte-identical; legacy re-exports keep
web_server.<handler> importable for tests.
- Verified: ordered route table (method, path) identical to pre-refactor app
(291 routes); import smoke; ruff; windows-footguns clean.
- test_web_server_sessiondb_eventloop.py: structural AST scan now reads both
web_server.py and web_routers/sessions.py (handlers moved; helpers stayed).
Follow-up hardening on the request-id grant path.
approve_request took the same lockout treatment as approve_code: gated by
it, and recording a miss toward it. But the two paths defend different
things. The lockout exists to stop guessing at the 8-char code space over a
messaging channel; a request id is only ever obtained by an admin already
authenticated to the store, so a miss means the row they clicked went stale.
Counting those let a handful of clicks on a stale list lock the operator out
of `hermes pairing approve` for an hour — the GUI DoSing the CLI.
Also drops the `code`/`code_hash_prefix` compat fields from list_pending.
The hash prefix is what admin surfaces mistook for an approvable code in the
first place, and re-exporting the request id under the old `code` key just
preserves the ambiguity; both consumers in the tree read `request_id` now.
The 16-hex sniffing that had been copy-pasted into the CLI and the endpoint
(where a chained conditional consulted it against the wrong field) moves to
one owner, PairingStore.looks_like_request_id.
The endpoint no longer reports a 429 on the request-id path, where lockout
can't apply — a stale id surfaced as a bogus "locked out" while the platform
sat locked for something else entirely.
approve_code()'s success path never cleared _failures:{platform}. The
counter is incremented on every non-matching code, persisted in
_rate_limits.json, and only ever reset to 0 when it reaches
MAX_FAILED_ATTEMPTS (firing the lockout). So it counts failures over the
gateway's entire lifetime, not consecutive ones.
An owner who mistypes a pairing code on a handful of separate occasions
— each time immediately retyping it correctly and successfully pairing —
accumulates those isolated typos. A later single fresh typo then hits
MAX_FAILED_ATTEMPTS and locks the whole platform out for an hour, at
which point _is_locked_out gates approve_code and even the *correct*
code is rejected.
Reset the counter on a successful approval, matching standard
brute-force-guard semantics (the counter tracks consecutive failures).
This does not weaken protection: an attacker cannot produce a success
without a valid code, and 5 consecutive wrong attempts still lock out.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three surfaces start updates against one checkout: a terminal
"hermes update", the dashboard's Update button (which spawns that same
command detached), and the desktop's, which hands off to the Tauri
updater. Only the Tauri updater published the in-progress marker, and
only Electron read it -- to gate backend startup, not to stop a second
updater. So a dashboard-spawned update and an installer-driven git
checkout could mutate the same tree concurrently, rewriting source under
a live interpreter.
Claim the same marker from cmd_update rather than adding a second
mechanism: same path, same pid+started_at payload the Rust and Electron
readers already parse. A marker only counts as live when its pid is alive
and it is inside the shared age ceiling, so a crashed updater self-heals
instead of wedging every future update. Release only removes a marker we
still own, leaving a handoff partner's claim intact.
Refusing exits 2, matching the existing concurrent-instance contract the
Tauri updater already recognizes.
Main's 243c9182b1/a16fd675df/7142dc4580 added load_config_readonly
sibling stubs across 38 files; our pruned versions of 11 of those files
kept only the load_config stubs. Re-applied the pairing at every
surviving site (26 patch()/setattr sites) — same return_value/
side_effect as the adjacent load_config stub. 494 tests green across
the 11 files.
hermes-setup.exe bakes its build-time commit into the binary
(BUILD_PIN_COMMIT) and passes it as -Commit on every install-mode run,
including the retry the desktop's "Update didn't finish" screen kicks
off. The repository stage checked that SHA out unconditionally, so an
installer built months earlier rewound a current managed checkout to its
build commit -- 9,160 commits in the reported case -- leaving ancient
source against a current venv. npm then failed on workspaces that did not
exist yet at that commit, and every later update ran against the wrong
tree.
Skip the pin when its target is already an ancestor of HEAD. Fresh clones
have no such ancestry so reproducible/CI pinning is unchanged, and
--force-commit / -ForceCommit still rolls back on purpose.
The #71637 prune fix cut one stage of -w startup, but the base-ref
resolution right after it still ran an uncapped-in-practice
'git fetch origin main' (timeout=30) on every launch — and on a flaky
smart-HTTP connection that fetch intermittently stalled to the full 30s,
then cascaded into step 2's SECOND 30s fetch. Measured: back-to-back
fetches of 0.9s, 1.0s, 63.5s on the same box with healthy TLS (~185ms).
_resolve_worktree_base now:
- skips the fetch entirely when FETCH_HEAD is < 5 min old and the
tracking ref exists (repeat launches pay zero network cost)
- caps the fetch at 5s and falls back to the locally-known tracking
ref (labelled 'cached') on timeout/failure instead of cascading into
a second fetch — genuine staleness stays backstopped by the pre-push
stale-base gate
- caps 'git remote show origin' the same way
Worst case drops ~60s -> ~5s; warm path is ~0.02s (was up to 30.8s).
sync_base=False and the offline HEAD fallback are unchanged.
The change watcher (#73673) missed one always-on-while-mounted timer: the
Messaging page polled /api/messaging/platforms every 6s for connection
status. The gateway already persists platform connect/disconnect/health to
gateway_state.json, so watch that file's mtime and broadcast
platforms.changed (floored to 5s — the gateway also rewrites the file for
in-flight-count bookkeeping), route it through live-sync like its
siblings, and refresh the page on the tick. Older backends keep the
legacy visible-tab poll verbatim.
Finishes the always-on poll sweep for #73618.
Final pass: per_model_threshold_init_ordering, memory_provider_init,
plugin_context_engine_init, api_max_retries_config,
invalid_context_length_warning, tool_call_guardrail_runtime all stub
agent_init config reads that now resolve through load_config_readonly().
Verified by running the complete 59-file suspect list (every test file
that stubs load_config in string or attribute form and intersects the
swapped modules): 3,063 tests, 0 failed.
Second stub shape the first sweep missed: monkeypatch.setattr(config_mod,
"load_config", ...) — attribute-form instead of string-form. Four files
(compression_max_attempts, preflight_compression_cap_e2e,
codex_gpt55_autoraise_notice, proactive_prune_config) stub agent_init
config reads that now go through load_config_readonly(). Swept the whole
tree for the attribute form; remaining hits stub modules that still use
the mutable loader. 1,210 tests green across the 19-file re-sweep.
The readonly swaps mean agent/ modules now read config through
load_config_readonly(); tests that stubbed only load_config stopped
intercepting those reads. Added sibling readonly stubs (same
return_value/side_effect/lambda) at every affected site across 11 test
files, found via a tree-wide sweep of load_config stubs cross-referenced
against the swapped modules. 3,915 tests green across the 38-file
sibling sweep.
agent_init's config reads now go through load_config_readonly(); 8
tests that stubbed only load_config stopped intercepting them. Each of
the 10 patch sites gains a sibling readonly patch with the same
return_value. 461/461 file-local tests green.
Pins the audit findings: normalizer never mutates its input
(api_key_env + camelCase forms), providers-dict round-trips leave a
cached config byte-identical, and the normalized models mapping does
not alias the caller's dict.
The normalizer writes alias keys into the entry it is given
(entry['key_env'] = entry['api_key_env'] and entry[snake] =
entry[camel]) while building its normalized copy. Two of its three
callers — get_compatible_custom_providers and
providers_dict_to_custom_providers — pass live sub-dicts straight
from load_config_readonly()'s shared cache (only
_custom_provider_entry_to_provider_config defends with dict(entry)).
A config written with the documented camelCase / api_key_env aliases
therefore gets its cached copy polluted with injected duplicate keys,
violating the cache's explicit no-mutation contract; every later
load_config() deepcopy inherits the duplicates, and any
save_config(load_config()) flow (setup wizard, dashboard writes,
model persist) writes them back to config.yaml. The aux-client TLS
resolution runs this on every auxiliary client build, so the
mutation also happens unlocked on worker threads against a shared
object.
Shallow-copy the entry up front; the function's return value is a
separately-built dict, so behavior is otherwise unchanged.
The 1.0s sleep between sequential tool calls has been present since the
initial commit with no documented rationale. It sleeps between local
tool executions — the next LLM request goes out only after the whole
batch — so it rate-limits nothing, and the parallel read-only path
already runs with no delay. Every multi-tool turn pays (N-1) seconds
of dead time. Remove the sleep, the internal tool_delay plumbing, and
dead test assignments. AIAgent.__init__ keeps tool_delay as a
deprecated no-op keyword for one release so existing programmatic
callers construct cleanly; passing it emits a DeprecationWarning.
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
