Follow-up to the partial-recovery salvage. _copy_source_bundle() raw-copied
the source state.db and its -wal/-shm sidecars with no live-connection check.
Copying a database file is an open()/close() on it, and close() cancels every
POSIX advisory lock the process holds on that file -- including a running
VACUUM's EXCLUSIVE lock (fe431651c5). hermes_state._backup_db_file already
refuses that situation; this path did not, leaving two policies for one
hazard. Verified against the merged guard: with a tracked connection open,
_backup_db_file returned None (refused) while _copy_source_bundle copied
anyway.
Recovery normally runs as its own short-lived CLI process against an
offline/quarantined file, so this should never fire in practice. It is a
consistency fix, not a live corruption path -- but it is exactly the drift
that reintroduces the bug later.
Regression test verified by sabotage: removing the guard fails it.
Addresses review findings on the previous commits. Three of them were real
defects I reproduced against my own head before fixing.
1. Check/use race (BLOCKING). read_header_bytes_preopen() checked
has_live_connection() under _live_lock, released it, then did the raw
open/read/close outside the lock; connect_tracked() opened before
registering. A thread could pass the "nothing is live" check, another
could open a connection and BEGIN IMMEDIATE, and the first thread's
close() then cancelled its POSIX locks -- the exact bug this guard
exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED).
_live_lock now spans all three lifecycle transitions: open+register,
unregister+close, and check+open+read+close.
2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's
read-only path opens file:/…/state.db?mode=ro; that string was fed to
Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of
the real Path could match, so read-only connections were invisible to
the guard and their locks cancellable. Reproduced with no forced
scheduling. Keys now come from PRAGMA database_list (canonical path),
with an explicit tracking_path override.
3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception
and retried an untracked plain connect, so any error silently disabled
the guard. Now only ImportError (scaffold installs without hermes_cli)
falls back; real failures propagate.
4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db()
and _backup_db_file() raw-read live databases; they now REFUSE when a
connection is live rather than warning. Losing a forensic copy beats
corrupting the database being rescued.
Custom factories are no longer rejected (that broke legitimate callers) nor
silently untracked -- the tracking close() is mixed into whatever factory is
in play, including when an opener substitutes its own after the fact.
WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was
confounded -- the clean WAL result came from 3.53.1, which carries both the
WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the
bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and
DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the
WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL
until a fixed runtime ships is the conservative call, and the WAL policy does
not belong in this root-cause fix.
Six sabotage runs confirm each new test fails when its defect is reinstated
(including two that initially did NOT -- the race test was rewritten to pause
inside the byte read, and a separate test added for the opener-substituted
factory path). 1124 targeted tests green.
Completeness pass over the previous commit.
Registry leak (would have silently disabled the guard): the first version
incremented on open but decremented only in SessionDB.close(). kanban's
connect() hands raw connections to callers who close them directly (4 sites),
so its counter only ever went up — after enough kanban operations every
byte-probe on that path would be refused forever, disabling zeroed-file and
header detection. Replaced manual track/untrack with a TrackedConnection
subclass that untracks in close(), the one method every close path goes
through. Verified across plain close, contextlib.closing, double close,
nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not
a close) correctly stays tracked.
Also: a caller-supplied factory now wins instead of raising TypeError on a
duplicate kwarg, since tracking is an optimisation for the probe guard, not a
precondition for opening the database.
Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert.
Audit notes:
- DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains
correct there; those users are protected by the raw-read fix, not by the
journal mode.
- The remaining whole-file reads are on genuinely offline artifacts:
_backup_db_file (DB won't open; bytes preserved for forensics),
_backup_corrupt_db (quarantine path, now warns if a connection is live),
and backup verification of snapshots. backup._safe_copy_db already uses
the SQLite backup API rather than a byte copy.
- The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped,
not process-scoped), so this is a Linux/macOS correctness fix; the change is
platform-neutral and safe on Windows.
Sibling tests updated: session recovery no longer expects a DELETE-mode
recovered DB, and the kanban WAL-fallback test patches the connect site that
actually runs now.
`hermes sessions optimize` could corrupt state.db. Root cause is Hermes,
not the SQLite WAL-reset bug (#69784).
close() on ANY file descriptor for a SQLite database cancels every POSIX
advisory lock the process holds on that file, including a running VACUUM's
EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed
live databases in several hot paths: the zeroed-state.db detector runs on every
SessionDB construction (and the gateway builds those constantly), and kanban's
post-commit invariant check ran after every COMMIT. While VACUUM rewrote the
file, those probes dropped its lock and let other processes write into it.
A/B against the real code, only variable being the raw read:
SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode
raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt
no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt
SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt.
After this change: 0/4 corrupt, 0 vacuum errors.
Because the upgraded runtime corrupts too, replacing the embedded SQLite does
not fix this class; and because DELETE is where it reproduces, #70055's
"force DELETE on vulnerable builds" mitigation steered users into the failing
mode. That gate is reverted here: vulnerable builds get WAL again and still
warn so operators can upgrade.
- add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the
existing connection instead of open()+seek(28); byte-level probes are
restricted to before any connection exists and refused once one is live,
with an explicit force= escape for offline artifacts (snapshots, archives)
- track live connections in SessionDB and kanban's connect so that guard is
enforced rather than merely documented
- kanban's torn-extend check now only applies under a rollback journal; in WAL
a committed page may still legitimately sit in the -wal file
- revert the force-DELETE WAL gate and update the tests that pinned it
Regression tests assert the behavioural contract (an external process stays
locked out across Hermes' inspection calls) and were verified to fail when the
old raw-open behaviour is restored.
The background write guard decided ownership from `isinstance(usage_rec, dict)`,
so a local skill with NO usage record passed. That successful write called
bump_patch(), which created a `created_by: null` record — and the identical
write was refused from then on. "Allowed exactly once, then never" is a race
with our own bookkeeping, not a policy. Reproduced on main: patch #1 succeeds,
patch #2 with the same arguments is refused.
Option B from the issue. Option A (split `session_review` from
`scheduled_curator` and let the session fork patch user-owned skills it
consulted) would widen autonomous write permission onto skills the user owns
with no user present to consent — wrong direction for a no-user-present actor.
- skill_manager_tool: missing and explicit-null records now resolve
IDENTICALLY, both fail closed. The refusal names the reason and points at
`hermes curator adopt <name>`.
- background_review: both review prompts told the reviewer to patch any skill
consulted in the session and claimed pinned skills could be improved, while
enforcement refused both. Prompts now list pinned, external, and user-owned
skills as protected, and tell the reviewer to RECOMMEND adoption instead of
attempting a write that will be refused.
- skill_usage: document that `created_by` is a curator-management policy flag,
not a provenance claim, and add `is_curator_managed()` so call sites read as
the question they ask. Field name retained — it is on disk in every
`.usage.json` and renaming would strand those records.
- curator CLI: `hermes curator list-unmanaged` itemizes unmanaged skills with
the reason each is unmanaged (completes the #67139 spec).
Foreground writes are untouched: a user-directed edit to a user-owned skill
still works, including on pinned skills.
Sibling tests: 9 failures in test_skill_manager_tool.py were fixtures that
created record-less skills to exercise OTHER guards (consolidation-delete,
read-before-write) and relied on ownership falling through. Fixed at the
fixture, since the real curator only ever operates on managed sediment. One
test asserted the old "manually authored" wording; rewritten to assert the
behavior contract instead of the string.
Validation: 274 targeted tests + all 7 background-review files (60 tests) pass.
E2E on a temp HERMES_HOME (30 checks) covers the flip, foreground writes,
adoption unblocking, pin semantics, prompt/enforcement parity, and the new verb.
Each new test sabotage-verified: revert the fix, confirm it goes red.
Fixes#67140
The residual Windows-on-ARM integrity-gate fix added GetNativeSystemInfo as a
second "OS-native" probe behind IsWow64Process2. Microsoft documents the
opposite behavior: "the API GetNativeSystemInfo also returns emulated
processor details when run from an app under emulation."
On the exact hosts the probe was added to rescue -- an x64 hermes-setup.exe
emulated on an ARM64 Surface -- it therefore reports AMD64, making it a
duplicate of the PROCESSOR_ARCHITECTURE rung already below it rather than a
fallback for it. Its test only passed because the kernel32 fake was hand-fed
ARM64, asserting behavior real Windows does not exhibit.
Replace it with GetMachineTypeAttributes, which answers the question the gate
actually asks -- "can this host load a PE of machine X?" -- instead of
inferring it from an architecture name. It is also the only documented API
that reports AMD64-on-ARM64 emulation support. The gate prefers it and falls
back to the existing name-based mapping on pre-Windows-11 hosts.
The load-bearing half of the previous fix (typing GetCurrentProcess as HANDLE
so IsWow64Process2 stops failing ERROR_INVALID_HANDLE) is unchanged.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: Teknium <teknium1@users.noreply.github.com>
`hermes curator status` reported only the skills it manages, staying silent
about curation-eligible skills it can never touch. On a 237-skill library that
meant 112 skills were invisible to every automatic transition with no signal
anywhere — the curator looked broken when it was working as designed.
A skill becomes curator-managed only when `created_by: agent` lands on its
usage record, and only the background review fork writes that marker. Two
populations therefore never qualify: records written before the marker existed
(no key at all, authorship unknowable) and every foreground
`skill_manage(create)` (unset by design — those skills belong to the user).
- skill_usage: `list_unmanaged_skill_names()` / `unmanaged_report()` enumerate
the blind spot, tagging each row with `has_provenance_key` so the two causes
are distinguishable. `adopt_skill()` writes the marker on user declaration
and refuses bundled, hub-installed, external, and protected built-ins.
Adoption never resets the inactivity clock.
- curator CLI: status prints an `unmanaged (no provenance marker)` block on
BOTH the managed and no-managed-skills paths; new `adopt` verb takes names or
`--all-unmanaged`, with `--dry-run` and a confirmation prompt on bulk.
- skills_sync: `_backfill_optional_provenance()` matched candidates by
repo-derived path only, so a skill installed at `mlops/chroma` that upstream
later moved to `mlops/vector-databases/chroma` was skipped forever and
`hermes skills repair-optional` could not fix it. Falls back to an
unambiguous name match, still gated on identical content, and records the
ACTUAL install path.
Provenance stays a declaration, never an inference: a high patch count proves
the agent MAINTAINS a skill, not that it authored one, since Hermes edits
user-written skills on the user's behalf routinely. An "looks agent-made"
heuristic would eventually archive hand-written work.
Validation: 347 targeted tests pass. Each new test verified via sabotage run
(revert the fix, confirm the test goes red) — one initially passed for the
wrong reason because the fixture pinned `prune_builtins` off, masking the guard
under test; fixed to force the shipped default on.
The retry loop skipped only the current version, so on a uv whose download
catalog is stale it walked backwards through older patches. In #71250 the
newest indexed 3.11 is 3.11.14 — the installed version — so all five retries
went to 3.11.13..3.11.9, each a real download+install+probe+delete cycle that
the existing downgrade guard was always going to reject, before failing anyway.
Only newer patches can carry the SQLite fix. Skipping <= current makes the
stale-catalog case cost one attempt instead of six, and still finds 3.11.15
on the first retry when the catalog knows about it.
Fixes#71250.
`hermes update` could not repair the embedded Python runtime's
vulnerable SQLite (WAL-reset bug range 3.7.0-3.51.2, except backports
3.50.7/3.44.6) on installs where `uv python install <minor>` (bare
request, e.g. "3.11") resolves to an older cached/indexed patch that
still links a vulnerable SQLite build, even though a newer
non-vulnerable patch on the same minor line is available and known to
uv. The smoke test correctly rejected the vulnerable candidate every
time, but the provisioner gave up immediately after one attempt,
leaving the repair permanently stuck in the same failing loop on every
`hermes update` run.
Implements option D from the issue (query the index, retry with an
explicit newer patch), which the reporter identified as cleanest:
- New `_list_available_patches()`: runs `uv python list <minor>
--all-versions --only-downloads --output-format json --no-config`,
filters to cpython/default-variant entries (excluding pypy/graalpy),
and returns known patch versions newest-first. Fails safe (returns
[]) on any network/parse error.
- Refactored the single install+find+probe cycle out of
`_install_safe_python_generation()` into `_attempt_install_generation()`,
reusable per attempt with its own generation directory (so a
rejected candidate's files are fully cleaned up before the next
attempt, matching the existing --reinstall semantics).
- `_install_safe_python_generation()` still tries the bare minor-line
request first (preserves the original comment's rationale: for a
given exact patch, python-build-standalone may have no artifact with
fixed SQLite at all). If that resolves vulnerable, it now queries
`_list_available_patches()` and retries with explicit newer patches,
newest-first, bounded to `_MAX_PATCH_RETRIES` (5) attempts -- each
attempt is a real download+install+probe cycle, so the cap keeps
worst-case repair time bounded.
Sanity-checked `_list_available_patches()` against the real `uv`
binary (0.11.7) in this environment: correctly parses live `uv python
list --all-versions --output-format json 3.11` output, returns 14
patches sorted newest-first starting at 3.11.15.
9/9 new tests pass (retry succeeds with a newer patch, exhausts
gracefully when every known patch is vulnerable, empty patch list
degrades to None without crashing, retry count is bounded, plus direct
JSON-parsing unit tests for realistic/malformed/empty uv output);
47/47 in the full tests/hermes_cli/test_managed_uv.py file (including
the 3 pre-existing tests for the original bare-minor success path,
confirming no regression there).
A failed `hermes sessions repair` printed "keep state.db and the backup"
and stopped, so a user whose sessions had vanished had no way to discover
that a non-destructive recovery path exists — the reported dead end.
The failure branch now names the next command, read-only step first, seeded
with the backup path it just preserved. Covered by a CLI-surface regression
test that drives the real subprocess against an unrepairable database.
* fix(update): bind IsWow64Process2 HANDLE and fall back to GetNativeSystemInfo
The #71218 OS-native probe still rejected correct ARM64 Desktop rebuilds on
Windows-on-ARM when ctypes truncated GetCurrentProcess()'s pseudo-handle and
IsWow64Process2 failed with ERROR_INVALID_HANDLE, falling through to the
lying PROCESSOR_ARCHITECTURE=AMD64 value. Type the HANDLE correctly and use
GetNativeSystemInfo before the env-var fallback.
* test(update): cover WoA IsWow64Process2 handle failure and system-info fallback
Pin the residual #71218 shape where IsWow64Process2 returns FALSE, the env
arch lies as AMD64, and GetNativeSystemInfo must still report ARM64 so the
integrity gate accepts a correctly-built ARM64 Hermes.exe.
Two tests fail deterministically on main depending only on which SQLite the
test interpreter links — nothing about the code under test. Both are green in
isolation and red in the suite / on an older library, the worst diagnostic
shape.
Root cause A — WAL is not always WAL. Hermes refuses journal_mode=WAL on
SQLite builds carrying the upstream WAL-reset corruption bug (3.7.0–3.51.2,
excluding backports 3.50.7 / 3.44.6) and falls back to DELETE. On such a
build NO -wal sidecar is ever created, so
test_wal_checkpoint_truncates_wal_file asserts on a file that cannot exist.
Invisible locally when the repo .venv and the Hermes managed runtime link
different versions (observed: .venv 3.50.4 → DELETE, runtime 3.53.1 → WAL),
so the same test passes for one interpreter and fails for the other.
- tests/conftest.py: add a `requires_wal` marker plus a
pytest_collection_modifyitems hook that skips such tests when the linked
library will fall back to DELETE. The skip reason names the actual
version so it is diagnosable rather than mysterious.
- pyproject.toml: register the marker.
- test_kanban_db_repair.py: mark the -wal-sidecar test.
Root cause B — process-global warn-once dedup. The WAL-fallback warning is
emitted at most once per (process, db_label). Any earlier test in
test_kanban_db.py that opens a kanban.db consumes that one-shot, so
test_connect_falls_back_to_delete_on_locking_protocol sees zero warnings and
fails — but only as part of the file, never alone.
- test_kanban_db.py: clear both dedup sets in the test that asserts on the
warning, with a comment explaining the isolation trap.
The gate deliberately does NOT import hermes_state. That module computes
DEFAULT_DB_PATH from get_hermes_home() at import time, so importing it during
collection — before the per-test _isolate_hermes_home fixture redirects
HERMES_HOME — permanently caches the developer's REAL ~/.hermes/state.db for
the whole session. The first version of this change did exactly that and made
tests read a live 31,881-session production database (test_console_engine
asserted "Total sessions: 2" and got 31881). The version predicate is
duplicated instead, and tests/test_conftest_wal_gate.py pins the two
implementations in agreement across every documented upstream boundary plus
guards against the import coming back.
Verified: on SQLite 3.50.4 the sidecar test SKIPS naming the version; on
3.53.1 it RUNS and passes, so coverage is not lost where WAL works. Clean
main fails exactly these 2 tests under
`scripts/run_tests.sh tests/hermes_cli/ tests/test_hermes_state.py`
(9926 passed, 2 failed); with this change the same scope is green.
Tests: 726 passed across the two kanban files, test_hermes_state.py, and the
new gate tests.
Reviewer egilewski identified two recovery-loss paths:
Path A — quarantine race (hermes_state.py):
SessionDB checks state.db before quarantine_zeroed_state_db() without a
shared cross-process lock, and Path.rename() may replace an existing
destination. Two writable startups can therefore move the first
instance's newly created database over the .zeroed-*.bak, erase the
original damaged-file evidence, and replace the live database with
another empty one.
Fix: add a cross-process lock (msvcrt on Windows, fcntl on POSIX)
around quarantine_zeroed_state_db() with a 5s bounded timeout. Under
the lock: re-check is_zeroed_state_db (another process may have already
quarantined it and created a fresh DB), use a PID-suffixed unique
destination, and non-clobbering rename with counter fallback.
Path B — size-cap pruning gap (hermes_cli/backup.py):
_too_large() runs before failed-database tracking. With keep=1 and a
size cap, an oversized state.db is omitted while failed_dbs stays empty,
so automatic pruning deletes the older complete snapshot that may
contain the only recoverable database.
Fix: track oversized DB files in a new oversized_skipped list (both in
the directory walk and top-level file loop). The manifest now records
oversized_skipped. Pruning is suppressed when failed_dbs or
oversized_skipped is non-empty, preserving the older complete snapshot
as recovery source.
Tests:
- test_concurrent_quarantine_no_clobber: two threads racing on the
same zeroed state.db — verifies quarantine backup survives with
original bytes and live DB is valid.
- test_oversized_db_suppresses_pruning: keep=1 + oversized state.db
verifies the older complete snapshot is not pruned.
All 33 tests pass (3 zeroed_state_db + 25 TestQuickSnapshot + 5
quarantine_forensic_logging).
Hardening for the Windows zeroed-state.db class (#68474):
- Surface critical stdout when pre-update/quick snapshot cannot copy a
present *.db (was log-only; update still looked successful).
- Detect all-NUL SQLite header on SessionDB open, quarantine the bytes,
and open a fresh DB with recovery guidance to state-snapshots.
Does not claim storage-stack root cause.
The #71119 integrity gate rejected CORRECT ARM64 desktop rebuilds on
Windows-on-ARM (#69179 follow-up): platform.machine() reports the PROCESS
architecture, and the update chain runs an x64 hermes-setup.exe / x64
Python under ARM64 emulation, so the gate compared the ARM64 Hermes.exe
against a phantom 'AMD64 host' and aborted the update.
_windows_native_machine() now asks the OS via IsWow64Process2 (whose
nativeMachine is truthful from emulated processes), falling back to
PROCESSOR_ARCHITEW6432/PROCESSOR_ARCHITECTURE (pre-1511 Win10) and then
platform.machine(). All three consumers — the integrity gate, the
packaged-exe arch preference, and backup validation — go through it.
Sabotage-verified: the three new regression tests fail against the old
process-arch behavior and pass with the fix.
Builds on @ms-alan's label fix: the negative figure had a second cause
that relabelling alone leaves in place.
`hermes sessions optimize-storage` reported "reclaimed -3820.1 MB" on a
database that had in fact shrunk 60% (25069 MB -> 9975 MB). Both figures
came from os.path.getsize(). In WAL mode a VACUUM's rewrite lands in the
-wal file, and the checkpoint that folds it back is REFUSED
(SQLITE_BUSY) while any other connection holds a read-mark — e.g. the
live gateway. So the main file still carried its pre-VACUUM size AND kept
growing while the command ran, making the after-figure larger than the
before-figure. The TRUNCATE checkpoint in close() lands after the caller
has already measured and printed.
- hermes_state.py: add SessionDB.logical_size_bytes() — page_count *
page_size, the size the file settles at once the WAL is folded back.
Correct immediately, readers or not; returns None when the connection
is gone so callers fall back to stat().
- hermes_state.py: best-effort TRUNCATE checkpoint after the optimize
VACUUM so the file settles promptly when nothing else holds the DB.
Documented as insufficient alone (busy under a live gateway) so the
stat() approach is not reintroduced.
- hermes_cli/main.py: both `optimize-storage` and `optimize` report via
logical_size_bytes(); fixing only the reported command would have left
the same bug class next door.
- hermes_cli/main.py: lift the contributor's inline label to a
module-level _size_delta_label() shared by both sites, so the "grew by"
wording is consistent and unit-testable without reading source.
The two halves are complementary: page accounting removes the phantom
negative, and "grew by" still covers a genuine increase when concurrent
session writes outweigh what the rebuild freed.
Verified: sabotaging logical_size_bytes() back to stat() fails the new
test with a 22 MB overstatement, reproducing the report. The test asserts
its own precondition (stat() must actually be lagging) so it cannot
silently stop exercising the bug.
Tests: 464 passed (test_hermes_state.py + the new label tests).
Co-authored-by: chenbin <h-chenbin@voyah.com.cn>
The post-update state.db integrity guard called verify_sqlite_integrity()
with max_bytes=0, which disables the size ceiling and forces a full
PRAGMA integrity_check. That pragma walks every b-tree page in the file,
so its cost scales with database size — measured on a real 30 GB state.db:
143.5s with a cold page cache (worse under an update's memory pressure),
with zero output on screen. The update looks hung right after
"✓ Code updated!" and a CPU sits pegged.
Multi-GB session databases are normal for heavy users, so a
size-unbounded check is never an acceptable default on the update path.
- verify_sqlite_integrity(): max_bytes now defaults to
DEFAULT_INTEGRITY_CHECK_MAX_BYTES (2 GiB) instead of 0. max_bytes=0
remains the explicit opt-in for a full scan.
- The oversized path no longer degrades to a header-only check: it adds a
constant-time structural probe (read-only open + schema_version +
sqlite_master read) so the malformed-schema class is still caught, not
just the #68474 zeroed-file signature.
- Drop the explicit max_bytes=0 at the post-update guard and in
copy_db_and_verify() so both inherit the bounded default.
Measured on the reporter's real 30 GB state.db: 143.5s cold → 0.001s,
still valid=True. Corruption detection verified at multi-GB scale for
both classes (zeroed header, malformed schema) — both still fail closed.
Tests: default-is-bounded invariant, oversized probe catches malformed
schema, max_bytes=0 still forces the full check.
Bug-class coverage for both fixes: the full catalogue survives Save, context
lengths are preserved, the key never lands in config.yaml on either write
path, blank clears it, a pre-fix plaintext key migrates while a ${VAR}
template is left alone, two endpoints on one host keep separate credentials,
and an IP-derived name is still a valid POSIX env var.
The two delete tests asserted on the plaintext mirror; they now assert the
same invariants against the credential reference.
The desktop self-update chain (Desktop -> hermes-setup --update ->
hermes update -> hermes desktop --build-only -> relaunch) rebuilds
Hermes.exe on the user's machine and declared success on bare file
EXISTENCE. A truncated PE (corrupt cached Electron zip / interrupted
extraction or rcedit rewrite / full disk) or a wrong-architecture
unpacked tree therefore shipped as the 'updated' app, which Windows
refuses to load with 'This app can't run on your computer'
(此应用无法在你的电脑上运行) — and the previous working build had
already been wiped by before-pack.mjs, leaving nothing to fall back to.
Fix, in three parts:
- hermes_cli/main.py: post-build integrity gate on Windows
(_ensure_desktop_exe_launchable). Parses the PE header of the freshly
built Hermes.exe — MZ/PE magic, section-table completeness vs file
size (catches truncation), and COFF machine vs the host arch (catches
arm64/x64 mixups). On failure it purges the (likely corrupt) cached
Electron zip, invalidates the content-hash build stamp so the
updater's retry-once genuinely re-downloads and rebuilds, restores
the previous build from the .bak tree when one exists (keeping the
corrupt tree as .corrupt for diagnostics), tells the user the update
was aborted and their old version kept, and exits nonzero.
_desktop_packaged_executable also now prefers a host-loadable PE over
pure newest-mtime when multiple win-*-unpacked trees coexist.
- apps/desktop/scripts/before-pack.mjs: on win32, the previous unpacked
tree is preserved as <appOutDir>.bak (only when it holds the product
exe — partial/corrupt trees still get the plain wipe) instead of
being destroyed, providing the rollback material for the gate above.
Non-Windows behavior is unchanged.
- Behavior-contract tests: tests/hermes_cli/test_desktop_exe_integrity.py
(23 tests — synthetic PE fixtures for truncation/non-PE/arch-mismatch,
rollback semantics, and the build-only exit contract) and 6 new vitest
cases in before-pack.test.mjs for the .bak preservation rules.
Progresses #69179
/api/status is the only public liveness route, and its handler loads the
gateway config, probes gateway health, and counts sessions before it can
answer. That work is wrong for a readiness probe: a caller that only needs
to know the process is up pays for a cold plugin import tree.
Add /api/health, which returns process liveness, version, and the auth-gate
shape and touches nothing else.
`_render_distribution_plan` reads the target profile's `.env` to decide
whether a required env var is already set (so it doesn't nag the user),
using `Path.read_text()` with no encoding. Two bugs:
1. `Path.read_text()` defaults to the system locale (cp1251/GBK on Windows),
which raises `UnicodeDecodeError` on any non-ASCII byte. The surrounding
`except OSError` does NOT catch that — `UnicodeDecodeError` is a
`ValueError` — so a mis-encoded `.env` aborts the entire install preview.
2. Even on a UTF-8 locale, a Notepad-added BOM prefixes the first key
(`KEY`), so the very first required env var is mis-reported as
"needs setting" when it is actually present.
`.env` is written as UTF-8 everywhere in the codebase. Read it as
`utf-8-sig` (tolerates the BOM) and also catch `UnicodeDecodeError` so a
genuinely un-decodable file skips the pre-check instead of crashing.
Regression tests: a BOM-prefixed `.env` whose first key must still read as
"set", and an invalid-UTF-8 `.env` that must not abort the preview.
/api/status?profile= now passes pid_path=/path=/expected_home= to the
PID and runtime-status readers; the profile-unification fakes had
zero-arg signatures and raised TypeError. Plain /api/status call shapes
are unchanged (pinned by the existing zero-arg tests in
test_web_server.py).
Follow-up for the salvaged #70498 fix: replace the original PR's
mock-signature churn (28 lambda **kw edits, needed only because it changed
the no-profile call shape) with two targeted regression tests:
- ?profile=<name> must pass the profile's gateway.pid / gateway_state.json
paths and expected_home to the gateway status readers (HOME-anchored
per-profile state under ~/.hermes/profiles/<name>/)
- ?profile=<unknown> must 404 via _resolve_profile_dir
The production change keeps plain /api/status on the exact zero-arg calls,
so every pre-existing test passes unmodified.
The .env sanitizer inferred missing newlines from known KEY= substrings
inside existing values. Plain secrets containing those bytes could therefore
be split into synthetic assignments and rewritten to disk.
Treat each physical line as the only assignment boundary and keep bytes after
the first equals sign opaque for boundary discovery. Preserve safe formatting,
null-byte removal, BOM handling, and normal one-assignment-per-line parsing.
Cover direct loading, dotenv loading, sanitization, writers, and migration
with behavioral regressions.
Fixes#29155
Follow-up for salvaged PR #69141, addressing the last open review point:
cmd_prune() only set orphan_allowlist inside 'if orphans or pre_v2_orphans',
so a zero-orphan preview passed the unrestricted None sentinel down to
prune_checkpoints(), authorizing deletion of any project that became
orphaned between the preview and the rescan — with zero confirmation
calls. The allowlist is now bound unconditionally for every non-force
run (empty preview => empty allowlist); --force keeps None. Adds the
zero-orphan-preview timing regression plus allowlist-identity tests.
Requested by egilewski on #69141: the orphan confirmation flow had no
test coverage at all before this. Exercises hermes_cli.checkpoints.cmd_prune
directly against pre-v2-only and mixed (v2 + pre-v2) fake stores —
decline aborts with nothing deleted, accept deletes both layouts,
--force and --keep-orphans skip the prompt as expected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two compression-tip hydration tests simulated legacy state by emptying
the parent AFTER end_session(compression) — exactly the durable write
the new closed-parent guard refuses. Reordered: empty first, close
second. The tests' actual contract (old id hydrates from the live tip)
is unchanged and still pinned.
The runtime-repair change passes repair_observer= to update_managed_uv/
ensure_uv; the autouse fixture fakes had zero-arg signatures and raised
TypeError through the mock. Sibling test file to the PR's own suite.
Follow-up on the #70186 salvage. The cherry-picked repair pinned the
candidate to the exact current CPython patch (e.g. 3.11.14). Verified
live with uv 0.11.19: every published python-build-standalone artifact
for 3.11.14 links vulnerable SQLite 3.50.4 — even with --reinstall — so
the exact-patch pin made the repair permanently impossible on the
installs that need it most (repair_vulnerable_runtime returned
'failed: could not provision a fixed private Python runtime').
Request the minor line (3.11) instead — the same resolution a fresh
'uv python install' would make, still inside requires-python — and
tighten the drift gate to 'same minor, no downgrade'. E2E-verified
end-to-end on a real vulnerable venv: repair_vulnerable_runtime()
provisioned 3.11.15, built + smoke-tested the sibling venv, cut over,
and reported SQLite 3.50.4 → 3.53.1 with the old venv parked for
rollback.
Follow-up fixes on top of the cherry-picked guard:
- verify_sqlite_integrity(): an oversized (max_bytes-exceeding) database
now still fails on a zeroed/invalid header — previously valid=True was
set before the header check, so a >1GiB zeroed state.db (the exact
#68474 signature at 95MB scale) passed as valid.
- _run_pre_update_backup(): the guard referenced get_hermes_home before a
later function-local 'from hermes_constants import get_hermes_home'
shadowed it → UnboundLocalError swallowed by the snapshot try/except,
silently disabling the post-snapshot check AND the snapshot-id output.
Alias the import explicitly.
- Import _quick_snapshot_root where used (was NameError in all 3 guards).
- tests/hermes_cli/test_state_db_guard.py: real-SQLite E2E coverage —
valid/zeroed/truncated files, oversized header gate, copy+verify
roundtrip, snapshot restore flow, and the live _run_pre_update_backup
path against a temp HERMES_HOME with mid-flight zeroing.
A brew Python upgrade (original report) or an interrupted venv rebuild
(v0.19.0 report in the same thread) can leave certifi importable while
its bundled cacert.pem is missing or a dangling symlink. Every TLS
connection then fails — Feishu/Telegram/WeChat/DingTalk all down —
with an opaque 'Could not find a suitable TLS CA certificate bundle'
from deep inside httpx/requests.
The existing repair infrastructure only probed
`hasattr(certifi, 'contents')`, which PASSES in exactly this failure
state, so neither the early venv self-heal nor `hermes update`'s
import-probe repair ever classified certifi as broken. Extended, not
replaced:
- hermes_cli/_early_recovery.py: the in-process probe now also
validates that certifi.where() exists and is a plausible bundle
(>=1KiB), so the pre-import self-heal repairs it like any other
wiped core package.
- hermes_cli/main.py (_detect_broken_lazy_refresh_imports): the
subprocess probe script used by `hermes update`'s venv repair
applies the same bundle-file check inside the target venv.
- hermes_cli/doctor.py: `hermes doctor` already failed the cert
check; `hermes doctor --fix` now repairs it (pip force-reinstall
certifi + module-cache invalidation + re-verify), covering
brew/manual venvs where no update marker exists. Failures funnel
into the manual-action list with the exact command.
- agent/ssl_guard.py: the startup SSLConfigurationError hint now leads
with `hermes doctor --fix` instead of only the raw pip command.
Fixes#29866
Follow-up on the #69500 salvage: 'launchctl submit' jobs remain
registered with launchd after they exit, so every plist reload leaked
one dead '<label>.reload.<pid>.<ts>' label. The helper script now ends
with 'launchctl remove' of its own transient label, and the recovery
test asserts the self-removal is present.
The deferred launchd reload helper used start_new_session=True to detach
from the gateway's process group. However, setsid(2) alone does NOT move
the child outside the launchd job's process coalition — when launchctl
bootout fires on the gateway label, launchd terminates ALL processes in
that coalition, including the setsid-detached helper, leaving the service
permanently unloaded.
Fix by spawning the helper via launchctl submit, which creates a
transient launchd one-shot job that is wholly independent of the
gateway's coalition. This ensures the helper survives bootout and can
complete the bootstrap+verify cycle.
Also writes a durable pre-bootout marker to the reload log so the
distinction between 'helper never started' and 'helper ran but
bootout/bootstrap failed' can be diagnosed.
Fixes#69098
- Strip the salvaged commit's inline encoding kwargs where main had since
gained its own (process_registry, local env, cua doctor, gateway,
commands, gateway_windows — the latter keeps its locale-aware
_schtasks_encoding() from #38186)
- Revert encoding kwargs mistakenly applied to non-subprocess APIs
(exa get_contents, tempfile.mkstemp in webhook.py)
- Guard the ddgs worker Popen (new on main since #55339)
- Update two kwarg-snapshot test assertions for the new kwargs
Add authenticated GET /api/model/options to the gateway API server,
sharing the dashboard/TUI picker payload builder so external clients
can sync to the user's configured Hermes provider catalog instead of
scraping the single OpenAI-compatible /v1/models alias.
- new shared hermes_cli.inventory.build_model_options_payload() wraps
build_models_payload with the stable picker shape and safe
custom-provider probe policy (probe current only on normal open,
probe all + cache bust on explicit refresh)
- dashboard web_server and TUI gateway model.options refactored onto
the shared builder; dashboard build moved off the event loop via
run_in_threadpool
- capabilities endpoint advertises model_options
- docs for both API server and programmatic integration
Salvaged from PR #54689 by @abundantbeing.
New sessions.auto_archive / auto_archive_days config: soft-hide (never
delete) sessions with no activity for N days, aging on last activity
rather than creation so an old-but-active chat is spared. Sweeps are
throttled through state_meta and fire from CLI startup, gateway startup
+ hourly housekeeping, and the serve/dashboard backend (opportunistic
on session list + an hourly lifespan ticker), so every surface honours
one setting.
A new pinned column (declaratively migrated) exempts sessions from the
sweep; PATCH /api/sessions/{id} accepts pinned and flips the whole
compression lineage as a unit, mirroring set_session_archived.
Widens the salvaged .env injection fix (#50315) to the sibling site it
missed: hermes_cli/memory_setup.py::_write_env_vars is the near-identical
core writer the openviking plugin's copy was forked from, is fed directly
by interactive _prompt() (pasted API keys), and is reused by other memory
plugins (e.g. supermemory imports it). A pasted secret with an embedded
CR/LF injected an arbitrary extra KEY=VALUE line on the next read.
Same _env_line_safe() treatment as the plugin writer (strip every
str.splitlines() separator + NUL), matching config.save_env_value's
existing newline strip. Mutation-checked: reverting the sanitizer makes
the new regression tests fail.
Flips the default fan-out cadence from per_iteration (advisors re-run on
every tool iteration, multiplying advisor spend by tool-loop depth) to
user_turn (advisors run once on the first message of each user turn; the
acting aggregator works the rest of the tool loop with that turn's
advice). Until per-mode benchmarks justify a costlier default, MoA
defaults to the cheapest, lowest-impact cadence (#67199).
One default for everyone — no split legacy/new-preset semantics; presets
that want per-step advising set fanout: per_iteration explicitly. All
three modes (user_turn / per_iteration / every_n:N) remain selectable;
every_n:1 still collapses to per_iteration (semantic identity), while
unparseable values now fall to user_turn (the default).
Docs updated with a default-change note; the per-iteration rerun test
pins its mode explicitly.
Co-authored-by: skyer-flyyy <188930297+skyer-flyyy@users.noreply.github.com>
