Reported in Discord by @spherohero: `sessions recover --allow-partial` copied
20,817 of 20,824 message rows, then orphan cleanup deleted every one of them
because no session row was salvageable. Final output: 0 sessions, 0 messages.
The salvage worked and then threw the result away -- the exact opposite of
what --allow-partial exists to do.
_cleanup_partial_orphans() removed dependent rows whose session_id had no
matching sessions row. That is correct when a few sessions are lost; it is
catastrophic when the sessions b-tree is damaged worse than messages, which
is the common shape (sessions is small and hot, messages is large).
Reproduced exactly: 500 readable messages, unreadable sessions -> 500
copied, 500 removed, empty output.
Now _reconstruct_missing_sessions() runs FIRST, inside the same transaction,
synthesizing a minimal row per orphaned session_id (only id/source/started_at
are NOT NULL). started_at comes from the earliest surviving message.
Placeholders carry source='recovered' and an explicit title so a fabricated
session can never be mistaken for an original. Same repro now retains all
500 messages under 1 reconstructed session.
Reconstruction is reported as LOSS, not a clean recovery: session metadata
(title, model, timestamps, cost) is genuinely gone even though the
conversation text survived. loss_detected=True, partial=True,
complete=False, with a warning naming the counts.
Also fixes total_removed_or_relinked, which summed every dict value and would
now have counted retained messages as removed.
Sabotage-verified: removing the reconstruction call restores the wipe and
fails the test. 942 targeted tests green.
Closes the last false-pass window @helix4u flagged on #71779. He explicitly
said not to hold the PR for it; it is two lines, so worth doing rather than
leaving a known-soft assertion in a concurrency test.
connection_opened.wait(1.0) proved the connection had not opened, but not
that the connector thread had actually reached connect_tracked() -- an
unscheduled thread produces the same observation. The connector now sets
connect_attempted immediately before the blocking call, and the test waits
on that first, so "still blocked" means blocked at the lock rather than
not yet started.
15/15 stable at ~1.1s. Removed-lock sabotage still fails.
Second round of @helix4u review on #71779. Both findings reproduced before
fixing.
1. My previous fix turned a crash into SILENT DATA LOSS. Returning
status="missing" for a present-but-unusable state_meta looked like a safe
degrade, but _verify_recovered_database only escalates "failed"/"partial"
into a warning + loss_detected. Measured on the branch: a run that dropped
a real metadata table reported warnings=[], loss_detected=False,
partial=False, complete=True. Strictly worse than the ValueError it
replaced -- that at least failed loudly.
Now "failed" when the table exists but lacks key/value, "missing" only
when genuinely absent. The damaged case yields
warnings=['state_meta copy status is failed'], loss_detected=True,
partial=True, complete=False, while staying verified=True so the output
is still installable-with-review.
2. The race test I wrote had its own scheduling race: after the guard
released the lock, the racer could win before the main thread set the
release event, failing on a correct implementation. Rewritten per
helix4u's design -- copy runs in a worker parked inside the patched
copy, a second worker attempts connect_tracked(), assert it stays blocked,
release, assert it then opens. Deterministic and ~1.1s instead of 10s;
12/12 stable.
Sabotage-verified. Note the third scenario only failed after adding a
unit-level test: recover_session_database short-circuits on the inspection
result when state_meta is entirely absent, so the helper's absent-branch is
unreachable end-to-end and a regression there was invisible. Both statuses
are now pinned directly.
939 targeted tests green.
Post-merge follow-up to #71770. Both defects were found by @helix4u in review
and reproduced against merged main before fixing.
1. Check/use race in _copy_source_bundle (my bug, from the #71770 follow-up
commit). It called has_live_connection(), released the registry lock, and
only then ran shutil.copy2() over the bundle. A tracked connection could
open in that window; the copy's close() then cancels its POSIX advisory
locks -- the exact class #71724 closed. Measured on main: a racer thread
opened a connection mid-copy after blocking 0.000s.
Adds sqlite_safe_read.offline_file_access(), a context manager that holds
the connection-lifecycle lock across an entire multi-step raw access, and
routes the bundle copy through it. Same racer now blocks 10.0s until every
raw descriptor is closed. Any future raw read of a database file (hashing,
moving a bundle aside) should use this rather than a bare pre-check.
2. _copy_state_meta_salvage assumed a 'key' column. A damaged state_meta can
keep 'value' and lose 'key'; columns.index("key") then raised ValueError
and aborted the whole partial recovery. The mirror case (key without
value) would have copied key-only rows and reported the table complete.
Now requires both, matching the non-partial _copy_state_meta, so an
unusable optional table is recorded as missing/failed and --allow-partial
still recovers sessions and messages.
Both regression tests verified by sabotage: reinstating the bare pre-check
fails the race test, removing the key/value requirement fails the other.
937 targeted tests green.
Adds build_aux_picker_rows() + format_aux_picker_entries() to
hermes_cli/inventory.py and converts both auxiliary-task pickers to them,
so custom providers, exclusions, and picker visibility can no longer be
dropped one call site at a time.
The two salvaged commits each fixed one kwarg at these same two sites
(user/custom providers, then for_picker). Both were per-site patches, so
the next aux picker would have reintroduced the gap. The seam makes the
correct behaviour the default a new caller cannot forget:
- user `providers:` and saved `custom_providers:` entries
- `model_catalog.excluded_providers`, matching /model
- exhausted-credential-pool providers stay visible
- only the active custom endpoint is probed, so an offline saved local
server can't hang the picker
- the virtual `moa` row is excluded (auxiliary_client unwraps it to the
aggregator anyway, so offering it is a silently-rewritten choice)
The vision picker also gains the current-selection marker it never had.
Also threads for_picker through build_models_payload, which had no way to
express it despite list_authenticated_providers supporting it.
credential pool has entries but is entirely rate-limited (exhausted): rate
limits are per-model for many providers (e.g. Google Gemini), so an exhausted
key for model-A may still serve model-B, and the user should stay able to
select it. That fix wired `for_picker=True` into `list_picker_providers`.
The two sibling interactive pickers reached by `hermes model` still called
`list_authenticated_providers()` WITHOUT `for_picker=True`, so they kept
hiding exhausted-pool providers:
- `_aux_select_for_task` (hermes_cli/main.py) — the auxiliary-task
provider/model picker.
- `_configure_vision_provider_model` (hermes_cli/tools_config.py) — the vision
provider/model picker.
Both persist a per-task config the user runs *later* (once the momentary
cooldown clears), so hiding an exhausted-pool provider here is exactly the
Tests: two regressions assert each picker requests `for_picker=True` (they
omitted it before the fix); the existing picker/exhausted-pool suite still
passes (6 total).
Follow-up to the partial-recovery salvage. _copy_source_bundle() raw-copied
the source state.db and its -wal/-shm sidecars with no live-connection check.
Copying a database file is an open()/close() on it, and close() cancels every
POSIX advisory lock the process holds on that file -- including a running
VACUUM's EXCLUSIVE lock (fe431651c5). hermes_state._backup_db_file already
refuses that situation; this path did not, leaving two policies for one
hazard. Verified against the merged guard: with a tracked connection open,
_backup_db_file returned None (refused) while _copy_source_bundle copied
anyway.
Recovery normally runs as its own short-lived CLI process against an
offline/quarantined file, so this should never fire in practice. It is a
consistency fix, not a live corruption path -- but it is exactly the drift
that reintroduces the bug later.
Regression test verified by sabotage: removing the guard fails it.
Addresses review findings on the previous commits. Three of them were real
defects I reproduced against my own head before fixing.
1. Check/use race (BLOCKING). read_header_bytes_preopen() checked
has_live_connection() under _live_lock, released it, then did the raw
open/read/close outside the lock; connect_tracked() opened before
registering. A thread could pass the "nothing is live" check, another
could open a connection and BEGIN IMMEDIATE, and the first thread's
close() then cancelled its POSIX locks -- the exact bug this guard
exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED).
_live_lock now spans all three lifecycle transitions: open+register,
unregister+close, and check+open+read+close.
2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's
read-only path opens file:/…/state.db?mode=ro; that string was fed to
Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of
the real Path could match, so read-only connections were invisible to
the guard and their locks cancellable. Reproduced with no forced
scheduling. Keys now come from PRAGMA database_list (canonical path),
with an explicit tracking_path override.
3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception
and retried an untracked plain connect, so any error silently disabled
the guard. Now only ImportError (scaffold installs without hermes_cli)
falls back; real failures propagate.
4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db()
and _backup_db_file() raw-read live databases; they now REFUSE when a
connection is live rather than warning. Losing a forensic copy beats
corrupting the database being rescued.
Custom factories are no longer rejected (that broke legitimate callers) nor
silently untracked -- the tracking close() is mixed into whatever factory is
in play, including when an opener substitutes its own after the fact.
WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was
confounded -- the clean WAL result came from 3.53.1, which carries both the
WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the
bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and
DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the
WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL
until a fixed runtime ships is the conservative call, and the WAL policy does
not belong in this root-cause fix.
Six sabotage runs confirm each new test fails when its defect is reinstated
(including two that initially did NOT -- the race test was rewritten to pause
inside the byte read, and a separate test added for the opener-substituted
factory path). 1124 targeted tests green.
Completeness pass over the previous commit.
Registry leak (would have silently disabled the guard): the first version
incremented on open but decremented only in SessionDB.close(). kanban's
connect() hands raw connections to callers who close them directly (4 sites),
so its counter only ever went up — after enough kanban operations every
byte-probe on that path would be refused forever, disabling zeroed-file and
header detection. Replaced manual track/untrack with a TrackedConnection
subclass that untracks in close(), the one method every close path goes
through. Verified across plain close, contextlib.closing, double close,
nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not
a close) correctly stays tracked.
Also: a caller-supplied factory now wins instead of raising TypeError on a
duplicate kwarg, since tracking is an optimisation for the probe guard, not a
precondition for opening the database.
Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert.
Audit notes:
- DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains
correct there; those users are protected by the raw-read fix, not by the
journal mode.
- The remaining whole-file reads are on genuinely offline artifacts:
_backup_db_file (DB won't open; bytes preserved for forensics),
_backup_corrupt_db (quarantine path, now warns if a connection is live),
and backup verification of snapshots. backup._safe_copy_db already uses
the SQLite backup API rather than a byte copy.
- The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped,
not process-scoped), so this is a Linux/macOS correctness fix; the change is
platform-neutral and safe on Windows.
Sibling tests updated: session recovery no longer expects a DELETE-mode
recovered DB, and the kanban WAL-fallback test patches the connect site that
actually runs now.
`hermes sessions optimize` could corrupt state.db. Root cause is Hermes,
not the SQLite WAL-reset bug (#69784).
close() on ANY file descriptor for a SQLite database cancels every POSIX
advisory lock the process holds on that file, including a running VACUUM's
EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed
live databases in several hot paths: the zeroed-state.db detector runs on every
SessionDB construction (and the gateway builds those constantly), and kanban's
post-commit invariant check ran after every COMMIT. While VACUUM rewrote the
file, those probes dropped its lock and let other processes write into it.
A/B against the real code, only variable being the raw read:
SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode
raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt
no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt
SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt.
After this change: 0/4 corrupt, 0 vacuum errors.
Because the upgraded runtime corrupts too, replacing the embedded SQLite does
not fix this class; and because DELETE is where it reproduces, #70055's
"force DELETE on vulnerable builds" mitigation steered users into the failing
mode. That gate is reverted here: vulnerable builds get WAL again and still
warn so operators can upgrade.
- add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the
existing connection instead of open()+seek(28); byte-level probes are
restricted to before any connection exists and refused once one is live,
with an explicit force= escape for offline artifacts (snapshots, archives)
- track live connections in SessionDB and kanban's connect so that guard is
enforced rather than merely documented
- kanban's torn-extend check now only applies under a rollback journal; in WAL
a committed page may still legitimately sit in the -wal file
- revert the force-DELETE WAL gate and update the tests that pinned it
Regression tests assert the behavioural contract (an external process stays
locked out across Hermes' inspection calls) and were verified to fail when the
old raw-open behaviour is restored.
The background write guard decided ownership from `isinstance(usage_rec, dict)`,
so a local skill with NO usage record passed. That successful write called
bump_patch(), which created a `created_by: null` record — and the identical
write was refused from then on. "Allowed exactly once, then never" is a race
with our own bookkeeping, not a policy. Reproduced on main: patch #1 succeeds,
patch #2 with the same arguments is refused.
Option B from the issue. Option A (split `session_review` from
`scheduled_curator` and let the session fork patch user-owned skills it
consulted) would widen autonomous write permission onto skills the user owns
with no user present to consent — wrong direction for a no-user-present actor.
- skill_manager_tool: missing and explicit-null records now resolve
IDENTICALLY, both fail closed. The refusal names the reason and points at
`hermes curator adopt <name>`.
- background_review: both review prompts told the reviewer to patch any skill
consulted in the session and claimed pinned skills could be improved, while
enforcement refused both. Prompts now list pinned, external, and user-owned
skills as protected, and tell the reviewer to RECOMMEND adoption instead of
attempting a write that will be refused.
- skill_usage: document that `created_by` is a curator-management policy flag,
not a provenance claim, and add `is_curator_managed()` so call sites read as
the question they ask. Field name retained — it is on disk in every
`.usage.json` and renaming would strand those records.
- curator CLI: `hermes curator list-unmanaged` itemizes unmanaged skills with
the reason each is unmanaged (completes the #67139 spec).
Foreground writes are untouched: a user-directed edit to a user-owned skill
still works, including on pinned skills.
Sibling tests: 9 failures in test_skill_manager_tool.py were fixtures that
created record-less skills to exercise OTHER guards (consolidation-delete,
read-before-write) and relied on ownership falling through. Fixed at the
fixture, since the real curator only ever operates on managed sediment. One
test asserted the old "manually authored" wording; rewritten to assert the
behavior contract instead of the string.
Validation: 274 targeted tests + all 7 background-review files (60 tests) pass.
E2E on a temp HERMES_HOME (30 checks) covers the flip, foreground writes,
adoption unblocking, pin semantics, prompt/enforcement parity, and the new verb.
Each new test sabotage-verified: revert the fix, confirm it goes red.
Fixes#67140
The residual Windows-on-ARM integrity-gate fix added GetNativeSystemInfo as a
second "OS-native" probe behind IsWow64Process2. Microsoft documents the
opposite behavior: "the API GetNativeSystemInfo also returns emulated
processor details when run from an app under emulation."
On the exact hosts the probe was added to rescue -- an x64 hermes-setup.exe
emulated on an ARM64 Surface -- it therefore reports AMD64, making it a
duplicate of the PROCESSOR_ARCHITECTURE rung already below it rather than a
fallback for it. Its test only passed because the kernel32 fake was hand-fed
ARM64, asserting behavior real Windows does not exhibit.
Replace it with GetMachineTypeAttributes, which answers the question the gate
actually asks -- "can this host load a PE of machine X?" -- instead of
inferring it from an architecture name. It is also the only documented API
that reports AMD64-on-ARM64 emulation support. The gate prefers it and falls
back to the existing name-based mapping on pre-Windows-11 hosts.
The load-bearing half of the previous fix (typing GetCurrentProcess as HANDLE
so IsWow64Process2 stops failing ERROR_INVALID_HANDLE) is unchanged.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: Teknium <teknium1@users.noreply.github.com>
`hermes curator status` reported only the skills it manages, staying silent
about curation-eligible skills it can never touch. On a 237-skill library that
meant 112 skills were invisible to every automatic transition with no signal
anywhere — the curator looked broken when it was working as designed.
A skill becomes curator-managed only when `created_by: agent` lands on its
usage record, and only the background review fork writes that marker. Two
populations therefore never qualify: records written before the marker existed
(no key at all, authorship unknowable) and every foreground
`skill_manage(create)` (unset by design — those skills belong to the user).
- skill_usage: `list_unmanaged_skill_names()` / `unmanaged_report()` enumerate
the blind spot, tagging each row with `has_provenance_key` so the two causes
are distinguishable. `adopt_skill()` writes the marker on user declaration
and refuses bundled, hub-installed, external, and protected built-ins.
Adoption never resets the inactivity clock.
- curator CLI: status prints an `unmanaged (no provenance marker)` block on
BOTH the managed and no-managed-skills paths; new `adopt` verb takes names or
`--all-unmanaged`, with `--dry-run` and a confirmation prompt on bulk.
- skills_sync: `_backfill_optional_provenance()` matched candidates by
repo-derived path only, so a skill installed at `mlops/chroma` that upstream
later moved to `mlops/vector-databases/chroma` was skipped forever and
`hermes skills repair-optional` could not fix it. Falls back to an
unambiguous name match, still gated on identical content, and records the
ACTUAL install path.
Provenance stays a declaration, never an inference: a high patch count proves
the agent MAINTAINS a skill, not that it authored one, since Hermes edits
user-written skills on the user's behalf routinely. An "looks agent-made"
heuristic would eventually archive hand-written work.
Validation: 347 targeted tests pass. Each new test verified via sabotage run
(revert the fix, confirm the test goes red) — one initially passed for the
wrong reason because the fixture pinned `prune_builtins` off, masking the guard
under test; fixed to force the shipped default on.
The retry loop skipped only the current version, so on a uv whose download
catalog is stale it walked backwards through older patches. In #71250 the
newest indexed 3.11 is 3.11.14 — the installed version — so all five retries
went to 3.11.13..3.11.9, each a real download+install+probe+delete cycle that
the existing downgrade guard was always going to reject, before failing anyway.
Only newer patches can carry the SQLite fix. Skipping <= current makes the
stale-catalog case cost one attempt instead of six, and still finds 3.11.15
on the first retry when the catalog knows about it.
Fixes#71250.
`hermes update` could not repair the embedded Python runtime's
vulnerable SQLite (WAL-reset bug range 3.7.0-3.51.2, except backports
3.50.7/3.44.6) on installs where `uv python install <minor>` (bare
request, e.g. "3.11") resolves to an older cached/indexed patch that
still links a vulnerable SQLite build, even though a newer
non-vulnerable patch on the same minor line is available and known to
uv. The smoke test correctly rejected the vulnerable candidate every
time, but the provisioner gave up immediately after one attempt,
leaving the repair permanently stuck in the same failing loop on every
`hermes update` run.
Implements option D from the issue (query the index, retry with an
explicit newer patch), which the reporter identified as cleanest:
- New `_list_available_patches()`: runs `uv python list <minor>
--all-versions --only-downloads --output-format json --no-config`,
filters to cpython/default-variant entries (excluding pypy/graalpy),
and returns known patch versions newest-first. Fails safe (returns
[]) on any network/parse error.
- Refactored the single install+find+probe cycle out of
`_install_safe_python_generation()` into `_attempt_install_generation()`,
reusable per attempt with its own generation directory (so a
rejected candidate's files are fully cleaned up before the next
attempt, matching the existing --reinstall semantics).
- `_install_safe_python_generation()` still tries the bare minor-line
request first (preserves the original comment's rationale: for a
given exact patch, python-build-standalone may have no artifact with
fixed SQLite at all). If that resolves vulnerable, it now queries
`_list_available_patches()` and retries with explicit newer patches,
newest-first, bounded to `_MAX_PATCH_RETRIES` (5) attempts -- each
attempt is a real download+install+probe cycle, so the cap keeps
worst-case repair time bounded.
Sanity-checked `_list_available_patches()` against the real `uv`
binary (0.11.7) in this environment: correctly parses live `uv python
list --all-versions --output-format json 3.11` output, returns 14
patches sorted newest-first starting at 3.11.15.
9/9 new tests pass (retry succeeds with a newer patch, exhausts
gracefully when every known patch is vulnerable, empty patch list
degrades to None without crashing, retry count is bounded, plus direct
JSON-parsing unit tests for realistic/malformed/empty uv output);
47/47 in the full tests/hermes_cli/test_managed_uv.py file (including
the 3 pre-existing tests for the original bare-minor success path,
confirming no regression there).
A failed `hermes sessions repair` printed "keep state.db and the backup"
and stopped, so a user whose sessions had vanished had no way to discover
that a non-destructive recovery path exists — the reported dead end.
The failure branch now names the next command, read-only step first, seeded
with the backup path it just preserved. Covered by a CLI-surface regression
test that drives the real subprocess against an unrepairable database.
* fix(update): bind IsWow64Process2 HANDLE and fall back to GetNativeSystemInfo
The #71218 OS-native probe still rejected correct ARM64 Desktop rebuilds on
Windows-on-ARM when ctypes truncated GetCurrentProcess()'s pseudo-handle and
IsWow64Process2 failed with ERROR_INVALID_HANDLE, falling through to the
lying PROCESSOR_ARCHITECTURE=AMD64 value. Type the HANDLE correctly and use
GetNativeSystemInfo before the env-var fallback.
* test(update): cover WoA IsWow64Process2 handle failure and system-info fallback
Pin the residual #71218 shape where IsWow64Process2 returns FALSE, the env
arch lies as AMD64, and GetNativeSystemInfo must still report ARM64 so the
integrity gate accepts a correctly-built ARM64 Hermes.exe.
Two tests fail deterministically on main depending only on which SQLite the
test interpreter links — nothing about the code under test. Both are green in
isolation and red in the suite / on an older library, the worst diagnostic
shape.
Root cause A — WAL is not always WAL. Hermes refuses journal_mode=WAL on
SQLite builds carrying the upstream WAL-reset corruption bug (3.7.0–3.51.2,
excluding backports 3.50.7 / 3.44.6) and falls back to DELETE. On such a
build NO -wal sidecar is ever created, so
test_wal_checkpoint_truncates_wal_file asserts on a file that cannot exist.
Invisible locally when the repo .venv and the Hermes managed runtime link
different versions (observed: .venv 3.50.4 → DELETE, runtime 3.53.1 → WAL),
so the same test passes for one interpreter and fails for the other.
- tests/conftest.py: add a `requires_wal` marker plus a
pytest_collection_modifyitems hook that skips such tests when the linked
library will fall back to DELETE. The skip reason names the actual
version so it is diagnosable rather than mysterious.
- pyproject.toml: register the marker.
- test_kanban_db_repair.py: mark the -wal-sidecar test.
Root cause B — process-global warn-once dedup. The WAL-fallback warning is
emitted at most once per (process, db_label). Any earlier test in
test_kanban_db.py that opens a kanban.db consumes that one-shot, so
test_connect_falls_back_to_delete_on_locking_protocol sees zero warnings and
fails — but only as part of the file, never alone.
- test_kanban_db.py: clear both dedup sets in the test that asserts on the
warning, with a comment explaining the isolation trap.
The gate deliberately does NOT import hermes_state. That module computes
DEFAULT_DB_PATH from get_hermes_home() at import time, so importing it during
collection — before the per-test _isolate_hermes_home fixture redirects
HERMES_HOME — permanently caches the developer's REAL ~/.hermes/state.db for
the whole session. The first version of this change did exactly that and made
tests read a live 31,881-session production database (test_console_engine
asserted "Total sessions: 2" and got 31881). The version predicate is
duplicated instead, and tests/test_conftest_wal_gate.py pins the two
implementations in agreement across every documented upstream boundary plus
guards against the import coming back.
Verified: on SQLite 3.50.4 the sidecar test SKIPS naming the version; on
3.53.1 it RUNS and passes, so coverage is not lost where WAL works. Clean
main fails exactly these 2 tests under
`scripts/run_tests.sh tests/hermes_cli/ tests/test_hermes_state.py`
(9926 passed, 2 failed); with this change the same scope is green.
Tests: 726 passed across the two kanban files, test_hermes_state.py, and the
new gate tests.
Reviewer egilewski identified two recovery-loss paths:
Path A — quarantine race (hermes_state.py):
SessionDB checks state.db before quarantine_zeroed_state_db() without a
shared cross-process lock, and Path.rename() may replace an existing
destination. Two writable startups can therefore move the first
instance's newly created database over the .zeroed-*.bak, erase the
original damaged-file evidence, and replace the live database with
another empty one.
Fix: add a cross-process lock (msvcrt on Windows, fcntl on POSIX)
around quarantine_zeroed_state_db() with a 5s bounded timeout. Under
the lock: re-check is_zeroed_state_db (another process may have already
quarantined it and created a fresh DB), use a PID-suffixed unique
destination, and non-clobbering rename with counter fallback.
Path B — size-cap pruning gap (hermes_cli/backup.py):
_too_large() runs before failed-database tracking. With keep=1 and a
size cap, an oversized state.db is omitted while failed_dbs stays empty,
so automatic pruning deletes the older complete snapshot that may
contain the only recoverable database.
Fix: track oversized DB files in a new oversized_skipped list (both in
the directory walk and top-level file loop). The manifest now records
oversized_skipped. Pruning is suppressed when failed_dbs or
oversized_skipped is non-empty, preserving the older complete snapshot
as recovery source.
Tests:
- test_concurrent_quarantine_no_clobber: two threads racing on the
same zeroed state.db — verifies quarantine backup survives with
original bytes and live DB is valid.
- test_oversized_db_suppresses_pruning: keep=1 + oversized state.db
verifies the older complete snapshot is not pruned.
All 33 tests pass (3 zeroed_state_db + 25 TestQuickSnapshot + 5
quarantine_forensic_logging).
Hardening for the Windows zeroed-state.db class (#68474):
- Surface critical stdout when pre-update/quick snapshot cannot copy a
present *.db (was log-only; update still looked successful).
- Detect all-NUL SQLite header on SessionDB open, quarantine the bytes,
and open a fresh DB with recovery guidance to state-snapshots.
Does not claim storage-stack root cause.
The #71119 integrity gate rejected CORRECT ARM64 desktop rebuilds on
Windows-on-ARM (#69179 follow-up): platform.machine() reports the PROCESS
architecture, and the update chain runs an x64 hermes-setup.exe / x64
Python under ARM64 emulation, so the gate compared the ARM64 Hermes.exe
against a phantom 'AMD64 host' and aborted the update.
_windows_native_machine() now asks the OS via IsWow64Process2 (whose
nativeMachine is truthful from emulated processes), falling back to
PROCESSOR_ARCHITEW6432/PROCESSOR_ARCHITECTURE (pre-1511 Win10) and then
platform.machine(). All three consumers — the integrity gate, the
packaged-exe arch preference, and backup validation — go through it.
Sabotage-verified: the three new regression tests fail against the old
process-arch behavior and pass with the fix.
Builds on @ms-alan's label fix: the negative figure had a second cause
that relabelling alone leaves in place.
`hermes sessions optimize-storage` reported "reclaimed -3820.1 MB" on a
database that had in fact shrunk 60% (25069 MB -> 9975 MB). Both figures
came from os.path.getsize(). In WAL mode a VACUUM's rewrite lands in the
-wal file, and the checkpoint that folds it back is REFUSED
(SQLITE_BUSY) while any other connection holds a read-mark — e.g. the
live gateway. So the main file still carried its pre-VACUUM size AND kept
growing while the command ran, making the after-figure larger than the
before-figure. The TRUNCATE checkpoint in close() lands after the caller
has already measured and printed.
- hermes_state.py: add SessionDB.logical_size_bytes() — page_count *
page_size, the size the file settles at once the WAL is folded back.
Correct immediately, readers or not; returns None when the connection
is gone so callers fall back to stat().
- hermes_state.py: best-effort TRUNCATE checkpoint after the optimize
VACUUM so the file settles promptly when nothing else holds the DB.
Documented as insufficient alone (busy under a live gateway) so the
stat() approach is not reintroduced.
- hermes_cli/main.py: both `optimize-storage` and `optimize` report via
logical_size_bytes(); fixing only the reported command would have left
the same bug class next door.
- hermes_cli/main.py: lift the contributor's inline label to a
module-level _size_delta_label() shared by both sites, so the "grew by"
wording is consistent and unit-testable without reading source.
The two halves are complementary: page accounting removes the phantom
negative, and "grew by" still covers a genuine increase when concurrent
session writes outweigh what the rebuild freed.
Verified: sabotaging logical_size_bytes() back to stat() fails the new
test with a 22 MB overstatement, reproducing the report. The test asserts
its own precondition (stat() must actually be lagging) so it cannot
silently stop exercising the bug.
Tests: 464 passed (test_hermes_state.py + the new label tests).
Co-authored-by: chenbin <h-chenbin@voyah.com.cn>
The post-update state.db integrity guard called verify_sqlite_integrity()
with max_bytes=0, which disables the size ceiling and forces a full
PRAGMA integrity_check. That pragma walks every b-tree page in the file,
so its cost scales with database size — measured on a real 30 GB state.db:
143.5s with a cold page cache (worse under an update's memory pressure),
with zero output on screen. The update looks hung right after
"✓ Code updated!" and a CPU sits pegged.
Multi-GB session databases are normal for heavy users, so a
size-unbounded check is never an acceptable default on the update path.
- verify_sqlite_integrity(): max_bytes now defaults to
DEFAULT_INTEGRITY_CHECK_MAX_BYTES (2 GiB) instead of 0. max_bytes=0
remains the explicit opt-in for a full scan.
- The oversized path no longer degrades to a header-only check: it adds a
constant-time structural probe (read-only open + schema_version +
sqlite_master read) so the malformed-schema class is still caught, not
just the #68474 zeroed-file signature.
- Drop the explicit max_bytes=0 at the post-update guard and in
copy_db_and_verify() so both inherit the bounded default.
Measured on the reporter's real 30 GB state.db: 143.5s cold → 0.001s,
still valid=True. Corruption detection verified at multi-GB scale for
both classes (zeroed header, malformed schema) — both still fail closed.
Tests: default-is-bounded invariant, oversized probe catches malformed
schema, max_bytes=0 still forces the full check.
Bug-class coverage for both fixes: the full catalogue survives Save, context
lengths are preserved, the key never lands in config.yaml on either write
path, blank clears it, a pre-fix plaintext key migrates while a ${VAR}
template is left alone, two endpoints on one host keep separate credentials,
and an IP-derived name is still a valid POSIX env var.
The two delete tests asserted on the plaintext mirror; they now assert the
same invariants against the credential reference.
The desktop self-update chain (Desktop -> hermes-setup --update ->
hermes update -> hermes desktop --build-only -> relaunch) rebuilds
Hermes.exe on the user's machine and declared success on bare file
EXISTENCE. A truncated PE (corrupt cached Electron zip / interrupted
extraction or rcedit rewrite / full disk) or a wrong-architecture
unpacked tree therefore shipped as the 'updated' app, which Windows
refuses to load with 'This app can't run on your computer'
(此应用无法在你的电脑上运行) — and the previous working build had
already been wiped by before-pack.mjs, leaving nothing to fall back to.
Fix, in three parts:
- hermes_cli/main.py: post-build integrity gate on Windows
(_ensure_desktop_exe_launchable). Parses the PE header of the freshly
built Hermes.exe — MZ/PE magic, section-table completeness vs file
size (catches truncation), and COFF machine vs the host arch (catches
arm64/x64 mixups). On failure it purges the (likely corrupt) cached
Electron zip, invalidates the content-hash build stamp so the
updater's retry-once genuinely re-downloads and rebuilds, restores
the previous build from the .bak tree when one exists (keeping the
corrupt tree as .corrupt for diagnostics), tells the user the update
was aborted and their old version kept, and exits nonzero.
_desktop_packaged_executable also now prefers a host-loadable PE over
pure newest-mtime when multiple win-*-unpacked trees coexist.
- apps/desktop/scripts/before-pack.mjs: on win32, the previous unpacked
tree is preserved as <appOutDir>.bak (only when it holds the product
exe — partial/corrupt trees still get the plain wipe) instead of
being destroyed, providing the rollback material for the gate above.
Non-Windows behavior is unchanged.
- Behavior-contract tests: tests/hermes_cli/test_desktop_exe_integrity.py
(23 tests — synthetic PE fixtures for truncation/non-PE/arch-mismatch,
rollback semantics, and the build-only exit contract) and 6 new vitest
cases in before-pack.test.mjs for the .bak preservation rules.
Progresses #69179
/api/status is the only public liveness route, and its handler loads the
gateway config, probes gateway health, and counts sessions before it can
answer. That work is wrong for a readiness probe: a caller that only needs
to know the process is up pays for a cold plugin import tree.
Add /api/health, which returns process liveness, version, and the auth-gate
shape and touches nothing else.
`_render_distribution_plan` reads the target profile's `.env` to decide
whether a required env var is already set (so it doesn't nag the user),
using `Path.read_text()` with no encoding. Two bugs:
1. `Path.read_text()` defaults to the system locale (cp1251/GBK on Windows),
which raises `UnicodeDecodeError` on any non-ASCII byte. The surrounding
`except OSError` does NOT catch that — `UnicodeDecodeError` is a
`ValueError` — so a mis-encoded `.env` aborts the entire install preview.
2. Even on a UTF-8 locale, a Notepad-added BOM prefixes the first key
(`KEY`), so the very first required env var is mis-reported as
"needs setting" when it is actually present.
`.env` is written as UTF-8 everywhere in the codebase. Read it as
`utf-8-sig` (tolerates the BOM) and also catch `UnicodeDecodeError` so a
genuinely un-decodable file skips the pre-check instead of crashing.
Regression tests: a BOM-prefixed `.env` whose first key must still read as
"set", and an invalid-UTF-8 `.env` that must not abort the preview.
/api/status?profile= now passes pid_path=/path=/expected_home= to the
PID and runtime-status readers; the profile-unification fakes had
zero-arg signatures and raised TypeError. Plain /api/status call shapes
are unchanged (pinned by the existing zero-arg tests in
test_web_server.py).
Follow-up for the salvaged #70498 fix: replace the original PR's
mock-signature churn (28 lambda **kw edits, needed only because it changed
the no-profile call shape) with two targeted regression tests:
- ?profile=<name> must pass the profile's gateway.pid / gateway_state.json
paths and expected_home to the gateway status readers (HOME-anchored
per-profile state under ~/.hermes/profiles/<name>/)
- ?profile=<unknown> must 404 via _resolve_profile_dir
The production change keeps plain /api/status on the exact zero-arg calls,
so every pre-existing test passes unmodified.
The .env sanitizer inferred missing newlines from known KEY= substrings
inside existing values. Plain secrets containing those bytes could therefore
be split into synthetic assignments and rewritten to disk.
Treat each physical line as the only assignment boundary and keep bytes after
the first equals sign opaque for boundary discovery. Preserve safe formatting,
null-byte removal, BOM handling, and normal one-assignment-per-line parsing.
Cover direct loading, dotenv loading, sanitization, writers, and migration
with behavioral regressions.
Fixes#29155
Follow-up for salvaged PR #69141, addressing the last open review point:
cmd_prune() only set orphan_allowlist inside 'if orphans or pre_v2_orphans',
so a zero-orphan preview passed the unrestricted None sentinel down to
prune_checkpoints(), authorizing deletion of any project that became
orphaned between the preview and the rescan — with zero confirmation
calls. The allowlist is now bound unconditionally for every non-force
run (empty preview => empty allowlist); --force keeps None. Adds the
zero-orphan-preview timing regression plus allowlist-identity tests.
Requested by egilewski on #69141: the orphan confirmation flow had no
test coverage at all before this. Exercises hermes_cli.checkpoints.cmd_prune
directly against pre-v2-only and mixed (v2 + pre-v2) fake stores —
decline aborts with nothing deleted, accept deletes both layouts,
--force and --keep-orphans skip the prompt as expected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two compression-tip hydration tests simulated legacy state by emptying
the parent AFTER end_session(compression) — exactly the durable write
the new closed-parent guard refuses. Reordered: empty first, close
second. The tests' actual contract (old id hydrates from the live tip)
is unchanged and still pinned.
The runtime-repair change passes repair_observer= to update_managed_uv/
ensure_uv; the autouse fixture fakes had zero-arg signatures and raised
TypeError through the mock. Sibling test file to the PR's own suite.
Follow-up on the #70186 salvage. The cherry-picked repair pinned the
candidate to the exact current CPython patch (e.g. 3.11.14). Verified
live with uv 0.11.19: every published python-build-standalone artifact
for 3.11.14 links vulnerable SQLite 3.50.4 — even with --reinstall — so
the exact-patch pin made the repair permanently impossible on the
installs that need it most (repair_vulnerable_runtime returned
'failed: could not provision a fixed private Python runtime').
Request the minor line (3.11) instead — the same resolution a fresh
'uv python install' would make, still inside requires-python — and
tighten the drift gate to 'same minor, no downgrade'. E2E-verified
end-to-end on a real vulnerable venv: repair_vulnerable_runtime()
provisioned 3.11.15, built + smoke-tested the sibling venv, cut over,
and reported SQLite 3.50.4 → 3.53.1 with the old venv parked for
rollback.
Follow-up fixes on top of the cherry-picked guard:
- verify_sqlite_integrity(): an oversized (max_bytes-exceeding) database
now still fails on a zeroed/invalid header — previously valid=True was
set before the header check, so a >1GiB zeroed state.db (the exact
#68474 signature at 95MB scale) passed as valid.
- _run_pre_update_backup(): the guard referenced get_hermes_home before a
later function-local 'from hermes_constants import get_hermes_home'
shadowed it → UnboundLocalError swallowed by the snapshot try/except,
silently disabling the post-snapshot check AND the snapshot-id output.
Alias the import explicitly.
- Import _quick_snapshot_root where used (was NameError in all 3 guards).
- tests/hermes_cli/test_state_db_guard.py: real-SQLite E2E coverage —
valid/zeroed/truncated files, oversized header gate, copy+verify
roundtrip, snapshot restore flow, and the live _run_pre_update_backup
path against a temp HERMES_HOME with mid-flight zeroing.
A brew Python upgrade (original report) or an interrupted venv rebuild
(v0.19.0 report in the same thread) can leave certifi importable while
its bundled cacert.pem is missing or a dangling symlink. Every TLS
connection then fails — Feishu/Telegram/WeChat/DingTalk all down —
with an opaque 'Could not find a suitable TLS CA certificate bundle'
from deep inside httpx/requests.
The existing repair infrastructure only probed
`hasattr(certifi, 'contents')`, which PASSES in exactly this failure
state, so neither the early venv self-heal nor `hermes update`'s
import-probe repair ever classified certifi as broken. Extended, not
replaced:
- hermes_cli/_early_recovery.py: the in-process probe now also
validates that certifi.where() exists and is a plausible bundle
(>=1KiB), so the pre-import self-heal repairs it like any other
wiped core package.
- hermes_cli/main.py (_detect_broken_lazy_refresh_imports): the
subprocess probe script used by `hermes update`'s venv repair
applies the same bundle-file check inside the target venv.
- hermes_cli/doctor.py: `hermes doctor` already failed the cert
check; `hermes doctor --fix` now repairs it (pip force-reinstall
certifi + module-cache invalidation + re-verify), covering
brew/manual venvs where no update marker exists. Failures funnel
into the manual-action list with the exact command.
- agent/ssl_guard.py: the startup SSLConfigurationError hint now leads
with `hermes doctor --fix` instead of only the raw pip command.
Fixes#29866
Follow-up on the #69500 salvage: 'launchctl submit' jobs remain
registered with launchd after they exit, so every plist reload leaked
one dead '<label>.reload.<pid>.<ts>' label. The helper script now ends
with 'launchctl remove' of its own transient label, and the recovery
test asserts the self-removal is present.
The deferred launchd reload helper used start_new_session=True to detach
from the gateway's process group. However, setsid(2) alone does NOT move
the child outside the launchd job's process coalition — when launchctl
bootout fires on the gateway label, launchd terminates ALL processes in
that coalition, including the setsid-detached helper, leaving the service
permanently unloaded.
Fix by spawning the helper via launchctl submit, which creates a
transient launchd one-shot job that is wholly independent of the
gateway's coalition. This ensures the helper survives bootout and can
complete the bootstrap+verify cycle.
Also writes a durable pre-bootout marker to the reload log so the
distinction between 'helper never started' and 'helper ran but
bootout/bootstrap failed' can be diagnosed.
Fixes#69098
- Strip the salvaged commit's inline encoding kwargs where main had since
gained its own (process_registry, local env, cua doctor, gateway,
commands, gateway_windows — the latter keeps its locale-aware
_schtasks_encoding() from #38186)
- Revert encoding kwargs mistakenly applied to non-subprocess APIs
(exa get_contents, tempfile.mkstemp in webhook.py)
- Guard the ddgs worker Popen (new on main since #55339)
- Update two kwarg-snapshot test assertions for the new kwargs
