The nonzero-exit guard already knew a failing exit code with real output
usually isn't a failure worth painting red. It only looked at output /
stdout / stderr, but background-process polls report their text under
output_preview — so a poll of any process that exited nonzero rendered
destructive-red with no error to show, even when the output was routine
npm chatter. Half the red process rows in a real session history were
this case.
Sentinel-released processes exposed a second bare-sample assertion in the
same file. The unread-dot check ran a synchronous .count() 140ms after the
running dot cleared:
15.91s poll "dot should disappear" -> 2
16.05s ... -> 0 (running dot gone)
16.05s bare .count() for unread dot -> 0 FAIL
"Finished — unread" is an event-driven transition that lands just after the
running dot clears. The old fixed `sleep 5` happened to leave enough slack
between the two that a single sample usually caught it; releasing the
process deterministically removed that incidental slack and made the latent
race deterministic instead.
Poll for it, matching how sidebar-states.spec.ts already asserts this exact
dot. The split-tile assertion at line 234 stays a bare sample on purpose —
it asserts an absence (toBe(0)), where polling would only wait for something
that must never appear.
The cross-session sidebar specs asserted a state that could expire before
they looked at it, making them the flakiest tests in the suite — two reds
on unrelated PRs within three minutes on 2026-07-26.
Root cause, from the failing run's trace: the tests need a background
process that is still RUNNING after the agent turn finishes, but the
process was a fixed `sleep 5` racing two other clocks — the turn itself
(two model round trips plus a real subagent delegation) and the 4s
success linger before a finished task auto-dismisses. On a loaded runner
the "dot should appear" poll took 7.5s to see the dot; by then `sleep 5`
had already exited, `waitForFunction(finalText)` returned in 0.08s
because the turn was long done, and the next line — a bare synchronous
`.count()`, not a wait — sampled 0.
The process lifetime is now test-controlled: `createBackgroundReleaseHandle()`
mints a sentinel path, the scripted command blocks until that file
appears, and the test releases it exactly when it wants the dot to clear.
One clock instead of three, and the "turn done, process still running"
state is stable rather than a window to catch. The wait is bounded (60s)
so a forgotten release can't hang a worker, and `sleep 5` stays as the
default for callers that pass no handle.
No product code touched — E2E harness only.
Memory rows fell back to the auto-derived name, so they read as a bare
"Memory" / "Running memory" next to a generic icon. Add the tool to
TOOL_META with proper copy across all five locales, plus a solid brain
glyph in the existing Phosphor fill set.
A memory tool call rendered destructive-red with an alert glyph and, when
expanded, a raw dump of the whole args + result payload.
Two causes. The red came from toolStatus treating any error text as a
failure: the memory store rejects an over-budget batch so the agent can
retry a smaller one, which is routine bookkeeping, not something the user
acts on. Those now land on the existing amber warning tier.
The payload dump came from memory having no case in toolSubtitle or
toolDetailText, so both fell through to the generic stringify-everything
fallback. Both now surface just the human-readable line.
Reported in Discord by @spherohero: `sessions recover --allow-partial` copied
20,817 of 20,824 message rows, then orphan cleanup deleted every one of them
because no session row was salvageable. Final output: 0 sessions, 0 messages.
The salvage worked and then threw the result away -- the exact opposite of
what --allow-partial exists to do.
_cleanup_partial_orphans() removed dependent rows whose session_id had no
matching sessions row. That is correct when a few sessions are lost; it is
catastrophic when the sessions b-tree is damaged worse than messages, which
is the common shape (sessions is small and hot, messages is large).
Reproduced exactly: 500 readable messages, unreadable sessions -> 500
copied, 500 removed, empty output.
Now _reconstruct_missing_sessions() runs FIRST, inside the same transaction,
synthesizing a minimal row per orphaned session_id (only id/source/started_at
are NOT NULL). started_at comes from the earliest surviving message.
Placeholders carry source='recovered' and an explicit title so a fabricated
session can never be mistaken for an original. Same repro now retains all
500 messages under 1 reconstructed session.
Reconstruction is reported as LOSS, not a clean recovery: session metadata
(title, model, timestamps, cost) is genuinely gone even though the
conversation text survived. loss_detected=True, partial=True,
complete=False, with a warning naming the counts.
Also fixes total_removed_or_relinked, which summed every dict value and would
now have counted retained messages as removed.
Sabotage-verified: removing the reconstruction call restores the wipe and
fails the test. 942 targeted tests green.
A slash command in a ⌘T tab or split pane routes through the primary
chat's dispatcher, which read the FOREGROUND view's identity for two
decisions it had no business asking the foreground about.
Busy: the gate read `busyRef`, a mirror of whatever chat is on screen.
A brand-new tab with zero turns was told "session busy — message queued"
because an unrelated chat was mid-stream, and the converse let a
background send fire into a live turn.
Identity: the output writer bound to the foreground's stored session, so
a tab's transcript writes re-keyed its cache entry onto the primary's
stored id and its queued payload landed on the primary's queue — the
kickoff would then drain into the wrong conversation. `submitText` also
dropped an explicit target when it routed to a slash command, running a
queue drain's command against whatever was in front.
Read the target session's own published state for both. One shared
resolver so submit and slash cannot drift apart again.
A slash command runs against the session `resolveTargetSessionId` picks,
which is routinely not the session on screen — a tile, a route rebind, or
a session created by the call itself. Both prompt pipelines gated on
`busyRef`, the FOREGROUND view's busy flag, so one session's send was
gated on another session's turn: a stale foreground `true` (a warm resume
of a still-running chat leaves one behind) parked an idle session's
command on the composer queue and reported "session busy" about a session
doing nothing. The converse also leaked — a background send could fire
mid-turn while the foreground happened to be idle.
Read the published per-session state instead, falling back to the
foreground flag only when the target has no state yet (a just-minted
session whose first publish hasn't landed). One shared resolver so submit
and slash cannot drift apart again.
Two existing cases in tests/test_tui_gateway_server.py covered the row's cwd
contract and this change had to answer both.
`_persisted_session_cwd` reached through `_session_cwd`, which falls back to
the gateway-wide completion cwd when the session carries none. That belongs to
no session in particular, so a session that never had a directory was given
one. Read the session's own `cwd` instead.
The remaining case asserted that a terminal session's directory is discarded,
which is the behavior this branch deliberately changes. Split it in two: a
terminal session now records its workspace, and the desktop keeps the "No
workspace" default it was written to protect.
The preview window masked its top 28% to transparent while reasoning
streamed, so a gradient appeared over the thinking text and vanished the
moment the block finished. Remove the mask; the max-height window and
bottom-pinned scroll still keep the preview compact and following the
newest tokens.
Sessions started from the terminal were stored with no `cwd` and no
`git_repo_root`, so the sidebar had nothing to group them by and they never
appeared under their project — they fell into "No workspace" instead. On one
real profile this covered 690 of 1063 TUI sessions.
The row write persisted a cwd only when `explicit_cwd` was set, which happens
only if the user switches directory mid-session. The intent was to avoid
filing chats under whatever folder the app launched in, and that reasoning
holds for the desktop, whose launch directory is an artifact of how the bundle
was opened. It does not hold for a terminal session: the user cd'd into that
directory before running hermes, and it is where the agent's terminal runs.
Split the two cases behind one helper. An explicit pick is always persisted;
otherwise the launch directory is recorded for terminal-started sessions and
left unset for the desktop, preserving the existing "No workspace" default
there.
Existing rows are unaffected: they carry neither cwd nor git_repo_root, so
there is nothing to recover them from.
The sidebar's project overview put git checkouts with zero Hermes sessions
above the repos the user actually works in, and dragging a project into place
did not stick.
Two causes. The repo discovery payload folded `discovered_repos.last_seen`
into `last_active`, but `last_seen` is when the disk scan last saw the
directory, so every scanned checkout was stamped "just now" and outranked
real work. Activity is now session-derived only; a repo with no sessions
reports no activity.
The overview also applied the manual drag-order through `orderByIds`, which
floats every id missing from the saved order to the top. That is right for
sessions, where a new chat should not sink, but the overview keeps receiving
newly-scanned repos — so once the user dragged anything, each new discovery
jumped above their hand-picked list. Projects the user has not ordered now
keep their deterministic position: ones with real activity still surface on
top, zero-session discoveries sort below the ordered list.
probe-model-picker.mjs times pill-click → menu painted over N rounds;
profile-model-picker.mjs wraps one open in a CPU profile with a
top-self-time table. Both attach to the perf:serve instance.
Each kept-alive tab whose model hasn't resolved ticks a GlyphSpinner —
setInterval + setState + a React commit every ~80ms, per mounted tab,
for pixels behind the active one. Gate the interval on the pane's
visibility context; the visible tab keeps its spinner, hidden tabs
resume from frame 0 on reveal.
ModelMenuPanel mounts a ModelEditSubmenu per model row, and each body ran
its hooks and built its JSX eagerly on menu open — ~90 rows of switches,
radio groups, and preset lookups nobody hovered yet, the largest app-code
slice in the open-latency profile. Wrap the body in a child component under
SubContent so Radix's presence gate leaves it unrendered until the submenu
actually opens; open latency drops roughly in half on a 91-model catalog.
Closes the last false-pass window @helix4u flagged on #71779. He explicitly
said not to hold the PR for it; it is two lines, so worth doing rather than
leaving a known-soft assertion in a concurrency test.
connection_opened.wait(1.0) proved the connection had not opened, but not
that the connector thread had actually reached connect_tracked() -- an
unscheduled thread produces the same observation. The connector now sets
connect_attempted immediately before the blocking call, and the test waits
on that first, so "still blocked" means blocked at the lock rather than
not yet started.
15/15 stable at ~1.1s. Removed-lock sabotage still fails.
Second round of @helix4u review on #71779. Both findings reproduced before
fixing.
1. My previous fix turned a crash into SILENT DATA LOSS. Returning
status="missing" for a present-but-unusable state_meta looked like a safe
degrade, but _verify_recovered_database only escalates "failed"/"partial"
into a warning + loss_detected. Measured on the branch: a run that dropped
a real metadata table reported warnings=[], loss_detected=False,
partial=False, complete=True. Strictly worse than the ValueError it
replaced -- that at least failed loudly.
Now "failed" when the table exists but lacks key/value, "missing" only
when genuinely absent. The damaged case yields
warnings=['state_meta copy status is failed'], loss_detected=True,
partial=True, complete=False, while staying verified=True so the output
is still installable-with-review.
2. The race test I wrote had its own scheduling race: after the guard
released the lock, the racer could win before the main thread set the
release event, failing on a correct implementation. Rewritten per
helix4u's design -- copy runs in a worker parked inside the patched
copy, a second worker attempts connect_tracked(), assert it stays blocked,
release, assert it then opens. Deterministic and ~1.1s instead of 10s;
12/12 stable.
Sabotage-verified. Note the third scenario only failed after adding a
unit-level test: recover_session_database short-circuits on the inspection
result when state_meta is entirely absent, so the helper's absent-branch is
unreachable end-to-end and a regression there was invisible. Both statuses
are now pinned directly.
939 targeted tests green.
Post-merge follow-up to #71770. Both defects were found by @helix4u in review
and reproduced against merged main before fixing.
1. Check/use race in _copy_source_bundle (my bug, from the #71770 follow-up
commit). It called has_live_connection(), released the registry lock, and
only then ran shutil.copy2() over the bundle. A tracked connection could
open in that window; the copy's close() then cancels its POSIX advisory
locks -- the exact class #71724 closed. Measured on main: a racer thread
opened a connection mid-copy after blocking 0.000s.
Adds sqlite_safe_read.offline_file_access(), a context manager that holds
the connection-lifecycle lock across an entire multi-step raw access, and
routes the bundle copy through it. Same racer now blocks 10.0s until every
raw descriptor is closed. Any future raw read of a database file (hashing,
moving a bundle aside) should use this rather than a bare pre-check.
2. _copy_state_meta_salvage assumed a 'key' column. A damaged state_meta can
keep 'value' and lose 'key'; columns.index("key") then raised ValueError
and aborted the whole partial recovery. The mirror case (key without
value) would have copied key-only rows and reported the table complete.
Now requires both, matching the non-partial _copy_state_meta, so an
unusable optional table is recorded as missing/failed and --allow-partial
still recovers sessions and messages.
Both regression tests verified by sabotage: reinstating the bare pre-check
fails the race test, removing the key/value requirement fails the other.
937 targeted tests green.
The name-based sibling probe can't reach every dead worktree: a dir renamed away
from its repo's prefix (`hermes-salvage-drafts` next to `hermes-agent`) shares
nothing to trim back to, and a scratch dir under /tmp was never a worktree at
all. Those fall through to the path-only heuristic, which reports the cwd as its
own repo root, and Tier 2 promotes it to a top-level project — one that can't be
opened and can only be dismissed by hand.
Gate auto-project promotion on the directory still existing, threaded in as an
injected predicate to keep the builder pure. The guard keys on the directory,
not on git-ness, so a plain non-git folder that's still on disk keeps its
project; callers that can't stat (remote backends) omit it and keep every
candidate. A stale persisted `git_repo_root` gets the same treatment, so a
deleted repo can't resurrect from the recorded value alone.
`_probe_sibling_worktree` recovers a deleted `<repo>-<suffix>` worktree by
trimming its name back to a sibling that still resolves, but it only trimmed
the LEAF segment. A session's cwd is usually a subdir of the worktree
(`<repo>-<suffix>/apps/desktop`), whose basename shares nothing with the repo,
so the probe no-oped and the dead path fell through to the path-only heuristic
— which minted it as its own main repo root, and then as a top-level project.
Walk the ancestors, deepest first, with the probe budget shared across the whole
walk so a deeply nested cwd can't fan out into a probe storm.
Adds build_aux_picker_rows() + format_aux_picker_entries() to
hermes_cli/inventory.py and converts both auxiliary-task pickers to them,
so custom providers, exclusions, and picker visibility can no longer be
dropped one call site at a time.
The two salvaged commits each fixed one kwarg at these same two sites
(user/custom providers, then for_picker). Both were per-site patches, so
the next aux picker would have reintroduced the gap. The seam makes the
correct behaviour the default a new caller cannot forget:
- user `providers:` and saved `custom_providers:` entries
- `model_catalog.excluded_providers`, matching /model
- exhausted-credential-pool providers stay visible
- only the active custom endpoint is probed, so an offline saved local
server can't hang the picker
- the virtual `moa` row is excluded (auxiliary_client unwraps it to the
aggregator anyway, so offering it is a silently-rewritten choice)
The vision picker also gains the current-selection marker it never had.
Also threads for_picker through build_models_payload, which had no way to
express it despite list_authenticated_providers supporting it.
credential pool has entries but is entirely rate-limited (exhausted): rate
limits are per-model for many providers (e.g. Google Gemini), so an exhausted
key for model-A may still serve model-B, and the user should stay able to
select it. That fix wired `for_picker=True` into `list_picker_providers`.
The two sibling interactive pickers reached by `hermes model` still called
`list_authenticated_providers()` WITHOUT `for_picker=True`, so they kept
hiding exhausted-pool providers:
- `_aux_select_for_task` (hermes_cli/main.py) — the auxiliary-task
provider/model picker.
- `_configure_vision_provider_model` (hermes_cli/tools_config.py) — the vision
provider/model picker.
Both persist a per-task config the user runs *later* (once the momentary
cooldown clears), so hiding an exhausted-pool provider here is exactly the
Tests: two regressions assert each picker requests `for_picker=True` (they
omitted it before the fix); the existing picker/exhausted-pool suite still
passes (6 total).
2107b86024 flipped compression.in_place to True but left both explanatory
comments reading "Default False during rollout". The contradiction is
load-bearing: it is why two recent PRs (#71747, #48951) were built on the
premise that agent.session_id still rotates at every compaction.
Comment-only; no behavior change.
The salvaged predicate change (expires_at dropped from the WHERE clause)
had no test that failed without it — a sabotage run reverting it left the
suite fully green. Add the two missing cases:
- a refresher starved past its own TTL revives its still-unclaimed row
- a holder whose lock was legitimately reclaimed cannot resurrect it
Both verified to fail against the pre-fix predicate.
The compression lock was gated so that any holder on a Windows host was
assumed alive until the full TTL expired. A crashed holder therefore
stranded every other agent on the session for the whole lease window.
Probe liveness with psutil.pid_exists, which is safe on nt. Keep the
os.kill(pid, 0) path strictly for POSIX: on Windows signal 0 maps to
CTRL_C_EVENT (bpo-14484) and can kill the target's console group, so with
psutil absent the only safe answer stays 'assume alive' and let the TTL
run out.
Also carry the commit fence through cancelled compressions so an aborted
run leaves the lock reacquirable rather than half-held.
Auxiliary LLM calls routed through OpenRouter now carry the `session_id`
sticky routing key, so they pin to the same provider endpoint as the
conversation they belong to instead of routing independently.
`OpenRouterProfile.build_extra_body` sourced the key only from its explicit
argument. Auxiliary call sites — compression, title generation, vision,
web_extract, session_search, MoA slots — funnel through
`agent.auxiliary_client`, which has no session handle and never passes one.
Those calls sent NO sticky key at all, so OpenRouter fell back to hashing
the opening messages and each aux call could land on a different provider
than the conversation it served. Per OpenRouter's prompt-caching docs a
present `session_id` is used directly as the routing key and activates
stickiness on the first successful request rather than only after a cache
hit, so the missing key also delayed pinning for the main loop.
Resolve it from the ambient conversation contextvar first, explicit argument
as fallback — the same resolution the Nous Portal profile uses (f2f4df064d)
and the same one `nous_portal_tags` already used for the `conversation=` tag.
The ambient value is the session-lineage ROOT, so the key also stays stable
for installs that opt out of the default `compression.in_place: true` and
across delegate-subagent trees.
The xAI `x-grok-conv-id` cache-affinity header in `build_api_kwargs_extras`
had the identical gap and is fixed the same way; it remains model-gated to
`x-ai/grok-*` / `xai/grok-*`.
Fixes#70820. Credit to @webtecnica, who reported it and submitted #70883
first; that PR threaded `session_id` through `call_llm` to reach the same
goal. This takes the ambient-contextvar route instead because none of the
34 `call_llm` call sites currently pass a `session_id`, so the parameter
alone would not have changed any live request.
Tests: 2 cases in tests/providers/test_provider_profiles.py, both verified
to fail against the pre-fix behavior via a sabotage run. E2E-validated on a
real AIAgent driving the out-of-turn compaction path.
The prompt rail mounts in every chat surface, and a tab group keeps
inactive tabs mounted, so a background timeline was stringifying every
user prompt's full text on each store update — including on every
streamed assistant token, since the selector walked all messages — plus
running a scroll listener and a getBoundingClientRect per prompt against
a viewport nobody was looking at.
It now defers each piece until it can be seen. An inactive pane returns
before a single hook is declared (usePaneVisible, the same context the
hidden-tab transcript freeze uses), so no subscription is opened at all.
An active rail subscribes to prompt IDS rather than prompt text and
reads the transcript imperatively only when that signal changes, which
takes streaming off the derivation path entirely; an identical
derivation hands back the previous array so a filtered-out prompt
doesn't restart the measure effect. The offset pass bails below the
render threshold instead of measuring a rail that renders null, ahead of
the existing following-the-bottom fast path, and the hover popover keeps
its shell for the fade but builds its rows on first open.
N session tiles stacked as tabs, all mounted (keep-alive) and all
streaming concurrently through the real publishSessionState path — the
"several PR reviews at once" workload. Frame pacing + longtask metrics,
no backend or credits needed; the workload that exposed both fixes above
and the regression gate that keeps them fixed.
ThreadTimeline's scroll compute read getBoundingClientRect for every user
message per scroll frame; interleaved with React's streaming style writes
each read forced a full reflow — the hottest self-time frame in the
multitab profile (620ms over one 5-tab run). While the viewport is pinned
to the bottom the active prompt is simply the last entry, so answer from
data and save the layout reads for actual scrollback.
Keep-alive keeps every ever-active tab mounted, but each hidden tab's
ChatRuntimeBoundary still subscribed to its view's $messages — so every
streaming delta flush (~30x/s) re-rendered every busy tab's whole thread,
and five concurrent sessions dropped the app to a crawl.
Flow the pane layer's visibility down as PaneVisibleContext and gate the
$messages subscription on it: a hidden tab freezes its transcript (status
dots stay live through the separate status atoms) and catches up in one
commit on reveal, since subscribe fires immediately with the current value.
The Nous Portal profile publishes a top-level `session_id` that the Portal
uses as its sticky routing key, pinning a conversation to one upstream
endpoint so explicit Anthropic `cache_control` breakpoints stay warm
(those caches are instance-local, so a reroute cold-writes instead of reads).
`NousProfile.build_extra_body` sourced that key only from the explicit
`session_id` argument. Auxiliary call sites — compression, title generation,
vision, web_extract, session_search, MoA slots — funnel through
`agent.auxiliary_client`, which has no session handle and never passes one.
They carried the `conversation=` tag but NO sticky key at all, so each one
routed independently of the conversation it belonged to.
Resolve the key the way `nous_portal_tags` already resolves that tag:
ambient lineage-ROOT contextvar first, explicit argument as fallback. Fixes
every aux call site at once with no per-call-site plumbing.
Also publish the root in the `_compress_context` forwarder when nothing is
ambient. Out-of-turn entry points (`/compact`, the gateway `/compress`
command and its hygiene sweep, partial head compression) call it outside
`run_conversation`'s scope, so the summarizer's aux call ran untagged and
unkeyed; the caller's value is restored in `finally` so a compaction never
leaks its tag.
Scope note: this does NOT keep the compaction turn's own prompt cache warm.
Compaction replaces the history with a summary and rebuilds the system
prompt, so that request is a cold write on any endpoint — what a stable key
buys is the turns AFTER compaction reading the cache it wrote. Under the
default `compression.in_place: true` (#38763) the session id does not rotate
at compaction, so for the main loop the ambient root and the explicit
argument already agree; the ambient resolution additionally holds the key
stable for installs that opt back into rotating compaction and across
delegate-subagent trees.
Salvaged from #71747. Subsumes the aux-call half of #70883 (@webtecnica),
which threaded `session_id` through `call_llm` to close the same gap.
Tests: 4 cases in tests/agent/test_portal_tags.py; both fix halves verified
to fail against the pre-fix behavior via sabotage runs.
Follow-up to the partial-recovery salvage. _copy_source_bundle() raw-copied
the source state.db and its -wal/-shm sidecars with no live-connection check.
Copying a database file is an open()/close() on it, and close() cancels every
POSIX advisory lock the process holds on that file -- including a running
VACUUM's EXCLUSIVE lock (fe431651c5). hermes_state._backup_db_file already
refuses that situation; this path did not, leaving two policies for one
hazard. Verified against the merged guard: with a tracked connection open,
_backup_db_file returned None (refused) while _copy_source_bundle copied
anyway.
Recovery normally runs as its own short-lived CLI process against an
offline/quarantined file, so this should never fire in practice. It is a
consistency fix, not a live corruption path -- but it is exactly the drift
that reintroduces the bug later.
Regression test verified by sabotage: removing the guard fails it.
Addresses review findings on the previous commits. Three of them were real
defects I reproduced against my own head before fixing.
1. Check/use race (BLOCKING). read_header_bytes_preopen() checked
has_live_connection() under _live_lock, released it, then did the raw
open/read/close outside the lock; connect_tracked() opened before
registering. A thread could pass the "nothing is live" check, another
could open a connection and BEGIN IMMEDIATE, and the first thread's
close() then cancelled its POSIX locks -- the exact bug this guard
exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED).
_live_lock now spans all three lifecycle transitions: open+register,
unregister+close, and check+open+read+close.
2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's
read-only path opens file:/…/state.db?mode=ro; that string was fed to
Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of
the real Path could match, so read-only connections were invisible to
the guard and their locks cancellable. Reproduced with no forced
scheduling. Keys now come from PRAGMA database_list (canonical path),
with an explicit tracking_path override.
3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception
and retried an untracked plain connect, so any error silently disabled
the guard. Now only ImportError (scaffold installs without hermes_cli)
falls back; real failures propagate.
4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db()
and _backup_db_file() raw-read live databases; they now REFUSE when a
connection is live rather than warning. Losing a forensic copy beats
corrupting the database being rescued.
Custom factories are no longer rejected (that broke legitimate callers) nor
silently untracked -- the tracking close() is mixed into whatever factory is
in play, including when an opener substitutes its own after the fact.
WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was
confounded -- the clean WAL result came from 3.53.1, which carries both the
WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the
bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and
DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the
WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL
until a fixed runtime ships is the conservative call, and the WAL policy does
not belong in this root-cause fix.
Six sabotage runs confirm each new test fails when its defect is reinstated
(including two that initially did NOT -- the race test was rewritten to pause
inside the byte read, and a separate test added for the opener-substituted
factory path). 1124 targeted tests green.
Completeness pass over the previous commit.
Registry leak (would have silently disabled the guard): the first version
incremented on open but decremented only in SessionDB.close(). kanban's
connect() hands raw connections to callers who close them directly (4 sites),
so its counter only ever went up — after enough kanban operations every
byte-probe on that path would be refused forever, disabling zeroed-file and
header detection. Replaced manual track/untrack with a TrackedConnection
subclass that untracks in close(), the one method every close path goes
through. Verified across plain close, contextlib.closing, double close,
nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not
a close) correctly stays tracked.
Also: a caller-supplied factory now wins instead of raising TypeError on a
duplicate kwarg, since tracking is an optimisation for the probe guard, not a
precondition for opening the database.
Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert.
Audit notes:
- DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains
correct there; those users are protected by the raw-read fix, not by the
journal mode.
- The remaining whole-file reads are on genuinely offline artifacts:
_backup_db_file (DB won't open; bytes preserved for forensics),
_backup_corrupt_db (quarantine path, now warns if a connection is live),
and backup verification of snapshots. backup._safe_copy_db already uses
the SQLite backup API rather than a byte copy.
- The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped,
not process-scoped), so this is a Linux/macOS correctness fix; the change is
platform-neutral and safe on Windows.
Sibling tests updated: session recovery no longer expects a DELETE-mode
recovered DB, and the kanban WAL-fallback test patches the connect site that
actually runs now.
`hermes sessions optimize` could corrupt state.db. Root cause is Hermes,
not the SQLite WAL-reset bug (#69784).
close() on ANY file descriptor for a SQLite database cancels every POSIX
advisory lock the process holds on that file, including a running VACUUM's
EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed
live databases in several hot paths: the zeroed-state.db detector runs on every
SessionDB construction (and the gateway builds those constantly), and kanban's
post-commit invariant check ran after every COMMIT. While VACUUM rewrote the
file, those probes dropped its lock and let other processes write into it.
A/B against the real code, only variable being the raw read:
SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode
raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt
no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt
SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt.
After this change: 0/4 corrupt, 0 vacuum errors.
Because the upgraded runtime corrupts too, replacing the embedded SQLite does
not fix this class; and because DELETE is where it reproduces, #70055's
"force DELETE on vulnerable builds" mitigation steered users into the failing
mode. That gate is reverted here: vulnerable builds get WAL again and still
warn so operators can upgrade.
- add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the
existing connection instead of open()+seek(28); byte-level probes are
restricted to before any connection exists and refused once one is live,
with an explicit force= escape for offline artifacts (snapshots, archives)
- track live connections in SessionDB and kanban's connect so that guard is
enforced rather than merely documented
- kanban's torn-extend check now only applies under a rollback journal; in WAL
a committed page may still legitimately sit in the -wal file
- revert the force-DELETE WAL gate and update the tests that pinned it
Regression tests assert the behavioural contract (an external process stays
locked out across Hermes' inspection calls) and were verified to fail when the
old raw-open behaviour is restored.
