`list_sessions_rich` returns one recency-ordered window, so a pinned
conversation that hadn't been touched in a while simply wasn't in the
payload. The desktop's Pinned section resolves pins against the loaded
rows, so the pin rendered as nothing until something dragged the row
back onto the page.
A pin is a "this must always be reachable" statement, which makes
falling off the page a bug rather than a paging outcome. `include_pinned`
adds one bounded query for the rows carrying `pinned = 1` that the
window missed, reusing the page's own WHERE clause — an archived or
filtered-out conversation stays out, and a pin is never a filter bypass.
It runs before compression projection, so a back-filled root surfaces
under its live tip exactly like a row that made the page on its own.
Co-authored-by: hrnbld <260600092+hrnbld@users.noreply.github.com>
Co-authored-by: liuhao1024 <11816344+liuhao1024@users.noreply.github.com>
Co-authored-by: Tamaz-sujashvili <56168197+Tamaz-sujashvili@users.noreply.github.com>
Co-authored-by: ferminquant <14808645+ferminquant@users.noreply.github.com>
Follow-up on top of #65554 (@the3asic):
- Lower FTS5 'usermerge' to its minimum of 2 (persisted in the config
shadow table, applied once per SessionDB instance). Without this a
positive-rank 'merge' skips any level holding fewer than 4 segments
(SQLite FTS5 §6.8), so the fragmented-index case the cadence targets
never converges.
- Run up to _FTS_MERGE_COMMANDS_PER_PASS (4) bounded merge commands per
index per cadence, stopping early on the documented no-progress
signal (total_changes delta < 2). Each command is its own implicit
transaction, so the write lock is released between commands.
- Skip a missing messages_fts instead of raising: the chunked
optimize-storage rebuild legitimately drops + backfills FTS tables
while writers keep running; warning every 1000 writes for the whole
backfill window would be noise, and optimize_fts() has always
treated missing tables as skippable.
- Replace traced-SQL shape assertions with behavioral tests: real
fragmented-index convergence (automerge suppressed, 60 segments) and
a 3-segment below-default-usermerge compaction test that fails
against a bare positive-rank merge (sabotage-verified).
Validation on a sqlite3.backup() copy of a real 10.7 GB production
state.db (1.49M messages, 1.3 GB + 2.9 GB FTS shadow tables):
worst per-command write-lock hold 41.8 ms (was 9.2 s / 18.1 s per
index with 'optimize'), search results byte-identical, fts5
integrity-check and PRAGMA integrity_check clean, steady-state pass
0.0 ms.
Two log-spam bugs found in live gateway logs:
1. gateway_routing UNIQUE-constraint spam (261 warnings in one errors.log):
early builds of the #59203 routing-index migration created
gateway_routing with 'session_key TEXT PRIMARY KEY' and no scope
column. _reconcile_columns() ADDs the missing scope column but SQLite
cannot ALTER a primary key, so the shipped composite
PRIMARY KEY (scope, session_key) never lands on those databases. Both
write paths then fail on every save:
- save_gateway_routing_entry: 'ON CONFLICT clause does not match any
PRIMARY KEY or UNIQUE constraint'
- replace_gateway_routing_entries: 'UNIQUE constraint failed:
gateway_routing.session_key' whenever the same session_key exists
under another scope (e.g. test-suite scopes leaked into a live DB).
New _heal_gateway_routing_pk() rebuilds the table once with the
composite key, preserving rows (newest wins on collisions, NULL scope
coalesced to ''). Same one-time-heal pattern as the #51646 active-
column repair. Verified E2E against a copy of a real affected state.db.
2. pre_gateway_dispatch warned ''GatewayRunner' object has no attribute
'session_store'' and silently dropped the hook for every message on
partially-initialized runners (bare object.__new__ runners in tests,
and any future init-order change). Pass
getattr(self, 'session_store', None) so the hook always fires
(pitfall #17 pattern).
Both regression tests fail without their fixes (sabotage-verified).
- Route _get_read_conn through _connect_tracked_db so per-thread
read-only connections are registered with the POSIX lock-safety
guard (connect_tracked), matching the writer and existing read-only
paths. Without this, byte-level probes of state.db could close() an
fd that cancels locks held by an untracked read connection.
- Convert _search_unindexed_gap, _run_trigram_search, CJK-bigram FTS
search, and get_meta to _read_ctx — these are pure SELECT queries
called from search_messages that were still taking self._lock,
defeating the PR's contention fix for those paths.
- Add @pytest.mark.requires_wal to the 5 tests that assume WAL is
active. Hermes disables WAL on SQLite < 3.51.3 (WAL-reset bug),
so these tests fail on the venv's SQLite 3.46.0 without the marker.
- Remove unused 'time' import.
The gateway shares ONE SessionDB across every agent, so every recall/browse
read (session_search discover/scroll/browse, memory prefetch, title resolve)
queued behind every writer flush on self._lock — one Python lock in front of
a WAL database that natively supports concurrent readers. Measured convoy:
a 0.23s FTS query stretched to 112s and a browse flush to 137s while 6-8
concurrent turns flushed hundreds of tool results.
Fix: under WAL, read-only methods (get_session, resolve_session_by_title,
list_sessions_rich, get_messages, get_messages_around, get_anchored_view,
search_messages) run on a per-thread mode=ro connection via _read_ctx(),
taking no lock at all. Fresh read transactions begin per statement, so
read-your-committed-writes holds for flush-then-search patterns. Non-WAL
(NFS DELETE fallback) or read-conn open failure keeps the legacy locked
single-connection path, remembered per thread to avoid per-query retries.
list_recent_user_messages and the in-memory /retry + session.undo walkers
treated every role=user row as a real user turn. Timeline bookkeeping
(model_switch, async_delegation_complete, auto_continue, hidden) is stored
that way, so /undo soft-deleted from a marker and /retry re-sent opaque
bookkeeping text. Exclude display_kind the same way CLI resume counting and
the prompt.submit ordinal path do.
Replaces the O(N) list_sessions_rich histogram in /api/sessions/stats
with a single GROUP BY query, reducing response time from ~575ms to
<1ms on large databases.
Original PR #48921 by @liuhao1024. Salvage fixes based on review
feedback from teknium1 and @wernerhp:
1. Preserve try/except guard — a DB error still degrades to empty
by_source instead of failing the whole stats response.
2. GROUP BY COALESCE(source, 'cli') — the original GROUP BY source
could emit duplicate 'cli' keys (NULL group + literal 'cli' group)
that the dict comprehension silently dropped.
3. Add exclude_children=True — list_sessions_rich excludes subagent
runs, delegates, and compression continuations by default; the
bare GROUP BY counted all rows, inflating source counts.
Aggregate shape (exclude_children/include_archived/limit params)
adapted from closed duplicate #61120 by @mijanx.
Closes#48914
Co-authored-by: mijanx <mijanx@users.noreply.github.com>
Follow-up to PR #64171. The writer loop and flush_token_counts'
caller-drain both set _token_writer_busy BEFORE popping the queue —
that ordering is what makes flush's lock-free fast path (reads queue
then busy, no cond held) sound. _stop_token_writer's leftover drain did
it backwards (clear queue, then set busy), leaving a few-bytecode window
where a concurrent flush could observe 'empty and idle' and return True
with the popped batch still unapplied. Shutdown-only staleness, no data
loss — but the protocol now matches at all three drain sites.
Test: concurrent flush during a stop-drain mid-apply must time out
(False), never report drained.
Follow-up to PR #64171. Two writer-death hardening gaps:
- queue_token_counts only spawned the writer when the thread object was
None, so a writer that died from an unexpected exception could never be
replaced: deltas piled up on the uncapped deque until a reader's flush
drained them synchronously. Respawn on 'not thread.is_alive()' instead.
(atexit re-registration on respawn is safe: unregister removes all equal
bound-method registrations and the drain hook is idempotent.)
- _coalesce_token_deltas ran outside the per-delta try/except in
_apply_token_batch, so a merge bug (e.g. an unclassified future kwarg
summing None + 0) would escape and kill the writer thread. Wrap it and
fall back to applying the raw batch — coalescing is an optimization,
never load-bearing.
Tests: coalesce-failure fallback + dead-writer respawn.
Every API call in the tool loop persisted its token/cost delta by
calling SessionDB.update_token_counts() synchronously on the turn
thread — a BEGIN IMMEDIATE sessions UPDATE plus a session_model_usage
upsert, measured in production at p50 3.3ms / p95 70.4ms per call and
up to 299ms against a cold multi-GB state.db. The tool loop stalls for
that long between calls, N times per multi-tool turn.
SessionDB gains queue_token_counts(): same signature and semantics as
update_token_counts(), but the critical path is a deque append plus a
condvar notify. A lazily started daemon thread applies deltas in
enqueue order through the existing update_token_counts ->
_execute_write path, so the established self._lock / BEGIN IMMEDIATE /
jitter-retry discipline is unchanged. When a backlog forms, adjacent
same-route incremental deltas coalesce into one UPDATE: token and
api-call fields sum, cost fields sum None-preservingly (an all-None
run stays None so COALESCE keeps the stored value), and absolute=True
deltas never merge and act as ordering barriers. Route equality is
required for a merge because those fields feed COALESCE backfill, the
last-non-None-wins status fields, and the per-model usage attribution
key — a merged apply is row-equivalent to sequential applies.
Correctness and durability:
- flush_token_counts() gives read-your-writes to token/cost readers
(get_session, list_sessions_rich, _get_session_rich_row,
list_gateway_sessions, InsightsEngine.generate) — a plain attribute
check when nothing is queued. The writer sets its busy flag before
popping the queue so the lock-free fast path can never miss an
in-flight batch.
- update_session_model / update_session_billing_route /
update_session_meta write the sessions row synchronously, bypassing
the queue, so they flush it first: a still-queued first-of-session
delta carries the pre-switch route, and applying it after the switch
UPDATE would trip the first_accounted_route branch (api_call_count
== 0 plus a route mismatch) and resurrect the old model/provider.
- AIAgent._persist_session flushes at turn finalize and every
error-exit persist point; close() stops and drains the writer before
the WAL checkpoint; an atexit hook (registered on first enqueue,
unregistered on close so closed instances are not pinned until
interpreter exit) drains at shutdown. Worst-case crash loss is the
in-flight call's delta — the same window as the old inline write.
- A flush trusts a live stop-flagged writer (its loop drains before
exiting) and only drains on the caller's thread when the writer is
dead or never started, claiming the same busy flag so concurrent
flushes wait instead of racing an in-flight batch.
- After close() has stopped the writer, queue_token_counts applies the
delta inline instead of parking it on a queue nothing will drain; a
closed-connection failure then raises at the call site, which
already guards for it, exactly like the old synchronous path.
- Writer apply failures are logged and never raise into a turn; the
writer thread survives and keeps applying.
Call sites switched to the queue: the per-call site in
agent/conversation_loop.py and both codex app-server sites in
agent/codex_runtime.py. In-memory per-turn counters
(agent.session_estimated_cost_usd etc.) stay synchronous, so live turn
displays never see the queue.
Tests: tests/agent/test_async_token_accounting.py (19 tests: enqueue
ordering, absolute-as-barrier, backlog coalescing with exact sums,
coalesced-vs-sequential row equivalence, merge unit rules, None-cost
preservation, read-your-writes, flush vs stop-flagged/concurrent
drains, inline apply after writer stop, close/atexit durability,
_persist_session drain, writer failure isolation);
tests/run_agent/test_token_persistence_non_cli.py updated to the
queue_token_counts contract.
Port from Kilo-Org/kilocode#12508: a stray read-only state.db / -wal /
-shm (sudo run, restored backup, copied dotfiles) previously killed
SessionDB init with an opaque 'sqlite3.OperationalError: attempt to
write a readonly database' raised from deep inside _init_schema —
naming no file and no fix — and the obvious wrong 'fix' (deleting the
-wal) silently loses committed transactions.
New preflight_db_writability() runs before the first connection on both
DB open paths (SessionDB.__init__ and hermes_cli.kanban_db.connect):
- files inside the Hermes home tree are repaired with chmod u+rw (the
safe scope: Hermes owns them, and the OS makes chmod fail on files
the user doesn't own, which bounds the repair exactly);
- anything else (root-owned files, read-only mounts, custom paths)
fails fast with an error naming the exact file and the exact chmod
command, plus an explicit 'do NOT delete the -wal' warning;
- WAL sidecars are never deleted or truncated — once writable, the
normal open path checkpoints committed frames into the DB.
Proven live on main first: chmod 444 state.db -> SessionDB() raises the
opaque readonly error. With the fix: in-home DBs self-heal; out-of-home
DBs get the actionable message. Sabotage run confirms the integration
tests fail without the wiring (2 failed / 10 passed).
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
`hermes -c`/`--resume` (continue last session) resolved the globally
most-recently-used session, then cd'd into *its* recorded cwd. So running
`hermes -c` from repo A could land you in repo B's session — the session
you last touched anywhere, not the last one *here*.
Now `_resolve_last_session` scopes to the current workspace first: the git
repo root when CWD is inside a repo (so all sessions across its
subdirs/worktrees group together), else the CWD itself — matching the
`workspace_key` identity `hermes sessions list --workspace` already groups
on. It falls back to the unscoped global MRU when no session matches the
current workspace, preserving the old behaviour for fresh directories.
Adds `workspace_key` param to `SessionDB.search_sessions` and a
`_workspace_key_clause` SQL helper that mirrors `workspace_key()`: a row
matches when its `git_repo_root` equals the key, or (legacy rows without
git metadata) when its `cwd` is at or under it.
`preview` is the head of the first user message and the title fallback on
every surface — sidebar rows, pickers, exports, the desktop's sessionTitle().
An untitled /skill session therefore read `[IMPORTANT: The user has invoked
the "work" skill, indicatin...` wherever it appeared.
A scaffolded row now selects a wide enough excerpt to reach the typed
instruction (head + tail spliced for a long body) and shapes it through
describe_skill_invocation(). Because previews are computed on read, existing
sessions are corrected without a migration.
The six copies of the preview subquery and four copies of its shaping collapse
into one expression and one helper along the way, and the /rewind picker gets
the same treatment.
The compression lock was gated so that any holder on a Windows host was
assumed alive until the full TTL expired. A crashed holder therefore
stranded every other agent on the session for the whole lease window.
Probe liveness with psutil.pid_exists, which is safe on nt. Keep the
os.kill(pid, 0) path strictly for POSIX: on Windows signal 0 maps to
CTRL_C_EVENT (bpo-14484) and can kill the target's console group, so with
psutil absent the only safe answer stays 'assume alive' and let the TTL
run out.
Also carry the commit fence through cancelled compressions so an aborted
run leaves the lock reacquirable rather than half-held.
Addresses review findings on the previous commits. Three of them were real
defects I reproduced against my own head before fixing.
1. Check/use race (BLOCKING). read_header_bytes_preopen() checked
has_live_connection() under _live_lock, released it, then did the raw
open/read/close outside the lock; connect_tracked() opened before
registering. A thread could pass the "nothing is live" check, another
could open a connection and BEGIN IMMEDIATE, and the first thread's
close() then cancelled its POSIX locks -- the exact bug this guard
exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED).
_live_lock now spans all three lifecycle transitions: open+register,
unregister+close, and check+open+read+close.
2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's
read-only path opens file:/…/state.db?mode=ro; that string was fed to
Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of
the real Path could match, so read-only connections were invisible to
the guard and their locks cancellable. Reproduced with no forced
scheduling. Keys now come from PRAGMA database_list (canonical path),
with an explicit tracking_path override.
3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception
and retried an untracked plain connect, so any error silently disabled
the guard. Now only ImportError (scaffold installs without hermes_cli)
falls back; real failures propagate.
4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db()
and _backup_db_file() raw-read live databases; they now REFUSE when a
connection is live rather than warning. Losing a forensic copy beats
corrupting the database being rescued.
Custom factories are no longer rejected (that broke legitimate callers) nor
silently untracked -- the tracking close() is mixed into whatever factory is
in play, including when an opener substitutes its own after the fact.
WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was
confounded -- the clean WAL result came from 3.53.1, which carries both the
WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the
bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and
DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the
WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL
until a fixed runtime ships is the conservative call, and the WAL policy does
not belong in this root-cause fix.
Six sabotage runs confirm each new test fails when its defect is reinstated
(including two that initially did NOT -- the race test was rewritten to pause
inside the byte read, and a separate test added for the opener-substituted
factory path). 1124 targeted tests green.
Completeness pass over the previous commit.
Registry leak (would have silently disabled the guard): the first version
incremented on open but decremented only in SessionDB.close(). kanban's
connect() hands raw connections to callers who close them directly (4 sites),
so its counter only ever went up — after enough kanban operations every
byte-probe on that path would be refused forever, disabling zeroed-file and
header detection. Replaced manual track/untrack with a TrackedConnection
subclass that untracks in close(), the one method every close path goes
through. Verified across plain close, contextlib.closing, double close,
nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not
a close) correctly stays tracked.
Also: a caller-supplied factory now wins instead of raising TypeError on a
duplicate kwarg, since tracking is an optimisation for the probe guard, not a
precondition for opening the database.
Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert.
Audit notes:
- DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains
correct there; those users are protected by the raw-read fix, not by the
journal mode.
- The remaining whole-file reads are on genuinely offline artifacts:
_backup_db_file (DB won't open; bytes preserved for forensics),
_backup_corrupt_db (quarantine path, now warns if a connection is live),
and backup verification of snapshots. backup._safe_copy_db already uses
the SQLite backup API rather than a byte copy.
- The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped,
not process-scoped), so this is a Linux/macOS correctness fix; the change is
platform-neutral and safe on Windows.
Sibling tests updated: session recovery no longer expects a DELETE-mode
recovered DB, and the kanban WAL-fallback test patches the connect site that
actually runs now.
`hermes sessions optimize` could corrupt state.db. Root cause is Hermes,
not the SQLite WAL-reset bug (#69784).
close() on ANY file descriptor for a SQLite database cancels every POSIX
advisory lock the process holds on that file, including a running VACUUM's
EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed
live databases in several hot paths: the zeroed-state.db detector runs on every
SessionDB construction (and the gateway builds those constantly), and kanban's
post-commit invariant check ran after every COMMIT. While VACUUM rewrote the
file, those probes dropped its lock and let other processes write into it.
A/B against the real code, only variable being the raw read:
SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode
raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt
no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt
SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt.
After this change: 0/4 corrupt, 0 vacuum errors.
Because the upgraded runtime corrupts too, replacing the embedded SQLite does
not fix this class; and because DELETE is where it reproduces, #70055's
"force DELETE on vulnerable builds" mitigation steered users into the failing
mode. That gate is reverted here: vulnerable builds get WAL again and still
warn so operators can upgrade.
- add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the
existing connection instead of open()+seek(28); byte-level probes are
restricted to before any connection exists and refused once one is live,
with an explicit force= escape for offline artifacts (snapshots, archives)
- track live connections in SessionDB and kanban's connect so that guard is
enforced rather than merely documented
- kanban's torn-extend check now only applies under a rollback journal; in WAL
a committed page may still legitimately sit in the -wal file
- revert the force-DELETE WAL gate and update the tests that pinned it
Regression tests assert the behavioural contract (an external process stays
locked out across Hermes' inspection calls) and were verified to fail when the
old raw-open behaviour is restored.
Reviewer egilewski identified that the 5s lock timeout fell open:
quarantine_zeroed_state_db() logged 'proceeding without the cross-
process lock' and continued to re-check + rename state.db. A slow or
paused startup that still owns the lock can overlap this fallback and
the two processes can again act on the same live file.
Fix: fail closed — return None without moving the file when the lock
cannot be acquired within 5s. Log an error with recovery guidance
(restore from state-snapshots).
Test: test_quarantine_fails_closed_when_lock_held holds the cross-
process lock from a background thread, calls quarantine, and asserts
it returns None without moving the zeroed file.
Reviewer egilewski identified two recovery-loss paths:
Path A — quarantine race (hermes_state.py):
SessionDB checks state.db before quarantine_zeroed_state_db() without a
shared cross-process lock, and Path.rename() may replace an existing
destination. Two writable startups can therefore move the first
instance's newly created database over the .zeroed-*.bak, erase the
original damaged-file evidence, and replace the live database with
another empty one.
Fix: add a cross-process lock (msvcrt on Windows, fcntl on POSIX)
around quarantine_zeroed_state_db() with a 5s bounded timeout. Under
the lock: re-check is_zeroed_state_db (another process may have already
quarantined it and created a fresh DB), use a PID-suffixed unique
destination, and non-clobbering rename with counter fallback.
Path B — size-cap pruning gap (hermes_cli/backup.py):
_too_large() runs before failed-database tracking. With keep=1 and a
size cap, an oversized state.db is omitted while failed_dbs stays empty,
so automatic pruning deletes the older complete snapshot that may
contain the only recoverable database.
Fix: track oversized DB files in a new oversized_skipped list (both in
the directory walk and top-level file loop). The manifest now records
oversized_skipped. Pruning is suppressed when failed_dbs or
oversized_skipped is non-empty, preserving the older complete snapshot
as recovery source.
Tests:
- test_concurrent_quarantine_no_clobber: two threads racing on the
same zeroed state.db — verifies quarantine backup survives with
original bytes and live DB is valid.
- test_oversized_db_suppresses_pruning: keep=1 + oversized state.db
verifies the older complete snapshot is not pruned.
All 33 tests pass (3 zeroed_state_db + 25 TestQuickSnapshot + 5
quarantine_forensic_logging).
Hardening for the Windows zeroed-state.db class (#68474):
- Surface critical stdout when pre-update/quick snapshot cannot copy a
present *.db (was log-only; update still looked successful).
- Detect all-NUL SQLite header on SessionDB open, quarantine the bytes,
and open a fresh DB with recovery guidance to state-snapshots.
Does not claim storage-stack root cause.
Builds on @ms-alan's label fix: the negative figure had a second cause
that relabelling alone leaves in place.
`hermes sessions optimize-storage` reported "reclaimed -3820.1 MB" on a
database that had in fact shrunk 60% (25069 MB -> 9975 MB). Both figures
came from os.path.getsize(). In WAL mode a VACUUM's rewrite lands in the
-wal file, and the checkpoint that folds it back is REFUSED
(SQLITE_BUSY) while any other connection holds a read-mark — e.g. the
live gateway. So the main file still carried its pre-VACUUM size AND kept
growing while the command ran, making the after-figure larger than the
before-figure. The TRUNCATE checkpoint in close() lands after the caller
has already measured and printed.
- hermes_state.py: add SessionDB.logical_size_bytes() — page_count *
page_size, the size the file settles at once the WAL is folded back.
Correct immediately, readers or not; returns None when the connection
is gone so callers fall back to stat().
- hermes_state.py: best-effort TRUNCATE checkpoint after the optimize
VACUUM so the file settles promptly when nothing else holds the DB.
Documented as insufficient alone (busy under a live gateway) so the
stat() approach is not reintroduced.
- hermes_cli/main.py: both `optimize-storage` and `optimize` report via
logical_size_bytes(); fixing only the reported command would have left
the same bug class next door.
- hermes_cli/main.py: lift the contributor's inline label to a
module-level _size_delta_label() shared by both sites, so the "grew by"
wording is consistent and unit-testable without reading source.
The two halves are complementary: page accounting removes the phantom
negative, and "grew by" still covers a genuine increase when concurrent
session writes outweigh what the rebuild freed.
Verified: sabotaging logical_size_bytes() back to stat() fails the new
test with a 22 MB overstatement, reproducing the report. The test asserts
its own precondition (stat() must actually be lagging) so it cannot
silently stop exercising the bug.
Tests: 464 passed (test_hermes_state.py + the new label tests).
Co-authored-by: chenbin <h-chenbin@voyah.com.cn>
Follow-up cleanup for PR #71123:
- Extract getattr(args, 'lineage', 'single') == 'logical' to a local
(appeared 3x in the export block)
- Document that the double _collect_delegate_child_ids traversal in
delete_session is an intentional TOCTOU guard inside the write txn
export_session() reads through get_messages(), so before the read fix an
already-serialized string went straight back into _insert_message_rows() and
got re-dumped — an export/import round trip permanently corrupted the row.
Guard the three write paths the same way tool_calls already is: parse a
string argument before storing it, and drop metadata that isn't an object
rather than persisting something no reader can use.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: aml1973 <aml1973@users.noreply.github.com>
get_messages(), get_messages_around() and get_anchored_view() returned the
raw display_metadata column instead of the dict every caller expects. The
desktop paints a resumed transcript from the REST prefetch, which reads
through get_messages(), so any session holding an async_delegation_complete
event failed resume with "Cannot use 'in' operator to search for
'task_count'" — on every such session, not just corrupted ones.
Route all four read paths through one shared codec that also unwraps rows
carrying a second JSON layer, so sessions already broken on disk recover on
read rather than needing a migration.
Co-authored-by: Studio729 <Studio729@users.noreply.github.com>
Co-authored-by: aml1973 <aml1973@users.noreply.github.com>
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Add a persisted, backend-confirmed provider/model lock for Hermes
Browser and other session API clients. A confirmed lock is an
execution contract rather than response metadata:
- POST /api/sessions/{session_id}/model validates and persists a
confirmed browser_model_lock (advertised in /v1/capabilities)
- session chat + chat/stream consume the persisted lock on body-only
follow-up turns; a confirmed lock wins over an older gateway session
/model override and the session-persisted model
- a later successful session /model switch explicitly clears and
replaces the lock while preserving lineage markers (_branched_from)
and invalidating cached system-prompt model/provider metadata
- ordinary one-off request overrides never replace a confirmed lock
- provider-resolution failure fails closed as a typed provider-auth
error (controlled response, never global-credential reuse)
- confirmed locks disable the global fallback model chain
- the completed agent's actual provider/model must match the locked
route or the turn fails with a runtime-mismatch error
- responses carry sanitized runtime metadata reporting actual vs
requested provider/model and lock state
Rebased onto the provider-aware request routing (#70853) and
session-model parity (#70931) that landed since the original branch;
the lock now slots into that precedence chain as the top rung.
Salvaged from PR #61236 by @abundantbeing.
New sessions.auto_archive / auto_archive_days config: soft-hide (never
delete) sessions with no activity for N days, aging on last activity
rather than creation so an old-but-active chat is spared. Sweeps are
throttled through state_meta and fire from CLI startup, gateway startup
+ hourly housekeeping, and the serve/dashboard backend (opportunistic
on session list + an hourly lifespan ticker), so every surface honours
one setting.
A new pinned column (declaratively migrated) exempts sessions from the
sweep; PATCH /api/sessions/{id} accepts pinned and flips the whole
compression lineage as a unit, mirroring set_session_archived.
profile_name was only written on the agent's initial lazy create
(e8b7ce8c1); every parented child row — compression rotation, TUI
/branch, desktop branch first-persist — was created without it. A
non-default profile's lineage therefore turned NULL on its first
compression or branch and aggregated as "default" in unified session
lists, completing the cross-profile session-jump.
Fix the class at the DB layer: _insert_session_row's parent backfill now
COALESCEs profile_name from the parent alongside cwd/git_* (#64709
pattern), so any parented child inherits its lineage's owning profile.
Stamp it explicitly at the three create sites as well — compression
rotation (mirroring _ensure_db_session), TUI session.branch, and the
TUI first-prompt row persist — so rows are self-describing even when the
parent row predates the profile_name column.
Hardening on top of the salvaged dead-PID lease reclamation from PR #65775
(@the3asic):
- Probe via psutil.pid_exists (hard dependency; CONTRIBUTING.md critical
rule #1) with the contributor's os.kill(pid, 0) POSIX probe retained
only as a scaffold-phase fallback when psutil is missing.
- Same-process holders (pid == os.getpid()) are never probed and never
self-reclaimed — another thread's live lease is owned by the lease
refresher/release path.
- Any probe doubt (exceptions, permission errors) conservatively keeps
the lease until normal TTL expiry; Windows stays TTL-only.
- Tests: psutil-first dead-pid reclaim (probe call pinned), os.kill
fallback path, probe-doubt keeps lease, same-pid no self-reclaim,
legacy holder + Windows paths assert NO probe via either API.
* fix(desktop): hide persisted agent-only history scaffolding
Filter verification-stop nudges and context-compaction handoffs at the
stored-history mapper boundary. Preserve a real reply when a compaction
handoff shares its stored message.
* test(desktop): build persisted E2E sessions through the real agent
Drive tui_gateway.entry over its stdio JSON-RPC transport against the mock
provider, wait for real completion events, and persist normal session history
through AIAgent and SessionDB. Migrate resume and hidden-history coverage,
including real compression and live verify-on-stop scaffolding, then remove
the unused direct SessionDB import scripts.
* fix(desktop): use the provisioned Python for real-session E2Es
Run the stdio gateway through uv's synced project environment outside the
Nix dev shell, while retaining the fully provisioned Nix Python when the
shell advertises HERMES_PYTHON_SRC_ROOT.
* fix(nix): expose the provisioned Python environment to uv
Mark the Nix-built Python environment active in the dev shell so the shared
E2E session builder can always run through `uv run --active --no-sync`.
* fix(timeline): persist typed display events
* fix(timeline): strip display-only fields from provider payloads, preserve through rewrites, fix /resume display history
Three review findings from PR #69771:
1. Provider payload leak: display_kind and display_metadata were forwarded
to the provider API as unknown message fields. Strict OpenAI-compatible
backends can reject the next request after a model switch or resumed
typed event. Strip both from the per-request api_msg copy in
conversation_loop alongside the existing api_content pop.
2. Rewrite/import data loss: _insert_message_rows preserved display_kind
but silently dropped display_metadata. After replace_messages,
archive_and_compact, or session import, async-delegation completion
events lost their task counts and fell back to generic display text.
Add display_metadata to the INSERT columns and bind tuple.
3. CLI /resume stale recap: startup --resume A set _resume_display_history
from A's lineage. A subsequent in-session /resume B loaded B only into
conversation_history via get_messages_as_conversation, leaving the stale
A display projection. _display_resumed_history preferentially read the
stale attribute, showing A's recap for B. Switch /resume to
get_resume_conversations and update _resume_display_history alongside
conversation_history.
Tests: 890 Python (5 files), 35 desktop TS — all green.
* feat(tui): render typed display events as ◈ markers in the Ink TUI
The TUI was not handling display_kind at all — model switch markers and
async delegation completions rendered as opaque user messages with the
full [System: ...] text, and hidden compaction handoffs were visible.
Wire display_kind through the full TUI chain:
- _history_to_messages (tui_gateway/server.py) forwards display_kind
and display_metadata to the gateway transcript payload.
- GatewayTranscriptMessage (gatewayTypes.ts) gains both fields.
- Msg.kind (types.ts) gains 'event' value.
- toTranscriptMessages (domain/messages.ts) maps:
- hidden → skip entirely
- model_switch → event "model changed"
- async_delegation_complete → event "N background agents finished"
(or "background agent work finished" without metadata)
- messageGroup (blockLayout.ts) routes event to its own group, with
SELF_SPACED + PAINTS_TRAILING_GAP so it owns its margins.
- messageLine.tsx renders event-kind as a dim ◈ marker with no gutter,
matching the CLI's ◈ event rendering.
- 4 new TUI tests for hidden/model_switch/async_delegation mapping.
TUI typecheck: clean. TUI lint: 0 errors (2 pre-existing warnings).
TUI tests: 9 passed (1 pre-existing failure on main, unrelated).
The anti-thrash guard (_ineffective_compression_count) was in-memory
only: a fresh compressor bound to a resumed, already-compacted session
started with compression_count=0 and a disarmed guard, so a
near-threshold session could legally re-compact once per process
restart, forever.
Persist the counter through the durable session-state channel,
mirroring the failure-cooldown (#54465) and fallback-streak (af7dceaf7)
pattern:
- hermes_state.py: sessions.compression_ineffective_count column
(declarative reconciliation adds it on existing DBs) +
get/set_compression_ineffective_count accessors.
- context_compressor.py: every strike/clear verdict routes through
_record_ineffective_compression_verdict() which writes through to the
session row (no-change verdicts skip the DB write);
bind_session_state() loads the persisted value; the compression
rotation boundary carries the counter onto the child row;
update_model()'s reset also clears the durable copy; the
ineffective-only fast path in _automatic_compression_blocked() is
removed because the counter is now durable and another agent's clear
must unblock a stale local snapshot.
- conversation_compression.py: _refresh_persisted_compression_guards
re-reads the counter alongside cooldown + fallback streak.
Reset semantics are unchanged: any real provider reading below the
threshold still clears the counter — and now clears it durably too.
Resolves the residual gap identified in #54923 by @lanyusea (the
second-threshold mechanism was superseded by persisting the existing
guard state).
Co-authored-by: lanyusea <lanyusea@gmail.com>
Consolidate the two near-identical warning strings in
_log_wal_reset_bug_once into a single logger.warning call with an
action variable. Remove overengineered defensive tuple-length handling
in is_sqlite_wal_reset_vulnerable (sqlite3.sqlite_version_info always
returns a 3-tuple). Remove extra blank line.
Follow-up cleanup for PR #69981.
On vulnerable SQLite (e.g. 3.50.4), do not enable WAL for fresh/non-WAL
shared databases — prefer DELETE instead. Leave existing on-disk WAL
alone (no live downgrade under concurrent gateway/cron openers). Surface
Python/SQLite version details as a doctor warning (#69784).
A pure-Latin query (no CJK characters) routes to the unicode61
`messages_fts` table, whose tokenizer does not insert a boundary between
Latin letters and adjacent CJK characters. Content like "修改youer服务端" is
indexed as a single token, so `search_messages("youer")` returned zero
results even though the substring is present, and the Latin path had no
fallback.
Add a zero-result trigram fallback to the pure-Latin path: when the
unicode61 search misses, retry against the existing `messages_fts_trigram`
table, which matches substrings regardless of word boundaries. The fallback
is gated on `_trigram_available` and on every token being >=3 chars (the
trigram minimum), and only fires on a zero-result miss, so successful Latin
searches keep their unicode61 ranking unchanged.
The trigram query construction shared with the CJK path is extracted into a
`_run_trigram_search()` helper; the CJK branch is refactored to use it with
no behavior change.
Adds regression tests in tests/test_hermes_state.py::TestCJKSearchFallback.
Integration layer for the cjk_unicode61 tokenizer, rebuilt on the v23
schema (the contributed integration in PR #65544 predated it):
- messages_fts_cjk: external-content FTS5 over a tool-row-excluding view
(same v23 storage discipline as the trigram index it supersedes — zero
inline text copies). Serves EVERY CJK query shape the legacy routing
split between trigram (>=3 chars/token) and LIKE full scans (1-2 char
tokens). Lone 1-char CJK runs and role_filter=['tool'] queries keep
their legacy routes.
- Dedicated marker pair (fts_cjk_rebuild_high_water/progress) gates the
id-scoped triggers, so a cjk-only backfill never gates the complete
messages_fts/trigram triggers.
- Transitions ride (the existing
throttled/resumable chunk engine): fresh DBs are born with the index;
legacy v22 DBs land on v23+cjk in one run; already-optimized v23 DBs
gaining the tokenizer get a marker-gated backfill; live writes are
indexed immediately in every case.
- Tokenizer-loss self-heal: a process that can't load the extension drops
the cjk triggers (writes keep working), leaves a stale breadcrumb, and
the index is rebuilt from scratch on the next optimize run — triggers
are never reinstalled over a gap (external-content 'delete' on an
unindexed rowid is the FTS5 corruption hazard the marker gating exists
to prevent).
- Capability classification: 'no such tokenizer: cjk_unicode61' joins the
degraded-runtime error class everywhere (read probe, write probe,
repair) so tokenizer absence is never misclassified as corruption.
- Config: sessions.cjk_fts (default on, inert without the .so) and
sessions.search_slow_ms in config.yaml, bridged to env by CLI + gateway
(startup + per-turn reload). build.sh falls back to vendored SQLite
headers so no libsqlite3-dev is needed.
Slow-query log path attribution updated: fts_cjk / fts5 / trigram /
like_scan. Tests: 14 lifecycle tests (fresh/legacy/stale/backfill paths,
tokenizer-loss round-trip) + 5 config-bridge tests + slow-log suite.
One INFO line per slow search naming the path taken (fts_cjk / fts5 /
trigram / like_scan), elapsed time, row count, and the query. The 2026-07
session_search investigation needed turn-trace archaeology plus workload
replay to discover that short-CJK queries were full-scanning the table —
with this line the next routing regression is a journalctl grep.
Threshold: sessions.search_slow_ms (default 1000ms; 0 logs every call),
bridged to HERMES_SEARCH_SLOW_MS.
Salvaged from PR #65544 (adapted to the v23 schema in follow-up commits).
Follow-ups on top of #64731's cwd/git_repo_root inheritance:
- git_branch joins the parent-row backfill (same NULL-only COALESCE hop):
the Desktop sidebar branch chip otherwise vanishes at every compaction
boundary even though the workspace didn't change.
- Belt-and-suspenders for #59527: compression forks (parent already ended
with end_reason='compression') also inherit the gateway origin columns
(user_id/session_key/chat_id/chat_type/thread_id/display_name/
origin_json) at DB-level child creation. The gateway re-records the peer
after rotation (d5b4879d4), but a hard crash in the window between child
creation and that write left the child unrecoverable by
find_latest_gateway_session_for_peer. Scoped to compression forks only —
delegate/subagent children (parent still live) must NOT inherit routing
keys, or peer recovery could repoint gateway traffic into a subagent's
session.
- Behavioral test driving the real _compress_context rotation path,
asserting the child row carries cwd/git_repo_root/git_branch and the
origin columns.
_insert_session_row never copied cwd/git_repo_root from a parent row when
parent_session_id was set, and git_repo_root wasn't even in the INSERT's
column list. The compression-fork path (and delegate/subagent spawns,
branch continuations) creates a child session without passing cwd/
git_repo_root at all, so the child's tip is born NULL — and since the
Desktop project sidebar groups sessions by cwd, the whole project silently
drops out of the sidebar every time a long conversation compresses. A
lineage that compresses repeatedly compounds this across generations.
Add git_repo_root to _insert_session_row's INSERT/COALESCE-on-conflict
column set, and backfill both cwd and git_repo_root from the immediate
parent row (single non-recursive hop, matching the existing COALESCE
"never overwrite an explicit value" contract) inside the same write
transaction whenever parent_session_id is set. A multi-generation chain
resolves correctly because each generation's own create_session call
already backfills from its (already-resolved) immediate parent.
Fixes#64709
