A background queue drain (fromQueue: true) whose runtime binding was
reaped by the gateway fires with sessionId=null. The expression
options?.sessionId ?? activeSessionIdRef.current
falls back to whichever runtime id the foreground happens to hold,
landing the queued prompt in the session the user is currently viewing
instead of the session that owns the queue entry — a cross-session
message leak.
Guard the fallback: only inherit the foreground runtime when the drain
targets the current view (no storedSessionId, or it matches the
foreground). A background drain (storedSessionId differs) is left with
sessionId=null so the existing session.resume path rebinds the correct
runtime before prompt.submit fires.
Includes a regression test: "a fromQueue drain with null runtime id
does NOT land in the foreground session (cross-session leak guard)".
All 53 existing tests pass.
Follow-up for salvaged PR #69380. The snippet
'export -p | grep -vE ... > $tmp.$BASHPID || true' attaches the redirect to
the grep pipeline segment, so $BASHPID expands inside grep's pipeline
subshell — a DIFFERENT pid than the parent shell that expands the follow-up
'mv $tmp' operand. The dump landed in an orphaned temp file, mv failed
silently (2>/dev/null), and the shared snapshot never updated again:
exported env / venv activation stopped persisting between commands
(tests/tools/test_local_shell_init.py TestSnapshotEndToEnd caught it).
Wrap the pipeline in a brace group and attach the redirect to the group so
the expansion happens in the current shell, matching mv's expansion. Also
rename the shell-init probe var HERMES_SESSION_ENV_PROBE ->
HERMES_STICKY_ENV_PROBE: it matched the HERMES_SESSION_ prefix the salvaged
fix now intentionally strips from snapshots, and the snippet-shape test is
updated to pin the brace-group contract.
A single long-lived backend serves many sessions through one
_active_environments['default'] LocalEnvironment (the messaging gateway, TUI,
and desktop/web dashboard all collapse the terminal to 'default'). That
environment persists a bash session snapshot file and sources it before every
command. 'export -p' dumped the FIRST session's HERMES_SESSION_ID into the
snapshot, so every LATER session sourced that stale value and its
'echo $HERMES_SESSION_ID' reported a FOREIGN session's id — overriding the
correct per-command Popen env injected by _inject_session_context_env.
Confirmed on staging-fp: session A read its own id, session B (same backend)
read A's id. Reproduced locally with two threads sharing one LocalEnvironment;
the snapshot held 'declare -x HERMES_SESSION_ID=...' from the first session.
Fix: strip the per-session bridged vars (HERMES_SESSION_* / HERMES_UI_SESSION_ID
/ HERMES_CRON_AUTO_DELIVER_*, i.e. gateway.session_context._VAR_MAP prefixes)
from the snapshot at both dump sites in base.py. They are re-injected fresh on
every command, so a snapshot should carry only the user's own shell state, not
Hermes' per-turn session identity.
Complements the _set_session_context session_id fix: that ensures the ContextVar
carries the right id; this ensures the shared snapshot can't override it with a
neighbour's. Adds tests/tools/test_snapshot_session_id_leak.py (regex unit +
real two-session LocalEnvironment integration) and updates the export-shape
assertions in test_base_environment.py for the new 'export -p | grep' dump.
_set_session_context() called set_session_vars() without session_id, so the
HERMES_SESSION_ID contextvar was set to "" (explicitly empty) on every
prompt.submit / session bind. The subprocess-env bridge
(_inject_session_context_env) treats an explicit "" as authoritative and does
NOT fall back to os.environ, so terminal/execute_code commands in a
dashboard/TUI/web session saw an empty $HERMES_SESSION_ID — even though
agent_init had populated it via set_current_session_id().
Every other set_session_vars() call site (gateway/run.py, cron, api_server)
passes session_id; tui_gateway was the only one that dropped it, affecting all
three of its frontends (Ink TUI, desktop, web).
Fix: derive the live id in the existing _sessions lookup loop
(agent.session_id, falling back to session_key — same derivation used at
session finalize) and pass it through to set_session_vars.
Adds tests/tui_gateway/test_session_id_injection.py covering agent id,
session_key fallback, and unknown/ephemeral key.
Reviewer egilewski identified that the 5s lock timeout fell open:
quarantine_zeroed_state_db() logged 'proceeding without the cross-
process lock' and continued to re-check + rename state.db. A slow or
paused startup that still owns the lock can overlap this fallback and
the two processes can again act on the same live file.
Fix: fail closed — return None without moving the file when the lock
cannot be acquired within 5s. Log an error with recovery guidance
(restore from state-snapshots).
Test: test_quarantine_fails_closed_when_lock_held holds the cross-
process lock from a background thread, calls quarantine, and asserts
it returns None without moving the zeroed file.
Reviewer egilewski identified two recovery-loss paths:
Path A — quarantine race (hermes_state.py):
SessionDB checks state.db before quarantine_zeroed_state_db() without a
shared cross-process lock, and Path.rename() may replace an existing
destination. Two writable startups can therefore move the first
instance's newly created database over the .zeroed-*.bak, erase the
original damaged-file evidence, and replace the live database with
another empty one.
Fix: add a cross-process lock (msvcrt on Windows, fcntl on POSIX)
around quarantine_zeroed_state_db() with a 5s bounded timeout. Under
the lock: re-check is_zeroed_state_db (another process may have already
quarantined it and created a fresh DB), use a PID-suffixed unique
destination, and non-clobbering rename with counter fallback.
Path B — size-cap pruning gap (hermes_cli/backup.py):
_too_large() runs before failed-database tracking. With keep=1 and a
size cap, an oversized state.db is omitted while failed_dbs stays empty,
so automatic pruning deletes the older complete snapshot that may
contain the only recoverable database.
Fix: track oversized DB files in a new oversized_skipped list (both in
the directory walk and top-level file loop). The manifest now records
oversized_skipped. Pruning is suppressed when failed_dbs or
oversized_skipped is non-empty, preserving the older complete snapshot
as recovery source.
Tests:
- test_concurrent_quarantine_no_clobber: two threads racing on the
same zeroed state.db — verifies quarantine backup survives with
original bytes and live DB is valid.
- test_oversized_db_suppresses_pruning: keep=1 + oversized state.db
verifies the older complete snapshot is not pruned.
All 33 tests pass (3 zeroed_state_db + 25 TestQuickSnapshot + 5
quarantine_forensic_logging).
Address #68805 review: when failed_dbs is non-empty, skip
_prune_quick_snapshots so the incomplete snapshot does not delete
the older snapshot containing the last good database. The updater's
keep=1 would otherwise evict the recovery source.
Hardening for the Windows zeroed-state.db class (#68474):
- Surface critical stdout when pre-update/quick snapshot cannot copy a
present *.db (was log-only; update still looked successful).
- Detect all-NUL SQLite header on SessionDB open, quarantine the bytes,
and open a fresh DB with recovery guidance to state-snapshots.
Does not claim storage-stack root cause.
The #71119 integrity gate rejected CORRECT ARM64 desktop rebuilds on
Windows-on-ARM (#69179 follow-up): platform.machine() reports the PROCESS
architecture, and the update chain runs an x64 hermes-setup.exe / x64
Python under ARM64 emulation, so the gate compared the ARM64 Hermes.exe
against a phantom 'AMD64 host' and aborted the update.
_windows_native_machine() now asks the OS via IsWow64Process2 (whose
nativeMachine is truthful from emulated processes), falling back to
PROCESSOR_ARCHITEW6432/PROCESSOR_ARCHITECTURE (pre-1511 Win10) and then
platform.machine(). All three consumers — the integrity gate, the
packaged-exe arch preference, and backup validation — go through it.
Sabotage-verified: the three new regression tests fail against the old
process-arch behavior and pass with the fix.
Builds on @ms-alan's label fix: the negative figure had a second cause
that relabelling alone leaves in place.
`hermes sessions optimize-storage` reported "reclaimed -3820.1 MB" on a
database that had in fact shrunk 60% (25069 MB -> 9975 MB). Both figures
came from os.path.getsize(). In WAL mode a VACUUM's rewrite lands in the
-wal file, and the checkpoint that folds it back is REFUSED
(SQLITE_BUSY) while any other connection holds a read-mark — e.g. the
live gateway. So the main file still carried its pre-VACUUM size AND kept
growing while the command ran, making the after-figure larger than the
before-figure. The TRUNCATE checkpoint in close() lands after the caller
has already measured and printed.
- hermes_state.py: add SessionDB.logical_size_bytes() — page_count *
page_size, the size the file settles at once the WAL is folded back.
Correct immediately, readers or not; returns None when the connection
is gone so callers fall back to stat().
- hermes_state.py: best-effort TRUNCATE checkpoint after the optimize
VACUUM so the file settles promptly when nothing else holds the DB.
Documented as insufficient alone (busy under a live gateway) so the
stat() approach is not reintroduced.
- hermes_cli/main.py: both `optimize-storage` and `optimize` report via
logical_size_bytes(); fixing only the reported command would have left
the same bug class next door.
- hermes_cli/main.py: lift the contributor's inline label to a
module-level _size_delta_label() shared by both sites, so the "grew by"
wording is consistent and unit-testable without reading source.
The two halves are complementary: page accounting removes the phantom
negative, and "grew by" still covers a genuine increase when concurrent
session writes outweigh what the rebuild freed.
Verified: sabotaging logical_size_bytes() back to stat() fails the new
test with a 22 MB overstatement, reproducing the report. The test asserts
its own precondition (stat() must actually be lagging) so it cannot
silently stop exercising the bug.
Tests: 464 passed (test_hermes_state.py + the new label tests).
Co-authored-by: chenbin <h-chenbin@voyah.com.cn>
Closes#70146
When hermes sessions optimize-storage runs and the database grows
(rather than shrinks), the old code always printed '(reclaimed X MB)'
with a negative value. Fix by using a conditional label:
- positive delta → 'reclaimed X MB'
- negative delta → 'grew by X MB'
GitHub's current signup rules forbid consecutive hyphens, but legacy
accounts with them exist and are valid (Roger--Han, hit live during the
July 24 sweep — the mapping had to be written by hand). Accept any
alphanumeric/hyphen login that doesn't start or end with a hyphen.
#71184 (stream resume) upgraded agent-build-failure delivery from a bare
'error' event to a terminal message.complete frame (status=error,
recoverable) so failed turns replay on resume. The #71140 test pinned the
old event shape and broke on main where the two merged within the hour.
Contract unchanged: build failure must reach the client visibly.
Each case mounts a stacked group with a hidden tab whose geometry
matches the visible one — the arrangement that made the original bug
invisible to selector order.
Same ambiguity in three more lookups: blurComposerInput could blur a
hidden input and leave the focused one alone, the timeline scrolled
whichever viewport it found first, and a clarify card waiting in a
background thread swallowed the foreground composer's letter keys.
The drag snapshotted every chat surface and composer in the document,
so with a stacked tab group the link/split resolved against whichever
pane came first — usually a hidden one — and the @session chip landed
in a composer nobody was looking at.
Inactive tabs stay mounted and hidden with `visibility`, which preserves
their layout box — so a background tab answers a document-wide selector
with a rect identical to the visible tab's. Tag hidden layers and route
those lookups through visible-scoped query helpers.
Client half of leg 2. The issue's failure mode was the desktop clearing the
optimistic first message and showing nothing when the backend dropped the
turn. With the gateway now preserving the message across slow builds and
emitting a real error event on genuine failure, these tests pin the desktop
contract that makes that visible:
- an agent-init error event renders an in-transcript assistant error bubble,
keeps the user's optimistic first message (never silently cleared), fires
the global error toast, and releases busy/awaitingResponse so the composer
is usable again;
- the #65567 pre-ready cancel emit renders the same way.
Covers failAssistantMessage + the gateway-event error branch, which no test
exercised at the session-state level (todo-cleanup only asserted todo
eviction).
Leg 2 of #63078: prompt.submit returns {"status":"streaming"} immediately and
runs _start_agent_build + _wait_agent(timeout=30s) behind it. The deferred
build (MCP discovery with per-server retry backoff, synchronous model-metadata
HTTP, skills scanning) routinely outlives 30s on cold starts; on timeout
run_after_agent_ready emitted an error EVENT and returned without ever calling
_run_prompt_submit — the user's first message was permanently discarded while
the build finished successfully in the background. The desktop's optimistic
row eventually cleared with no visible error: the blank first session.
New _wait_agent_for_prompt replaces the flat cliff for the deferred prompt
path only (_sess()'s RPC-blocking _wait_agent keeps its 30s contract):
- The pending prompt stays attached to the (already off-RPC) run thread and
is delivered the moment the still-running build completes — a slow build is
no longer message loss.
- The wait runs in 5s slices so a cancel (session.interrupt / churn) is
honored promptly; the cancelled path returns None and defers to the
caller's cancel branch (the #65567 emit) for user-visible messaging.
- Past 30s the client gets ONE keyed notification.show ('Still starting the
agent…', key=agent-build-slow, desktop toast / TUI status bar), cleared on
delivery — patient, never silent.
- Permanent failure only when the build itself fails: agent_error set at
ready, the build thread died without signalling ready (fail fast via the
new _agent_build_thread handle instead of sitting out the cap on a corpse),
or the bounded cap expired on a genuinely hung build. The cap defaults to
600s and is tunable via agent.build_wait_timeout in config.yaml (no new
env vars); the error message states the message was not sent.
Tests: slow-build delivery with zero error events; the keyed progress notice
shown once and cleared; build-failure surfacing exactly one error event with
the real reason; dead-thread fail-fast; cancel honored mid-wait; config
override + fallback semantics; cap expiry message. The compute-host fallback
test stubs the new waiter alongside _wait_agent.
From PR #64327 (@Kenmege), whose target-aware drift predicate (compare route
tokens by their routed chat target; ignore selection null-resets, search/hash
churn, and background active-ref retargets; never count a move onto the
submit's own target) already reached main via the #69578 salvage
(1bdd478efa, Co-authored-by Kennedy Umege) and gained the #70986 composerScope
prong. What main covered only at the unit level (session-context-drift.test.ts)
is here pinned at the pipeline level, both directions:
- a send from a second chat rides out simultaneous programmatic churn
(selection null-reset from a gateway/profile reconnect, active-ref retarget
from a background event, search/hash-only route change from an overlay)
mid-session.resume and still reaches prompt.submit;
- a genuine user switch (selection AND route moving to another real chat)
mid-submit still aborts before prompt.submit.
Part of the #63078 fix branch.
Salvaged from PR #62805 (@floatingrain), whose diagnosis of the deterministic
frontend self-abort (createBackendSessionForSend mutating the selected ref +
route, then the caller's drift guard reading its own re-home as a user switch)
was correct — and correct about the sweeper misread that closed it: the
sweeper's implemented_on_main verdict cited submit.ts:245, which was inside
the session.resume block, while the raw post-create guard at the
createBackendSessionForSend site was still aborting every new chat on the
then-current main (4281151ae8). The mechanism itself has since landed via
8c288760d0 + 1bdd478efa, so what remains distinct is this regression: after
the pipeline adopts the created chat as its pinned target, a GENUINE user
switch (selection and route both moving to another chat during the
attachment-sync await) must still abort instead of being masked by the
re-baseline.
Part of the #63078 fix branch.
Salvaged from PR #62990 (@Roger--Han), a competing leg-1 fix for #63078. Its
mechanism (a pinned-route-token set accepting both the pre-commit and the
deterministic created-session token) was superseded by main's target-aware
drift predicate (1bdd478efa), but the PR pinned a timing case nothing on main
covered: React Router exposing the STALE new-chat route to the submit
continuation after create returns, committing the created session's URL only
before the next await settles. Both snapshots are the pipeline's own
transition — the first prompt must still reach prompt.submit.
Includes the contributors/emails mapping (added directly:
scripts/add_contributor.py's login regex rejects the consecutive hyphen in
the real GitHub login Roger--Han).
Part of the #63078 fix branch.
Salvaged from PR #62562 (@giggling-ginger). The mechanism that PR proposed
(adopt the session identity a first-message submit creates as the pipeline's
new pinned target; distinguish the expected new-chat route replacement from a
genuine concurrent switch) has since landed on main via 8c288760d0 and the
target-aware drift predicate (1bdd478efa / #69578). What main still lacked was
this PR's stronger regression coverage:
- The positive leg now asserts the FULL RPC transcript — exactly one
prompt.submit addressed to the created runtime id with the user's text, no
session.resume detour — instead of only 'not resume', and makes the creator
stub faithful to the real one (re-homes selection AND route, receives the
preview text used to seed the sidebar row).
- A new abort leg pins the route-moves-first sidebar-navigation race: the
route changes to a different chat while the selected ref still points at
the just-created session (navigate() commits before resumeSession() updates
the ref), which must still abort rather than deliver into the wrong chat.
Part of the #63078 fix branch.
run_after_agent_ready() silently returned when _turn_cancel_requested or
running=False was set during lazy agent startup, leaving the Desktop with
a {"status":"streaming"} reply that never produced a message.start or
error event. The _wait_agent error branch 6 lines above already emits;
this mirrors it so the client can surface feedback instead of hanging.
This is the server-side half of #63078 — the client-side drift guard is
addressed by #64327, but even with that fix the server-side cancel race
still silently dropped the turn.
Adds two regression tests that capture _emit (the existing sibling test
mocked it to a no-op, so it could not catch the silent drop).
Closes#63078
The post-update state.db integrity guard called verify_sqlite_integrity()
with max_bytes=0, which disables the size ceiling and forces a full
PRAGMA integrity_check. That pragma walks every b-tree page in the file,
so its cost scales with database size — measured on a real 30 GB state.db:
143.5s with a cold page cache (worse under an update's memory pressure),
with zero output on screen. The update looks hung right after
"✓ Code updated!" and a CPU sits pegged.
Multi-GB session databases are normal for heavy users, so a
size-unbounded check is never an acceptable default on the update path.
- verify_sqlite_integrity(): max_bytes now defaults to
DEFAULT_INTEGRITY_CHECK_MAX_BYTES (2 GiB) instead of 0. max_bytes=0
remains the explicit opt-in for a full scan.
- The oversized path no longer degrades to a header-only check: it adds a
constant-time structural probe (read-only open + schema_version +
sqlite_master read) so the malformed-schema class is still caught, not
just the #68474 zeroed-file signature.
- Drop the explicit max_bytes=0 at the post-update guard and in
copy_db_and_verify() so both inherit the bounded default.
Measured on the reporter's real 30 GB state.db: 143.5s cold → 0.001s,
still valid=True. Corruption detection verified at multi-GB scale for
both classes (zeroed header, malformed schema) — both still fail closed.
Tests: default-is-bounded invariant, oversized probe catches malformed
schema, max_bytes=0 still forces the full check.
Mid-turn progress lives only in process memory — the agent flushes to
SQLite at turn end — so an app/backend/machine death mid-turn lost the
turn entirely: reopening the session showed the recovered partial, but
the work never finished and the prompt itself survived nowhere durable.
Turns now write a durable marker (bounded per-profile sidecar) when they
start running and clear it when they conclude; success, handled error,
and interrupt all clear it, so a surviving marker is positive proof of a
process death. session.resume reads the marker: a fresh interruption
(desktop.auto_continue.freshness_minutes, default 15) is re-submitted
automatically as a continuation turn carrying the original prompt in an
interruption note, streams live to the client that just resumed, and
renders as a "resumed interrupted turn" event row. Stale markers are
cleared and the recovered partial speaks for itself; a turn that keeps
crashing stops auto-continuing after max_attempts (default 2) — the same
freshness + crash-loop-breaker posture as the messaging gateway's
restart auto-resume.
message.complete frames with status "error" were detected only by a text
regex heuristic, which misses the gateway's "Error: <detail>" texts and
partial-text failures — a failed turn rendered as a healthy reply. The
structured error/partial fields now drive the failure state: the bubble is
marked failed from the frame's error field, and a partial failure keeps
its streamed text visible instead of stripping it. session.resume's
inflight projection likewise carries a retained failure's error onto the
projected assistant row, so a failed turn recovered after a disconnect
renders as failed rather than as a healthy partial answer.
Co-authored-by: Reza Sayar <rsayar@uvic.ca>
The renderer's session-state cache is memory-only and the backend's
inflight snapshot dies with the backend process, so nothing survived a
full app or machine death mid-turn: reopening the session showed the
transcript up to the last committed turn and silently dropped everything
the crashed turn had streamed.
While a turn runs, the visible tail (user prompt + streamed assistant
rows, tool calls included) is now journaled to localStorage — throttled
off the delta-flush hot path, bounded (24 entries / 7 days), cleared the
moment the turn settles. Session resume folds the journaled tail back
onto the restored transcript. When the backend also has a live text-only
inflight projection for the same turn, the journal overlays its richer
structure onto that row (longer text wins, base row id kept so live
deltas keep landing) instead of treating it as caught up — the ordering
defect that dropped locally recorded tool progress in the original PR.
Co-authored-by: Omar Baradei <omar@kostudios.io>
A turn that ended in error cleared inflight_turn and emitted its terminal
frame in the same breath. If the client was disconnected during that window
(the exact case for a failure like a network drop), the frame went to the
detached drop-transport and the in-memory state was already gone — the
desktop reconnected to a session with no trace of the failure.
Failed turns now retain a compact error snapshot (user prompt, partial
assistant text, error, recoverable) that session.resume's inflight payload
carries to a reconnecting client. Covers all three loss sites: the
returned-error result path, the turn exception path (which now closes with
the same status:"error" message.complete frame shape instead of a bare
error event), and agent-init failure. The snapshot lives until the next
turn starts or the session closes; _run_prompt_submit replaces a retained
error leftover instead of appending onto it.
Co-authored-by: Reza Sayar <rsayar@uvic.ca>
Follow-up cleanup for PR #71123:
- Extract getattr(args, 'lineage', 'single') == 'logical' to a local
(appeared 3x in the export block)
- Document that the double _collect_delegate_child_ids traversal in
delete_session is an intentional TOCTOU guard inside the write txn
A real APPLICATION_COMMAND interaction forwarded over the relay arrived
slash-less: _discord_interaction_to_event set text = data['name'] ("new",
not "/new"), MessageType.TEXT, and dropped options entirely — so a
registered /new dispatched as plain chat instead of a command
(MessageEvent.is_command() is text.startswith("/")).
Port the connector's Slack slash-command precedent (normalizeSlackCommand
builds `${command} ${args}`.trim() with a leading slash and explicit
command type): for type-2 interactions build "/" + name, append rendered
options space-separated (scalar options contribute their value, matching
the native adapter's f"/model {name}" shape; SUB_COMMAND/
SUB_COMMAND_GROUP contribute their name then recurse into nested
options), and set MessageType.COMMAND. Type-3 (custom_id) and other
interaction types are unchanged. This implements the interaction->command
sub-design previously flagged as deferred in the _on_passthrough
docstring.
Companion connector fix in gateway-gateway: fix(relay): strip own-mention
prefix so addressed slash commands dispatch.
The unit tests cover each layer in isolation, but nothing exercised the whole
chain the bug lived in: the real gateway persisting an attachment, SessionDB
holding it after the process exits, and the renderer rebuilding a thumbnail
from the stored turn.
Seeds a session through the real gateway with an image attached, then launches
desktop against it — so the first render is already the relaunch case. Pins
native image routing (the majority path, and the one where a text-only persist
override is dropped) and stages the file behind directory and file names with
spaces, mirroring the macOS composer's Application Support path.
Session previews are the first 60 characters of the first user message, so
persisting the @image: directives ahead of the caption labelled the session
with a truncated file path in the sidebar, session switcher, and command
palette. Clients lift the refs out of the body line by line, so moving them
after the caption changes nothing about how the turn renders.
