An ACP session registered the client's cwd for the *tools*
(`_register_task_cwd` -> `register_task_env_overrides`) but never pinned it
for the *prompt*. `agent/prompt_builder.py` reports
`Current working directory: {resolve_agent_cwd()}`, and `resolve_agent_cwd()`
reads the `_SESSION_CWD` contextvar, which ACP left unset — so it fell back
to `TERMINAL_CWD` / the launch dir.
The system prompt therefore advertised one root (commonly
`~/.hermes/workspace`, or the install tree) while the tools were rooted at the
editor's project. When the model emitted a *relative* path the tools resolved
it correctly; when it emitted an *absolute* path built from the advertised
root, the write landed outside the client's workspace and the turn still
reported success.
Observed in Zed/Buzz-shaped usage: the first prompt in a fresh workspace
creates the file under `~/.hermes/workspace/` and answers "Done." The client's
directory is untouched. Later sessions on the same cwd appear to work once a
session cwd record exists, which makes it look intermittent.
`_run_agent` already calls `set_session_vars(session_key=session_id)` inside
its `contextvars.copy_context()`, and that helper's `cwd` argument exists to
"pin the logical working directory for this context". It simply was not
passed. `gateway/session_context.py` and `tui_gateway/server.py` both already
pass it; ACP was the only surface that did not.
Adds a regression test asserting that the resolved cwd *during the turn* is
the cwd the client passed to `session/new`. It fails without this change
(resolving the install tree instead of the client's project).
- RelayAdapter.create_handoff_thread → one op-gated thread_create op
(Discord channel thread / Telegram forum topic / Slack named seed root);
None fallback contract preserved for the handoff watcher.
- RelayAdapter.rename_thread → thread_rename with only_if_current_name
no-clobber guard on the wire (connector enforces; Telegram guarded
renames fail safe). The native semantic-rename lane
(_is_discord_auto_thread_lane) lights over the relay via the
connector-stamped auto_thread_created/auto_thread_initial_name markers
parsed onto SessionSource in _event_from_wire.
- reply_to {text, author, is_own} wire parse onto the SAME MessageEvent
reply-context fields native adapters populate.
- gateway/relay/command_manifest.py: the gateway-declared slash-command
manifest (native Discord tree mirror) sent on the DISCORD hello; the
connector reconciles Discord's global registration (additive field,
older connectors ignore).
- Contract doc §OutboundAction ops + Phase 4 semantics sections.
- 12 new tests (tests/gateway/relay/test_relay_threads.py); relay suite
266 passed.
The retry loop skipped only the current version, so on a uv whose download
catalog is stale it walked backwards through older patches. In #71250 the
newest indexed 3.11 is 3.11.14 — the installed version — so all five retries
went to 3.11.13..3.11.9, each a real download+install+probe+delete cycle that
the existing downgrade guard was always going to reject, before failing anyway.
Only newer patches can carry the SQLite fix. Skipping <= current makes the
stale-catalog case cost one attempt instead of six, and still finds 3.11.15
on the first retry when the catalog knows about it.
Fixes#71250.
`hermes update` could not repair the embedded Python runtime's
vulnerable SQLite (WAL-reset bug range 3.7.0-3.51.2, except backports
3.50.7/3.44.6) on installs where `uv python install <minor>` (bare
request, e.g. "3.11") resolves to an older cached/indexed patch that
still links a vulnerable SQLite build, even though a newer
non-vulnerable patch on the same minor line is available and known to
uv. The smoke test correctly rejected the vulnerable candidate every
time, but the provisioner gave up immediately after one attempt,
leaving the repair permanently stuck in the same failing loop on every
`hermes update` run.
Implements option D from the issue (query the index, retry with an
explicit newer patch), which the reporter identified as cleanest:
- New `_list_available_patches()`: runs `uv python list <minor>
--all-versions --only-downloads --output-format json --no-config`,
filters to cpython/default-variant entries (excluding pypy/graalpy),
and returns known patch versions newest-first. Fails safe (returns
[]) on any network/parse error.
- Refactored the single install+find+probe cycle out of
`_install_safe_python_generation()` into `_attempt_install_generation()`,
reusable per attempt with its own generation directory (so a
rejected candidate's files are fully cleaned up before the next
attempt, matching the existing --reinstall semantics).
- `_install_safe_python_generation()` still tries the bare minor-line
request first (preserves the original comment's rationale: for a
given exact patch, python-build-standalone may have no artifact with
fixed SQLite at all). If that resolves vulnerable, it now queries
`_list_available_patches()` and retries with explicit newer patches,
newest-first, bounded to `_MAX_PATCH_RETRIES` (5) attempts -- each
attempt is a real download+install+probe cycle, so the cap keeps
worst-case repair time bounded.
Sanity-checked `_list_available_patches()` against the real `uv`
binary (0.11.7) in this environment: correctly parses live `uv python
list --all-versions --output-format json 3.11` output, returns 14
patches sorted newest-first starting at 3.11.15.
9/9 new tests pass (retry succeeds with a newer patch, exhausts
gracefully when every known patch is vulnerable, empty patch list
degrades to None without crashing, retry count is bounded, plus direct
JSON-parsing unit tests for realistic/malformed/empty uv output);
47/47 in the full tests/hermes_cli/test_managed_uv.py file (including
the 3 pre-existing tests for the original bare-minor success path,
confirming no regression there).
A failed `hermes sessions repair` printed "keep state.db and the backup"
and stopped, so a user whose sessions had vanished had no way to discover
that a non-destructive recovery path exists — the reported dead end.
The failure branch now names the next command, read-only step first, seeded
with the backup path it just preserved. Covered by a CLI-surface regression
test that drives the real subprocess against an unrepairable database.
The Codex image backend rejected our own request shape for every account, and
we then translated that rejection into "Image generation is not enabled for the
current Codex account. Switch the image provider to OpenAI API key, FAL, or
xAI." — telling every affected user to abandon a provider that had never
actually been tried. That message is why this reads as a setup failure rather
than a bug: the wire error was replaced with a confident, wrong diagnosis.
Removes the classifier and its exception, so any HTTP failure surfaces
verbatim. The paired request-shape fix (previous commit) is what makes the
400 stop happening; this commit makes the next one diagnosable.
Also fixes error-body truncation: bodies were head-truncated at 500 chars, and
Codex error payloads can carry hundreds of bytes of leading metadata, so the
user got a wall of padding and no message. _summarize_error_body() prefers the
parsed error.message and falls back to a truncated raw body.
Docs: drop the unqualified image-to-image claim for the Codex backend and note
that the hosted tool call cannot be forced, so it is best-effort.
Verified E2E against a local fake Codex backend: success path writes a real PNG
with no tool_choice on the wire; the 400 path now returns api_error carrying
"Tool choice 'image_generation' not found in 'tools' parameter" (148 chars)
instead of the entitlement message. Sabotage run confirms all 4 regression
tests fail when the old behavior is restored.
Refs #19505, #49008, #31335.
The chatgpt.com/backend-api/codex backend 400s on every tool_choice
shape for the hosted image_generation tool — it looks up tool_choice as
a function name and never recognizes hosted-tool entries. Removing the
field from _build_responses_payload() lets the host model decide; the
instructions field nudges it toward the tool.
Salvaged from PR #19979 (originally targeted the old
client.responses.stream call, which no longer exists on upstream/main;
the live request now flows through _build_responses_payload + httpx in
_collect_image_b64).
`_prune_stale_worktrees` runs synchronously before the banner on every
`hermes -w` launch and shells out to git several times per candidate
worktree. On a repo with dozens of accumulated worktrees this dominated
startup: measured 18.5s of a 20.6s cold start, 11.6s on warm repeats.
The `git cherry` patch-equivalence probe was the single largest cost
(9.4s of 11.5s across 24 trees). It is also pure waste on repeat runs: a
tree preserved because it holds unpushed work is re-diff-hashed on every
launch, forever, always reaching the same verdict. On this repo 19 of 24
aged trees were unreapable, so ~11s per launch bought zero reaps.
Two changes, both verdict-preserving:
- Split the loop into a stat-only age filter, a parallel read-only
classification phase (thread pool, bounded to min(8, cpu_count)), and
a serial mutation phase. Only reads are concurrent; unlock/remove/
branch -D stay ordered, so log output and removal order are unchanged.
A pool failure falls back to serial rather than blocking startup.
- Memoize `git cherry` verdicts to
$HERMES_HOME/cache/worktree_merge_verdicts.json, keyed on the exact
`(base_sha, head_sha, max_ahead)` range the verdict was computed from.
Because that key is the complete input to the git call, a cache hit is
identical to recomputation by construction: if either ref moves, the
key changes and real git runs again. Bounded to 1000 entries, written
atomically, and a corrupt/hand-edited cache degrades to recomputation.
Measured on a 44-worktree repo (24 aged candidates):
_prune_stale_worktrees before 11.5s after 2.05s cold / 0.42s warm
hermes -w to banner before 13.9s after 1.79s
Work-preservation is unchanged: all 44 worktrees survived, and every
dirty/unpushed/live-locked guard still fires. Verified the 24 real-tree
verdicts are byte-identical across serial, cold-cache, and warm-cache
runs.
Tests: 75/75 in tests/cli/test_worktree.py (67 existing + 8 new). The
new cache tests were sabotage-verified — swapping the exact-sha key for a
naive path-only key makes two of them fail by deleting a worktree that
had gained unmerged work, which is the data-loss case the key prevents.
* fix(update): bind IsWow64Process2 HANDLE and fall back to GetNativeSystemInfo
The #71218 OS-native probe still rejected correct ARM64 Desktop rebuilds on
Windows-on-ARM when ctypes truncated GetCurrentProcess()'s pseudo-handle and
IsWow64Process2 failed with ERROR_INVALID_HANDLE, falling through to the
lying PROCESSOR_ARCHITECTURE=AMD64 value. Type the HANDLE correctly and use
GetNativeSystemInfo before the env-var fallback.
* test(update): cover WoA IsWow64Process2 handle failure and system-info fallback
Pin the residual #71218 shape where IsWow64Process2 returns FALSE, the env
arch lies as AMD64, and GetNativeSystemInfo must still report ARM64 so the
integrity gate accepts a correctly-built ARM64 Hermes.exe.
`sync_skills()` keys the bundled manifest by frontmatter name but computes
the destination from the bundled path. When upstream renames or
recategorizes a skill, the manifest key still matches while the new dest
does not exist yet, so the loop fell into its "in manifest but not on
disk" branch and misread the skill as user-deleted: the user's copy was
stranded at the old path forever and never received another update.
Three skills hit this in the July 2026 reorg (computer-use,
evaluating-llms-harness, serving-llms-vllm) — silently frozen at their
pre-rename content on every machine that ran `hermes update`.
Recovery only moves a stale copy when it is byte-identical to the origin
hash recorded the last time sync wrote it, which proves the directory is
ours rather than the user's work. User-modified copies are kept in place
with a warning, hub-installed paths are never touched, and a genuine
deletion (no copy anywhere on disk) is still respected.
- tools/skills_sync.py: add _recover_renamed_skill() plus the
_index_active_skills() / _read_hub_install_paths() indexes; call it
before classification and report moves via a new `relocated` key.
- hermes_cli/main.py: surface relocations in both `hermes update` skill
sync reporting sites.
- tests: 4 cases covering relocate, user-modified preservation,
hub-installed exemption, and genuine-deletion respect.
Some OpenAI-compatible endpoints — notably Tencent Copilot
(copilot.tencent.com) — only accept streaming chat requests; any
non-streaming call returns HTTP 400 (code 11101, 'Non-stream chat
request is currently not supported'). The main conversation loop already
streams, so interactive chat works, but every auxiliary task (title
generation, compression, web extraction) used the non-streaming path and
failed on each call.
_provider_requires_stream() detects stream-only endpoints
(copilot.tencent.com built in, plus user-configurable
auxiliary.stream_only_base_urls substring markers in config.yaml).
Matching sync auxiliary calls route through _create_with_progress
(force_stream=True) and async calls through the new
_acreate_with_stream, aggregating the chunk stream — including tool-call
deltas and reasoning deltas — into a complete response via the shared
_ChatStreamAccumulator.
Salvaged from PR #60686 by @kudi88 onto the progress-aware streaming
machinery from #71508, addressing both sweeper-review gaps: the async
path now consumes the stream with 'async for' (awaiting create() and
iterating synchronously raised on AsyncOpenAI streams), and tool-call
deltas are reassembled instead of dropped (MCP passes tools= through
call_llm). Under force_stream there is no silent non-streaming retry —
a stream-only provider rejects those by definition, so the original
error surfaces to the normal recovery chains.
- RelayAdapter.send_exec_approval / send_slash_confirm / send_clarify:
override the base text fallbacks with ONE platform-abstract `prompt` op
(connector renders Discord components / Telegram inline keyboards /
Slack Block Kit / WhatsApp buttons+lists). Option sets mirror the native
adapters exactly (once/session/always/deny with the same
allow_session/allow_permanent/smart_denied gating; once/always/cancel;
choices + Other). Clarify option ids are positional (c0..cN/other) —
choice text is arbitrary UTF-8, callback budgets are 64 bytes.
- Pending-prompt registry: gateway-minted 8-hex prompt ids →
{kind, session_key, extras}; one answer wins, lazy expiry, unanswered
prompts swept opportunistically. Wire timeout_s stays advisory.
- _consume_prompt_response (wired into _on_inbound AND the Discord
passthrough lane): routes answers to the SAME primitives the native
button handlers call — tools.approval.resolve_gateway_approval,
tools.slash_confirm.resolve, tools.clarify_gateway
resolve/mark_awaiting_text — then acks in-channel. Unknown/expired ids
fall through as command-shaped text (typed-reply degradation, the
relay's analog of the native 'approval expired' edit).
- Discord type-3 stub replaced: an hp1:<prompt>:<option> custom_id decodes
to a structured prompt_response (codec mirrored from the connector's
promptCodec); foreign custom_ids keep the legacy best-effort text shape.
- MessageEvent.prompt_response field + ws_transport wire parsing (additive).
- react ack lifecycle: on_processing_start/complete → `react` ops
(👀 → ✅/❌, remove-then-add), op-gated on supported_ops, best-effort by
contract (a react failure never touches the turn).
- Op gating throughout: a connector not advertising `prompt` gets
success=False from send_exec_approval/send_slash_confirm (run.py's text
fallback takes over — same contract as a failed native button send) and
the base numbered-text clarify; `react` silently no-ops.
- docs/relay-connector-contract.md §4: prompt / prompt_response / react
semantics (callback token, budgets, authorization-parity, foreign-id
behavior, per-platform react mappings).
- tests: tests/gateway/relay/test_relay_interactive.py (19) — option-set
rendering + gating matrices, registry consume-once/expiry, resolver
routing for all three kinds (monkeypatched primitives), fall-through
cases, Discord hp1 decode + foreign-id shape, react lifecycle
(success/failure/cancelled), op-gated/best-effort react.
Cross-repo pair: gateway-gateway 'Phase 3 interactive' PR (prompt/react
senders on all four lanes + interaction ingest).
The gateway's pre-agent session-hygiene compression killed the summary
call at a fixed 30s wall-clock deadline (compression.hygiene_timeout_seconds),
regardless of whether the summary model was hung or merely slow. A reasoning
model happily streaming a large summary was cut off mid-generation, the user
got '⚠️ Context compression timed out after 30.0s', and a 300s failure
cooldown left the session oversized — a doom loop for slow-but-healthy
auxiliary models.
Timeouts are now liveness-based instead of wall-clock-based:
- agent/auxiliary_client.py: new thread-local aux_progress_hook. When
installed (only by context compression today), the primary call_llm
attempt streams (stream=True) and aggregates chunks back into a complete
response, ticking the hook per chunk. The configured timeout then acts
per stream read (idle) instead of as a total budget. Providers that
reject streaming fall back to the plain non-streaming call; auth/payment/
rate-limit/transport errors propagate unchanged into the existing
recovery chains. Codex Responses (per SSE event) and Anthropic Messages
(per stream event, via the new create_anthropic_message on_stream_event
callback) tick the same hook from inside their wire adapters.
- agent/conversation_compression.py: CompressionCommitFence gains
touch_progress()/seconds_since_progress(); compress_context() installs
fence.touch_progress as the progress hook around the compress call.
- gateway/run.py: the hygiene wait loop treats hygiene_timeout_seconds as
an INACTIVITY budget — while the fence reports fresh progress the wait
extends, bounded by the new compression.hygiene_total_ceiling_seconds
(default 600s, clamped >= the idle budget) so a degenerate trickle
stream still dies. The timeout warning now says the summary model
produced no output, which is the only case that still triggers it.
- config/docs: hygiene_total_ceiling_seconds added to DEFAULT_CONFIG and
configuration.md; hygiene_timeout_seconds documented as inactivity-based.
Tests: tests/agent/test_aux_progress_streaming.py (hook plumbing, stream
aggregation incl. tool-call deltas and reasoning deltas, rejection
fallback, ceiling kill, fence progress surface); two new gateway tests
prove a slow-but-streaming worker survives past the fixed timeout
(sabotage-verified: fails with the old fixed deadline) and a
forever-trickling worker is still cut off at the ceiling.
Two tests fail deterministically on main depending only on which SQLite the
test interpreter links — nothing about the code under test. Both are green in
isolation and red in the suite / on an older library, the worst diagnostic
shape.
Root cause A — WAL is not always WAL. Hermes refuses journal_mode=WAL on
SQLite builds carrying the upstream WAL-reset corruption bug (3.7.0–3.51.2,
excluding backports 3.50.7 / 3.44.6) and falls back to DELETE. On such a
build NO -wal sidecar is ever created, so
test_wal_checkpoint_truncates_wal_file asserts on a file that cannot exist.
Invisible locally when the repo .venv and the Hermes managed runtime link
different versions (observed: .venv 3.50.4 → DELETE, runtime 3.53.1 → WAL),
so the same test passes for one interpreter and fails for the other.
- tests/conftest.py: add a `requires_wal` marker plus a
pytest_collection_modifyitems hook that skips such tests when the linked
library will fall back to DELETE. The skip reason names the actual
version so it is diagnosable rather than mysterious.
- pyproject.toml: register the marker.
- test_kanban_db_repair.py: mark the -wal-sidecar test.
Root cause B — process-global warn-once dedup. The WAL-fallback warning is
emitted at most once per (process, db_label). Any earlier test in
test_kanban_db.py that opens a kanban.db consumes that one-shot, so
test_connect_falls_back_to_delete_on_locking_protocol sees zero warnings and
fails — but only as part of the file, never alone.
- test_kanban_db.py: clear both dedup sets in the test that asserts on the
warning, with a comment explaining the isolation trap.
The gate deliberately does NOT import hermes_state. That module computes
DEFAULT_DB_PATH from get_hermes_home() at import time, so importing it during
collection — before the per-test _isolate_hermes_home fixture redirects
HERMES_HOME — permanently caches the developer's REAL ~/.hermes/state.db for
the whole session. The first version of this change did exactly that and made
tests read a live 31,881-session production database (test_console_engine
asserted "Total sessions: 2" and got 31881). The version predicate is
duplicated instead, and tests/test_conftest_wal_gate.py pins the two
implementations in agreement across every documented upstream boundary plus
guards against the import coming back.
Verified: on SQLite 3.50.4 the sidecar test SKIPS naming the version; on
3.53.1 it RUNS and passes, so coverage is not lost where WAL works. Clean
main fails exactly these 2 tests under
`scripts/run_tests.sh tests/hermes_cli/ tests/test_hermes_state.py`
(9926 passed, 2 failed); with this change the same scope is green.
Tests: 726 passed across the two kanban files, test_hermes_state.py, and the
new gate tests.
Three foot-guns in the canonical test runner, each of which cost real
debugging time by making an unverified run look verified.
1. Zero collection across the whole run reported success-shaped output.
Per-file rc=5 is rewritten to rc=0 so a platform-gated file (every test
skipped on this OS) doesn't fail the suite — correct, but it also meant a
run where NOTHING was collected anywhere printed
"0 tests passed, 0 failed (100% complete)" and, with no failures
recorded, could exit 0. Now the run-level guard counts every collected
outcome (passed/failed/skipped/errors/xfailed/xpassed): an all-skipped
file still passes, but zero-collected-anywhere prints an explicit
"✗ NO TESTS RAN — this is NOT a pass" block naming the likely causes and
returns 1.
2. A venv without pytest was selected merely for existing. The probe
accepted any directory with bin/activate, so in a checkout/worktree
without a local .venv it picked the RELEASE venv
(~/.hermes/hermes-agent/venv, no pytest). Every file then died with
"No module named pytest" and the run reported 0 tests. Candidates are now
import-checked for pytest — the same guard the HERMES_PYTHON fallback
already applied — and a skipped candidate is named on stderr.
3. Pytest node ids were silently discarded. This runner is file-granular,
so `tests/foo.py::TestBar::test_baz` isn't an existing path: discovery
dropped it and the run ended "No test files to run" while the selector
looked accepted. Node ids are now translated to the FILE plus an inferred
`-k` on the leaf name (parametrized ids reduced to the function name),
with a note explaining the translation. An explicit caller `-k` wins over
the inferred one.
Tests: 4 behavior contracts in tests/test_run_tests_parallel.py. Verified
by sabotage — reverting the runner fails 3 of the 4 (the fourth pins the
pre-existing all-skipped tolerance so fix 1 can't regress it).
- gateway/relay/media.py: RelayMediaClient for the connector's /relay/media
plane (upload local files → re-host reference for send_media; download
re-hosted inbound attachments → local temp paths). Same connector base URL
the WS dials, same per-gateway signed bearer as the upgrade (auth.py) —
no new configuration. stdlib urllib in a thread executor (no new deps);
25MB cap mirroring the connector's MEDIA_MAX_BYTES.
- RelayAdapter: send_image / send_image_file / send_voice / send_video /
send_document overrides route through ONE send_media op (media by
reference: local paths upload first, public URLs pass through). Gated on
supported_ops advertising send_media — legacy connectors keep today's
text fallbacks; connector declines/failed uploads degrade the same way.
Scope/user egress discriminators ride metadata exactly like send.
- Inbound: _localize_inbound_media downloads each media_urls entry to a
local temp path (native-adapter parity — vision/file tools consume
paths); dead re-host refs are dropped, public URLs survive a missing
client. Best-effort, never blocks handle_message.
- docs/relay-connector-contract.md §4: send_media op row + media
ingress/egress semantics (replaces the 'deferred to a later revision'
note). Additive within contract_version 1.
- tests: tests/gateway/relay/test_relay_media.py (15) — kind mapping,
upload-first path handling, op gating (explicit + legacy-empty),
decline/upload-failure fallbacks, scope metadata, inbound localization
matrix, client URL derivation/credential gating. Stub connector grew a
canned send_media result.
Cross-repo pair: gateway-gateway 'Phase 2 media parity' PR (re-host plane +
four ingress lanes + four platform send_media senders).
Follow-up for salvaged PR #70615 — the 2-button smart_deny case
(Allow Once + Deny only) was exercised by an existing test but only
at the flat-label level, not asserting the row pairing. Adds the
missing row-structure assertion using the same capture pattern as
the 4-button and 3-button tests.
Follow-up for salvaged PR #69380. The snippet
'export -p | grep -vE ... > $tmp.$BASHPID || true' attaches the redirect to
the grep pipeline segment, so $BASHPID expands inside grep's pipeline
subshell — a DIFFERENT pid than the parent shell that expands the follow-up
'mv $tmp' operand. The dump landed in an orphaned temp file, mv failed
silently (2>/dev/null), and the shared snapshot never updated again:
exported env / venv activation stopped persisting between commands
(tests/tools/test_local_shell_init.py TestSnapshotEndToEnd caught it).
Wrap the pipeline in a brace group and attach the redirect to the group so
the expansion happens in the current shell, matching mv's expansion. Also
rename the shell-init probe var HERMES_SESSION_ENV_PROBE ->
HERMES_STICKY_ENV_PROBE: it matched the HERMES_SESSION_ prefix the salvaged
fix now intentionally strips from snapshots, and the snippet-shape test is
updated to pin the brace-group contract.
A single long-lived backend serves many sessions through one
_active_environments['default'] LocalEnvironment (the messaging gateway, TUI,
and desktop/web dashboard all collapse the terminal to 'default'). That
environment persists a bash session snapshot file and sources it before every
command. 'export -p' dumped the FIRST session's HERMES_SESSION_ID into the
snapshot, so every LATER session sourced that stale value and its
'echo $HERMES_SESSION_ID' reported a FOREIGN session's id — overriding the
correct per-command Popen env injected by _inject_session_context_env.
Confirmed on staging-fp: session A read its own id, session B (same backend)
read A's id. Reproduced locally with two threads sharing one LocalEnvironment;
the snapshot held 'declare -x HERMES_SESSION_ID=...' from the first session.
Fix: strip the per-session bridged vars (HERMES_SESSION_* / HERMES_UI_SESSION_ID
/ HERMES_CRON_AUTO_DELIVER_*, i.e. gateway.session_context._VAR_MAP prefixes)
from the snapshot at both dump sites in base.py. They are re-injected fresh on
every command, so a snapshot should carry only the user's own shell state, not
Hermes' per-turn session identity.
Complements the _set_session_context session_id fix: that ensures the ContextVar
carries the right id; this ensures the shared snapshot can't override it with a
neighbour's. Adds tests/tools/test_snapshot_session_id_leak.py (regex unit +
real two-session LocalEnvironment integration) and updates the export-shape
assertions in test_base_environment.py for the new 'export -p | grep' dump.
_set_session_context() called set_session_vars() without session_id, so the
HERMES_SESSION_ID contextvar was set to "" (explicitly empty) on every
prompt.submit / session bind. The subprocess-env bridge
(_inject_session_context_env) treats an explicit "" as authoritative and does
NOT fall back to os.environ, so terminal/execute_code commands in a
dashboard/TUI/web session saw an empty $HERMES_SESSION_ID — even though
agent_init had populated it via set_current_session_id().
Every other set_session_vars() call site (gateway/run.py, cron, api_server)
passes session_id; tui_gateway was the only one that dropped it, affecting all
three of its frontends (Ink TUI, desktop, web).
Fix: derive the live id in the existing _sessions lookup loop
(agent.session_id, falling back to session_key — same derivation used at
session finalize) and pass it through to set_session_vars.
Adds tests/tui_gateway/test_session_id_injection.py covering agent id,
session_key fallback, and unknown/ephemeral key.
Reviewer egilewski identified that the 5s lock timeout fell open:
quarantine_zeroed_state_db() logged 'proceeding without the cross-
process lock' and continued to re-check + rename state.db. A slow or
paused startup that still owns the lock can overlap this fallback and
the two processes can again act on the same live file.
Fix: fail closed — return None without moving the file when the lock
cannot be acquired within 5s. Log an error with recovery guidance
(restore from state-snapshots).
Test: test_quarantine_fails_closed_when_lock_held holds the cross-
process lock from a background thread, calls quarantine, and asserts
it returns None without moving the zeroed file.
Reviewer egilewski identified two recovery-loss paths:
Path A — quarantine race (hermes_state.py):
SessionDB checks state.db before quarantine_zeroed_state_db() without a
shared cross-process lock, and Path.rename() may replace an existing
destination. Two writable startups can therefore move the first
instance's newly created database over the .zeroed-*.bak, erase the
original damaged-file evidence, and replace the live database with
another empty one.
Fix: add a cross-process lock (msvcrt on Windows, fcntl on POSIX)
around quarantine_zeroed_state_db() with a 5s bounded timeout. Under
the lock: re-check is_zeroed_state_db (another process may have already
quarantined it and created a fresh DB), use a PID-suffixed unique
destination, and non-clobbering rename with counter fallback.
Path B — size-cap pruning gap (hermes_cli/backup.py):
_too_large() runs before failed-database tracking. With keep=1 and a
size cap, an oversized state.db is omitted while failed_dbs stays empty,
so automatic pruning deletes the older complete snapshot that may
contain the only recoverable database.
Fix: track oversized DB files in a new oversized_skipped list (both in
the directory walk and top-level file loop). The manifest now records
oversized_skipped. Pruning is suppressed when failed_dbs or
oversized_skipped is non-empty, preserving the older complete snapshot
as recovery source.
Tests:
- test_concurrent_quarantine_no_clobber: two threads racing on the
same zeroed state.db — verifies quarantine backup survives with
original bytes and live DB is valid.
- test_oversized_db_suppresses_pruning: keep=1 + oversized state.db
verifies the older complete snapshot is not pruned.
All 33 tests pass (3 zeroed_state_db + 25 TestQuickSnapshot + 5
quarantine_forensic_logging).
Hardening for the Windows zeroed-state.db class (#68474):
- Surface critical stdout when pre-update/quick snapshot cannot copy a
present *.db (was log-only; update still looked successful).
- Detect all-NUL SQLite header on SessionDB open, quarantine the bytes,
and open a fresh DB with recovery guidance to state-snapshots.
Does not claim storage-stack root cause.
The #71119 integrity gate rejected CORRECT ARM64 desktop rebuilds on
Windows-on-ARM (#69179 follow-up): platform.machine() reports the PROCESS
architecture, and the update chain runs an x64 hermes-setup.exe / x64
Python under ARM64 emulation, so the gate compared the ARM64 Hermes.exe
against a phantom 'AMD64 host' and aborted the update.
_windows_native_machine() now asks the OS via IsWow64Process2 (whose
nativeMachine is truthful from emulated processes), falling back to
PROCESSOR_ARCHITEW6432/PROCESSOR_ARCHITECTURE (pre-1511 Win10) and then
platform.machine(). All three consumers — the integrity gate, the
packaged-exe arch preference, and backup validation — go through it.
Sabotage-verified: the three new regression tests fail against the old
process-arch behavior and pass with the fix.
Builds on @ms-alan's label fix: the negative figure had a second cause
that relabelling alone leaves in place.
`hermes sessions optimize-storage` reported "reclaimed -3820.1 MB" on a
database that had in fact shrunk 60% (25069 MB -> 9975 MB). Both figures
came from os.path.getsize(). In WAL mode a VACUUM's rewrite lands in the
-wal file, and the checkpoint that folds it back is REFUSED
(SQLITE_BUSY) while any other connection holds a read-mark — e.g. the
live gateway. So the main file still carried its pre-VACUUM size AND kept
growing while the command ran, making the after-figure larger than the
before-figure. The TRUNCATE checkpoint in close() lands after the caller
has already measured and printed.
- hermes_state.py: add SessionDB.logical_size_bytes() — page_count *
page_size, the size the file settles at once the WAL is folded back.
Correct immediately, readers or not; returns None when the connection
is gone so callers fall back to stat().
- hermes_state.py: best-effort TRUNCATE checkpoint after the optimize
VACUUM so the file settles promptly when nothing else holds the DB.
Documented as insufficient alone (busy under a live gateway) so the
stat() approach is not reintroduced.
- hermes_cli/main.py: both `optimize-storage` and `optimize` report via
logical_size_bytes(); fixing only the reported command would have left
the same bug class next door.
- hermes_cli/main.py: lift the contributor's inline label to a
module-level _size_delta_label() shared by both sites, so the "grew by"
wording is consistent and unit-testable without reading source.
The two halves are complementary: page accounting removes the phantom
negative, and "grew by" still covers a genuine increase when concurrent
session writes outweigh what the rebuild freed.
Verified: sabotaging logical_size_bytes() back to stat() fails the new
test with a 22 MB overstatement, reproducing the report. The test asserts
its own precondition (stat() must actually be lagging) so it cannot
silently stop exercising the bug.
Tests: 464 passed (test_hermes_state.py + the new label tests).
Co-authored-by: chenbin <h-chenbin@voyah.com.cn>
GitHub's current signup rules forbid consecutive hyphens, but legacy
accounts with them exist and are valid (Roger--Han, hit live during the
July 24 sweep — the mapping had to be written by hand). Accept any
alphanumeric/hyphen login that doesn't start or end with a hyphen.
#71184 (stream resume) upgraded agent-build-failure delivery from a bare
'error' event to a terminal message.complete frame (status=error,
recoverable) so failed turns replay on resume. The #71140 test pinned the
old event shape and broke on main where the two merged within the hour.
Contract unchanged: build failure must reach the client visibly.
Leg 2 of #63078: prompt.submit returns {"status":"streaming"} immediately and
runs _start_agent_build + _wait_agent(timeout=30s) behind it. The deferred
build (MCP discovery with per-server retry backoff, synchronous model-metadata
HTTP, skills scanning) routinely outlives 30s on cold starts; on timeout
run_after_agent_ready emitted an error EVENT and returned without ever calling
_run_prompt_submit — the user's first message was permanently discarded while
the build finished successfully in the background. The desktop's optimistic
row eventually cleared with no visible error: the blank first session.
New _wait_agent_for_prompt replaces the flat cliff for the deferred prompt
path only (_sess()'s RPC-blocking _wait_agent keeps its 30s contract):
- The pending prompt stays attached to the (already off-RPC) run thread and
is delivered the moment the still-running build completes — a slow build is
no longer message loss.
- The wait runs in 5s slices so a cancel (session.interrupt / churn) is
honored promptly; the cancelled path returns None and defers to the
caller's cancel branch (the #65567 emit) for user-visible messaging.
- Past 30s the client gets ONE keyed notification.show ('Still starting the
agent…', key=agent-build-slow, desktop toast / TUI status bar), cleared on
delivery — patient, never silent.
- Permanent failure only when the build itself fails: agent_error set at
ready, the build thread died without signalling ready (fail fast via the
new _agent_build_thread handle instead of sitting out the cap on a corpse),
or the bounded cap expired on a genuinely hung build. The cap defaults to
600s and is tunable via agent.build_wait_timeout in config.yaml (no new
env vars); the error message states the message was not sent.
Tests: slow-build delivery with zero error events; the keyed progress notice
shown once and cleared; build-failure surfacing exactly one error event with
the real reason; dead-thread fail-fast; cancel honored mid-wait; config
override + fallback semantics; cap expiry message. The compute-host fallback
test stubs the new waiter alongside _wait_agent.
run_after_agent_ready() silently returned when _turn_cancel_requested or
running=False was set during lazy agent startup, leaving the Desktop with
a {"status":"streaming"} reply that never produced a message.start or
error event. The _wait_agent error branch 6 lines above already emits;
this mirrors it so the client can surface feedback instead of hanging.
This is the server-side half of #63078 — the client-side drift guard is
addressed by #64327, but even with that fix the server-side cancel race
still silently dropped the turn.
Adds two regression tests that capture _emit (the existing sibling test
mocked it to a no-op, so it could not catch the silent drop).
Closes#63078
The post-update state.db integrity guard called verify_sqlite_integrity()
with max_bytes=0, which disables the size ceiling and forces a full
PRAGMA integrity_check. That pragma walks every b-tree page in the file,
so its cost scales with database size — measured on a real 30 GB state.db:
143.5s with a cold page cache (worse under an update's memory pressure),
with zero output on screen. The update looks hung right after
"✓ Code updated!" and a CPU sits pegged.
Multi-GB session databases are normal for heavy users, so a
size-unbounded check is never an acceptable default on the update path.
- verify_sqlite_integrity(): max_bytes now defaults to
DEFAULT_INTEGRITY_CHECK_MAX_BYTES (2 GiB) instead of 0. max_bytes=0
remains the explicit opt-in for a full scan.
- The oversized path no longer degrades to a header-only check: it adds a
constant-time structural probe (read-only open + schema_version +
sqlite_master read) so the malformed-schema class is still caught, not
just the #68474 zeroed-file signature.
- Drop the explicit max_bytes=0 at the post-update guard and in
copy_db_and_verify() so both inherit the bounded default.
Measured on the reporter's real 30 GB state.db: 143.5s cold → 0.001s,
still valid=True. Corruption detection verified at multi-GB scale for
both classes (zeroed header, malformed schema) — both still fail closed.
Tests: default-is-bounded invariant, oversized probe catches malformed
schema, max_bytes=0 still forces the full check.
The renderer's session-state cache is memory-only and the backend's
inflight snapshot dies with the backend process, so nothing survived a
full app or machine death mid-turn: reopening the session showed the
transcript up to the last committed turn and silently dropped everything
the crashed turn had streamed.
While a turn runs, the visible tail (user prompt + streamed assistant
rows, tool calls included) is now journaled to localStorage — throttled
off the delta-flush hot path, bounded (24 entries / 7 days), cleared the
moment the turn settles. Session resume folds the journaled tail back
onto the restored transcript. When the backend also has a live text-only
inflight projection for the same turn, the journal overlays its richer
structure onto that row (longer text wins, base row id kept so live
deltas keep landing) instead of treating it as caught up — the ordering
defect that dropped locally recorded tool progress in the original PR.
Co-authored-by: Omar Baradei <omar@kostudios.io>
A turn that ended in error cleared inflight_turn and emitted its terminal
frame in the same breath. If the client was disconnected during that window
(the exact case for a failure like a network drop), the frame went to the
detached drop-transport and the in-memory state was already gone — the
desktop reconnected to a session with no trace of the failure.
Failed turns now retain a compact error snapshot (user prompt, partial
assistant text, error, recoverable) that session.resume's inflight payload
carries to a reconnecting client. Covers all three loss sites: the
returned-error result path, the turn exception path (which now closes with
the same status:"error" message.complete frame shape instead of a bare
error event), and agent-init failure. The snapshot lives until the next
turn starts or the session closes; _run_prompt_submit replaces a retained
error leftover instead of appending onto it.
Co-authored-by: Reza Sayar <rsayar@uvic.ca>
Asked to link to a session, the agent had no way to know the @session
reference syntax exists — every mention in the tool schema described
consuming a link the user dropped, never writing one — so it answered
with the title and timestamp as prose and the desktop had nothing to
render.
Every result now carries a ready-to-copy `link`, and the schema says to
write it inline instead of restating the title around it. The profile
segment is omitted when the active profile can't be named confidently;
a bare id still resolves.
Also skip linkifying a ref a model already wrapped in a markdown link,
which would otherwise rewrite into a nested link.
A real APPLICATION_COMMAND interaction forwarded over the relay arrived
slash-less: _discord_interaction_to_event set text = data['name'] ("new",
not "/new"), MessageType.TEXT, and dropped options entirely — so a
registered /new dispatched as plain chat instead of a command
(MessageEvent.is_command() is text.startswith("/")).
Port the connector's Slack slash-command precedent (normalizeSlackCommand
builds `${command} ${args}`.trim() with a leading slash and explicit
command type): for type-2 interactions build "/" + name, append rendered
options space-separated (scalar options contribute their value, matching
the native adapter's f"/model {name}" shape; SUB_COMMAND/
SUB_COMMAND_GROUP contribute their name then recurse into nested
options), and set MessageType.COMMAND. Type-3 (custom_id) and other
interaction types are unchanged. This implements the interaction->command
sub-design previously flagged as deferred in the _on_passthrough
docstring.
Companion connector fix in gateway-gateway: fix(relay): strip own-mention
prefix so addressed slash commands dispatch.
Session previews are the first 60 characters of the first user message, so
persisting the @image: directives ahead of the caption labelled the session
with a truncated file path in the sidebar, session switcher, and command
palette. Clients lift the refs out of the body line by line, so moving them
after the caption changes nothing about how the turn renders.
A turn routed to a model that takes pixels directly sends `content` as a
parts list, and the session store deliberately ignores a plain-string
persist override for a list payload — a text override must not erase a
turn's image summary. So the override was dropped for every user on a
vision-capable main model, and the durable row kept only the caption plus a
literal `[Image attached at: ...]` / `[screenshot]`, which the renderer
cannot turn back into an image. Only vision-preprocessed (text-mode) turns
were actually fixed.
Mirror the shape instead: swap the text part for the `@image:` ref form and
keep the image parts, so the model still has the pixels for the rest of the
session, and drop the `[screenshot]` stand-in on the way into the bubble
when a ref was lifted from the same message.
The unquoted alternative in the directive pattern is `\S+`, so a ref built
by string interpolation truncates at the first space and strands the tail
as loose text next to a broken thumbnail. Composer images live in the app's
userData dir, which on macOS is `~/Library/Application Support/<App>/` — so
every pasted or dropped image hit this.
Adds format_reference_value next to REFERENCE_PATTERN, mirroring
formatRefValue in the desktop's directive-text.tsx, and covers the
round-trip through the parser.
The desktop gateway passed the vision-enriched, model-only message text
(carrying an `image_url:<path>` hint) straight into run_conversation as
the persisted user turn. The renderer only parses `@image:<path>`, so it
could not rebuild the attachment from history: after a restart the image
was gone and only the caption survived, and on a live session switch the
warm cache disagreed with the authoritative text and the frontend
"rescued" the image by appending it after the caption.
run_conversation already supports persist_user_message for exactly this
"what the model sees" vs "what gets stored" split; it was simply never
wired up for the attachment path.
Bug-class coverage for both fixes: the full catalogue survives Save, context
lengths are preserved, the key never lands in config.yaml on either write
path, blank clears it, a pre-fix plaintext key migrates while a ${VAR}
template is left alone, two endpoints on one host keep separate credentials,
and an IP-derived name is still a valid POSIX env var.
The two delete tests asserted on the plaintext mirror; they now assert the
same invariants against the credential reference.
The desktop self-update chain (Desktop -> hermes-setup --update ->
hermes update -> hermes desktop --build-only -> relaunch) rebuilds
Hermes.exe on the user's machine and declared success on bare file
EXISTENCE. A truncated PE (corrupt cached Electron zip / interrupted
extraction or rcedit rewrite / full disk) or a wrong-architecture
unpacked tree therefore shipped as the 'updated' app, which Windows
refuses to load with 'This app can't run on your computer'
(此应用无法在你的电脑上运行) — and the previous working build had
already been wiped by before-pack.mjs, leaving nothing to fall back to.
Fix, in three parts:
- hermes_cli/main.py: post-build integrity gate on Windows
(_ensure_desktop_exe_launchable). Parses the PE header of the freshly
built Hermes.exe — MZ/PE magic, section-table completeness vs file
size (catches truncation), and COFF machine vs the host arch (catches
arm64/x64 mixups). On failure it purges the (likely corrupt) cached
Electron zip, invalidates the content-hash build stamp so the
updater's retry-once genuinely re-downloads and rebuilds, restores
the previous build from the .bak tree when one exists (keeping the
corrupt tree as .corrupt for diagnostics), tells the user the update
was aborted and their old version kept, and exits nonzero.
_desktop_packaged_executable also now prefers a host-loadable PE over
pure newest-mtime when multiple win-*-unpacked trees coexist.
- apps/desktop/scripts/before-pack.mjs: on win32, the previous unpacked
tree is preserved as <appOutDir>.bak (only when it holds the product
exe — partial/corrupt trees still get the plain wipe) instead of
being destroyed, providing the rollback material for the gate above.
Non-Windows behavior is unchanged.
- Behavior-contract tests: tests/hermes_cli/test_desktop_exe_integrity.py
(23 tests — synthetic PE fixtures for truncation/non-PE/arch-mismatch,
rollback semantics, and the build-only exit contract) and 6 new vitest
cases in before-pack.test.mjs for the .bak preservation rules.
Progresses #69179
The strict cold-start readiness gate (#67498) means adapter.connect() no
longer returns True until the mocked start_polling records a successful
getUpdates round trip for its generation. Update the conflict-suite
Application mocks accordingly:
- fake_start_polling side effects call
adapter._record_polling_progress(adapter._polling_generation) on the
initial connect (retry generations intentionally do NOT auto-progress
where a test asserts the conflict count survives an unproven retry).
- _build_polling_app takes the adapter so its start_polling mock can
record progress.
Without this, the cold connects in these tests wait out the full 60s
readiness deadline and fail — which is exactly the fail-closed behavior
the gate is supposed to provide when polling shows no progress.
