The compression lock was gated so that any holder on a Windows host was
assumed alive until the full TTL expired. A crashed holder therefore
stranded every other agent on the session for the whole lease window.
Probe liveness with psutil.pid_exists, which is safe on nt. Keep the
os.kill(pid, 0) path strictly for POSIX: on Windows signal 0 maps to
CTRL_C_EVENT (bpo-14484) and can kill the target's console group, so with
psutil absent the only safe answer stays 'assume alive' and let the TTL
run out.
Also carry the commit fence through cancelled compressions so an aborted
run leaves the lock reacquirable rather than half-held.
Auxiliary LLM calls routed through OpenRouter now carry the `session_id`
sticky routing key, so they pin to the same provider endpoint as the
conversation they belong to instead of routing independently.
`OpenRouterProfile.build_extra_body` sourced the key only from its explicit
argument. Auxiliary call sites — compression, title generation, vision,
web_extract, session_search, MoA slots — funnel through
`agent.auxiliary_client`, which has no session handle and never passes one.
Those calls sent NO sticky key at all, so OpenRouter fell back to hashing
the opening messages and each aux call could land on a different provider
than the conversation it served. Per OpenRouter's prompt-caching docs a
present `session_id` is used directly as the routing key and activates
stickiness on the first successful request rather than only after a cache
hit, so the missing key also delayed pinning for the main loop.
Resolve it from the ambient conversation contextvar first, explicit argument
as fallback — the same resolution the Nous Portal profile uses (f2f4df064d)
and the same one `nous_portal_tags` already used for the `conversation=` tag.
The ambient value is the session-lineage ROOT, so the key also stays stable
for installs that opt out of the default `compression.in_place: true` and
across delegate-subagent trees.
The xAI `x-grok-conv-id` cache-affinity header in `build_api_kwargs_extras`
had the identical gap and is fixed the same way; it remains model-gated to
`x-ai/grok-*` / `xai/grok-*`.
Fixes#70820. Credit to @webtecnica, who reported it and submitted #70883
first; that PR threaded `session_id` through `call_llm` to reach the same
goal. This takes the ambient-contextvar route instead because none of the
34 `call_llm` call sites currently pass a `session_id`, so the parameter
alone would not have changed any live request.
Tests: 2 cases in tests/providers/test_provider_profiles.py, both verified
to fail against the pre-fix behavior via a sabotage run. E2E-validated on a
real AIAgent driving the out-of-turn compaction path.
The Nous Portal profile publishes a top-level `session_id` that the Portal
uses as its sticky routing key, pinning a conversation to one upstream
endpoint so explicit Anthropic `cache_control` breakpoints stay warm
(those caches are instance-local, so a reroute cold-writes instead of reads).
`NousProfile.build_extra_body` sourced that key only from the explicit
`session_id` argument. Auxiliary call sites — compression, title generation,
vision, web_extract, session_search, MoA slots — funnel through
`agent.auxiliary_client`, which has no session handle and never passes one.
They carried the `conversation=` tag but NO sticky key at all, so each one
routed independently of the conversation it belonged to.
Resolve the key the way `nous_portal_tags` already resolves that tag:
ambient lineage-ROOT contextvar first, explicit argument as fallback. Fixes
every aux call site at once with no per-call-site plumbing.
Also publish the root in the `_compress_context` forwarder when nothing is
ambient. Out-of-turn entry points (`/compact`, the gateway `/compress`
command and its hygiene sweep, partial head compression) call it outside
`run_conversation`'s scope, so the summarizer's aux call ran untagged and
unkeyed; the caller's value is restored in `finally` so a compaction never
leaks its tag.
Scope note: this does NOT keep the compaction turn's own prompt cache warm.
Compaction replaces the history with a summary and rebuilds the system
prompt, so that request is a cold write on any endpoint — what a stable key
buys is the turns AFTER compaction reading the cache it wrote. Under the
default `compression.in_place: true` (#38763) the session id does not rotate
at compaction, so for the main loop the ambient root and the explicit
argument already agree; the ambient resolution additionally holds the key
stable for installs that opt back into rotating compaction and across
delegate-subagent trees.
Salvaged from #71747. Subsumes the aux-call half of #70883 (@webtecnica),
which threaded `session_id` through `call_llm` to close the same gap.
Tests: 4 cases in tests/agent/test_portal_tags.py; both fix halves verified
to fail against the pre-fix behavior via sabotage runs.
Follow-up to the partial-recovery salvage. _copy_source_bundle() raw-copied
the source state.db and its -wal/-shm sidecars with no live-connection check.
Copying a database file is an open()/close() on it, and close() cancels every
POSIX advisory lock the process holds on that file -- including a running
VACUUM's EXCLUSIVE lock (fe431651c5). hermes_state._backup_db_file already
refuses that situation; this path did not, leaving two policies for one
hazard. Verified against the merged guard: with a tracked connection open,
_backup_db_file returned None (refused) while _copy_source_bundle copied
anyway.
Recovery normally runs as its own short-lived CLI process against an
offline/quarantined file, so this should never fire in practice. It is a
consistency fix, not a live corruption path -- but it is exactly the drift
that reintroduces the bug later.
Regression test verified by sabotage: removing the guard fails it.
Addresses review findings on the previous commits. Three of them were real
defects I reproduced against my own head before fixing.
1. Check/use race (BLOCKING). read_header_bytes_preopen() checked
has_live_connection() under _live_lock, released it, then did the raw
open/read/close outside the lock; connect_tracked() opened before
registering. A thread could pass the "nothing is live" check, another
could open a connection and BEGIN IMMEDIATE, and the first thread's
close() then cancelled its POSIX locks -- the exact bug this guard
exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED).
_live_lock now spans all three lifecycle transitions: open+register,
unregister+close, and check+open+read+close.
2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's
read-only path opens file:/…/state.db?mode=ro; that string was fed to
Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of
the real Path could match, so read-only connections were invisible to
the guard and their locks cancellable. Reproduced with no forced
scheduling. Keys now come from PRAGMA database_list (canonical path),
with an explicit tracking_path override.
3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception
and retried an untracked plain connect, so any error silently disabled
the guard. Now only ImportError (scaffold installs without hermes_cli)
falls back; real failures propagate.
4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db()
and _backup_db_file() raw-read live databases; they now REFUSE when a
connection is live rather than warning. Losing a forensic copy beats
corrupting the database being rescued.
Custom factories are no longer rejected (that broke legitimate callers) nor
silently untracked -- the tracking close() is mixed into whatever factory is
in play, including when an opener substitutes its own after the fact.
WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was
confounded -- the clean WAL result came from 3.53.1, which carries both the
WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the
bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and
DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the
WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL
until a fixed runtime ships is the conservative call, and the WAL policy does
not belong in this root-cause fix.
Six sabotage runs confirm each new test fails when its defect is reinstated
(including two that initially did NOT -- the race test was rewritten to pause
inside the byte read, and a separate test added for the opener-substituted
factory path). 1124 targeted tests green.
Completeness pass over the previous commit.
Registry leak (would have silently disabled the guard): the first version
incremented on open but decremented only in SessionDB.close(). kanban's
connect() hands raw connections to callers who close them directly (4 sites),
so its counter only ever went up — after enough kanban operations every
byte-probe on that path would be refused forever, disabling zeroed-file and
header detection. Replaced manual track/untrack with a TrackedConnection
subclass that untracks in close(), the one method every close path goes
through. Verified across plain close, contextlib.closing, double close,
nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not
a close) correctly stays tracked.
Also: a caller-supplied factory now wins instead of raising TypeError on a
duplicate kwarg, since tracking is an optimisation for the probe guard, not a
precondition for opening the database.
Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert.
Audit notes:
- DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains
correct there; those users are protected by the raw-read fix, not by the
journal mode.
- The remaining whole-file reads are on genuinely offline artifacts:
_backup_db_file (DB won't open; bytes preserved for forensics),
_backup_corrupt_db (quarantine path, now warns if a connection is live),
and backup verification of snapshots. backup._safe_copy_db already uses
the SQLite backup API rather than a byte copy.
- The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped,
not process-scoped), so this is a Linux/macOS correctness fix; the change is
platform-neutral and safe on Windows.
Sibling tests updated: session recovery no longer expects a DELETE-mode
recovered DB, and the kanban WAL-fallback test patches the connect site that
actually runs now.
`hermes sessions optimize` could corrupt state.db. Root cause is Hermes,
not the SQLite WAL-reset bug (#69784).
close() on ANY file descriptor for a SQLite database cancels every POSIX
advisory lock the process holds on that file, including a running VACUUM's
EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed
live databases in several hot paths: the zeroed-state.db detector runs on every
SessionDB construction (and the gateway builds those constantly), and kanban's
post-commit invariant check ran after every COMMIT. While VACUUM rewrote the
file, those probes dropped its lock and let other processes write into it.
A/B against the real code, only variable being the raw read:
SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode
raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt
no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt
SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt.
After this change: 0/4 corrupt, 0 vacuum errors.
Because the upgraded runtime corrupts too, replacing the embedded SQLite does
not fix this class; and because DELETE is where it reproduces, #70055's
"force DELETE on vulnerable builds" mitigation steered users into the failing
mode. That gate is reverted here: vulnerable builds get WAL again and still
warn so operators can upgrade.
- add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the
existing connection instead of open()+seek(28); byte-level probes are
restricted to before any connection exists and refused once one is live,
with an explicit force= escape for offline artifacts (snapshots, archives)
- track live connections in SessionDB and kanban's connect so that guard is
enforced rather than merely documented
- kanban's torn-extend check now only applies under a rollback journal; in WAL
a committed page may still legitimately sit in the -wal file
- revert the force-DELETE WAL gate and update the tests that pinned it
Regression tests assert the behavioural contract (an external process stays
locked out across Hermes' inspection calls) and were verified to fail when the
old raw-open behaviour is restored.
- require positive agent ownership proof before any background-review mutation
- fail closed when usage record is missing, malformed, or unreadable
- preserve foreground user-directed mutations and valid agent-owned curator flows
- cover patch, edit, delete, write_file, and remove_file end to end
Closes#67073
The background write guard decided ownership from `isinstance(usage_rec, dict)`,
so a local skill with NO usage record passed. That successful write called
bump_patch(), which created a `created_by: null` record — and the identical
write was refused from then on. "Allowed exactly once, then never" is a race
with our own bookkeeping, not a policy. Reproduced on main: patch #1 succeeds,
patch #2 with the same arguments is refused.
Option B from the issue. Option A (split `session_review` from
`scheduled_curator` and let the session fork patch user-owned skills it
consulted) would widen autonomous write permission onto skills the user owns
with no user present to consent — wrong direction for a no-user-present actor.
- skill_manager_tool: missing and explicit-null records now resolve
IDENTICALLY, both fail closed. The refusal names the reason and points at
`hermes curator adopt <name>`.
- background_review: both review prompts told the reviewer to patch any skill
consulted in the session and claimed pinned skills could be improved, while
enforcement refused both. Prompts now list pinned, external, and user-owned
skills as protected, and tell the reviewer to RECOMMEND adoption instead of
attempting a write that will be refused.
- skill_usage: document that `created_by` is a curator-management policy flag,
not a provenance claim, and add `is_curator_managed()` so call sites read as
the question they ask. Field name retained — it is on disk in every
`.usage.json` and renaming would strand those records.
- curator CLI: `hermes curator list-unmanaged` itemizes unmanaged skills with
the reason each is unmanaged (completes the #67139 spec).
Foreground writes are untouched: a user-directed edit to a user-owned skill
still works, including on pinned skills.
Sibling tests: 9 failures in test_skill_manager_tool.py were fixtures that
created record-less skills to exercise OTHER guards (consolidation-delete,
read-before-write) and relied on ownership falling through. Fixed at the
fixture, since the real curator only ever operates on managed sediment. One
test asserted the old "manually authored" wording; rewritten to assert the
behavior contract instead of the string.
Validation: 274 targeted tests + all 7 background-review files (60 tests) pass.
E2E on a temp HERMES_HOME (30 checks) covers the flip, foreground writes,
adoption unblocking, pin semantics, prompt/enforcement parity, and the new verb.
Each new test sabotage-verified: revert the fix, confirm it goes red.
Fixes#67140
The whole-prompt scan for "Current working directory:" matched USER project
content, not just Hermes' own host-info line. The prompt embeds AGENTS.md /
CLAUDE.md / .cursorrules in the context tier, which sits AFTER the host-info
block, and line_value() takes the LAST match — so any project file containing
a line starting with that label won the scan.
The comparison then ran runtime state against project prose, a mismatch that
never clears: the stored prompt was rejected on EVERY turn, rebuilding the
system prompt each message and destroying the prefix cache for the whole
session. Strictly worse than the staleness the check exists to catch, and
invisible to CI because no test encoded the invariant.
"last match wins" was only ever safe because Model/Provider/Platform live in
the volatile tier at the very END of the prompt. The cwd line is the one field
read from the STABLE tier near the start, so it needs an anchored read:
locate Hermes' own "User home directory:" line and take the working-directory
line that follows it.
Verified against the real builder on a real AIAgent, not a stub:
- stable cwd, clean project -> prompt REUSED (0 rebuilds)
- stable cwd, AGENTS.md naming the
field -> prompt REUSED (was: rebuild/turn)
- drifted cwd -> rebuilt
- drifted cwd, AGENTS.md naming the
NEW cwd -> rebuilt (prose can't mask drift)
Also re-verified no spurious rebuild across all five cwd-resolution paths
(pinned contextvar, TERMINAL_CWD, launch dir, symlinked workspace,
trailing-slash cwd).
The contributor's fixtures omitted the host-info anchor, so they silently
stopped exercising the cwd path once the read was anchored; reshaped them to
match real prompt structure via a _host_block() helper.
The residual Windows-on-ARM integrity-gate fix added GetNativeSystemInfo as a
second "OS-native" probe behind IsWow64Process2. Microsoft documents the
opposite behavior: "the API GetNativeSystemInfo also returns emulated
processor details when run from an app under emulation."
On the exact hosts the probe was added to rescue -- an x64 hermes-setup.exe
emulated on an ARM64 Surface -- it therefore reports AMD64, making it a
duplicate of the PROCESSOR_ARCHITECTURE rung already below it rather than a
fallback for it. Its test only passed because the kernel32 fake was hand-fed
ARM64, asserting behavior real Windows does not exhibit.
Replace it with GetMachineTypeAttributes, which answers the question the gate
actually asks -- "can this host load a PE of machine X?" -- instead of
inferring it from an architecture name. It is also the only documented API
that reports AMD64-on-ARM64 emulation support. The gate prefers it and falls
back to the existing name-based mapping on pre-Windows-11 hosts.
The load-bearing half of the previous fix (typing GetCurrentProcess as HANDLE
so IsWow64Process2 stops failing ERROR_INVALID_HANDLE) is unchanged.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: Teknium <teknium1@users.noreply.github.com>
`hermes curator status` reported only the skills it manages, staying silent
about curation-eligible skills it can never touch. On a 237-skill library that
meant 112 skills were invisible to every automatic transition with no signal
anywhere — the curator looked broken when it was working as designed.
A skill becomes curator-managed only when `created_by: agent` lands on its
usage record, and only the background review fork writes that marker. Two
populations therefore never qualify: records written before the marker existed
(no key at all, authorship unknowable) and every foreground
`skill_manage(create)` (unset by design — those skills belong to the user).
- skill_usage: `list_unmanaged_skill_names()` / `unmanaged_report()` enumerate
the blind spot, tagging each row with `has_provenance_key` so the two causes
are distinguishable. `adopt_skill()` writes the marker on user declaration
and refuses bundled, hub-installed, external, and protected built-ins.
Adoption never resets the inactivity clock.
- curator CLI: status prints an `unmanaged (no provenance marker)` block on
BOTH the managed and no-managed-skills paths; new `adopt` verb takes names or
`--all-unmanaged`, with `--dry-run` and a confirmation prompt on bulk.
- skills_sync: `_backfill_optional_provenance()` matched candidates by
repo-derived path only, so a skill installed at `mlops/chroma` that upstream
later moved to `mlops/vector-databases/chroma` was skipped forever and
`hermes skills repair-optional` could not fix it. Falls back to an
unambiguous name match, still gated on identical content, and records the
ACTUAL install path.
Provenance stays a declaration, never an inference: a high patch count proves
the agent MAINTAINS a skill, not that it authored one, since Hermes edits
user-written skills on the user's behalf routinely. An "looks agent-made"
heuristic would eventually archive hand-written work.
Validation: 347 targeted tests pass. Each new test verified via sabotage run
(revert the fix, confirm the test goes red) — one initially passed for the
wrong reason because the fixture pinned `prune_builtins` off, masking the guard
under test; fixed to force the shipped default on.
Slash commands run on the event-loop thread, outside the per-turn
contextvars.copy_context() that pins the session cwd for the agent call.
/compress reaches agent._build_system_prompt(), whose "Current working
directory" line comes from resolve_agent_cwd() — so an unpinned handler
rebuilt the prompt against the Hermes install tree and PERSISTED it as
the session's cached prompt, re-poisoning every later turn even though
the turn itself is now pinned.
Pin inside a fresh context copy so the write cannot leak into other
concurrent ACP sessions on the shared loop and needs no teardown.
An ACP session registered the client's cwd for the *tools*
(`_register_task_cwd` -> `register_task_env_overrides`) but never pinned it
for the *prompt*. `agent/prompt_builder.py` reports
`Current working directory: {resolve_agent_cwd()}`, and `resolve_agent_cwd()`
reads the `_SESSION_CWD` contextvar, which ACP left unset — so it fell back
to `TERMINAL_CWD` / the launch dir.
The system prompt therefore advertised one root (commonly
`~/.hermes/workspace`, or the install tree) while the tools were rooted at the
editor's project. When the model emitted a *relative* path the tools resolved
it correctly; when it emitted an *absolute* path built from the advertised
root, the write landed outside the client's workspace and the turn still
reported success.
Observed in Zed/Buzz-shaped usage: the first prompt in a fresh workspace
creates the file under `~/.hermes/workspace/` and answers "Done." The client's
directory is untouched. Later sessions on the same cwd appear to work once a
session cwd record exists, which makes it look intermittent.
`_run_agent` already calls `set_session_vars(session_key=session_id)` inside
its `contextvars.copy_context()`, and that helper's `cwd` argument exists to
"pin the logical working directory for this context". It simply was not
passed. `gateway/session_context.py` and `tui_gateway/server.py` both already
pass it; ACP was the only surface that did not.
Adds a regression test asserting that the resolved cwd *during the turn* is
the cwd the client passed to `session/new`. It fails without this change
(resolving the install tree instead of the client's project).
- RelayAdapter.create_handoff_thread → one op-gated thread_create op
(Discord channel thread / Telegram forum topic / Slack named seed root);
None fallback contract preserved for the handoff watcher.
- RelayAdapter.rename_thread → thread_rename with only_if_current_name
no-clobber guard on the wire (connector enforces; Telegram guarded
renames fail safe). The native semantic-rename lane
(_is_discord_auto_thread_lane) lights over the relay via the
connector-stamped auto_thread_created/auto_thread_initial_name markers
parsed onto SessionSource in _event_from_wire.
- reply_to {text, author, is_own} wire parse onto the SAME MessageEvent
reply-context fields native adapters populate.
- gateway/relay/command_manifest.py: the gateway-declared slash-command
manifest (native Discord tree mirror) sent on the DISCORD hello; the
connector reconciles Discord's global registration (additive field,
older connectors ignore).
- Contract doc §OutboundAction ops + Phase 4 semantics sections.
- 12 new tests (tests/gateway/relay/test_relay_threads.py); relay suite
266 passed.
The retry loop skipped only the current version, so on a uv whose download
catalog is stale it walked backwards through older patches. In #71250 the
newest indexed 3.11 is 3.11.14 — the installed version — so all five retries
went to 3.11.13..3.11.9, each a real download+install+probe+delete cycle that
the existing downgrade guard was always going to reject, before failing anyway.
Only newer patches can carry the SQLite fix. Skipping <= current makes the
stale-catalog case cost one attempt instead of six, and still finds 3.11.15
on the first retry when the catalog knows about it.
Fixes#71250.
`hermes update` could not repair the embedded Python runtime's
vulnerable SQLite (WAL-reset bug range 3.7.0-3.51.2, except backports
3.50.7/3.44.6) on installs where `uv python install <minor>` (bare
request, e.g. "3.11") resolves to an older cached/indexed patch that
still links a vulnerable SQLite build, even though a newer
non-vulnerable patch on the same minor line is available and known to
uv. The smoke test correctly rejected the vulnerable candidate every
time, but the provisioner gave up immediately after one attempt,
leaving the repair permanently stuck in the same failing loop on every
`hermes update` run.
Implements option D from the issue (query the index, retry with an
explicit newer patch), which the reporter identified as cleanest:
- New `_list_available_patches()`: runs `uv python list <minor>
--all-versions --only-downloads --output-format json --no-config`,
filters to cpython/default-variant entries (excluding pypy/graalpy),
and returns known patch versions newest-first. Fails safe (returns
[]) on any network/parse error.
- Refactored the single install+find+probe cycle out of
`_install_safe_python_generation()` into `_attempt_install_generation()`,
reusable per attempt with its own generation directory (so a
rejected candidate's files are fully cleaned up before the next
attempt, matching the existing --reinstall semantics).
- `_install_safe_python_generation()` still tries the bare minor-line
request first (preserves the original comment's rationale: for a
given exact patch, python-build-standalone may have no artifact with
fixed SQLite at all). If that resolves vulnerable, it now queries
`_list_available_patches()` and retries with explicit newer patches,
newest-first, bounded to `_MAX_PATCH_RETRIES` (5) attempts -- each
attempt is a real download+install+probe cycle, so the cap keeps
worst-case repair time bounded.
Sanity-checked `_list_available_patches()` against the real `uv`
binary (0.11.7) in this environment: correctly parses live `uv python
list --all-versions --output-format json 3.11` output, returns 14
patches sorted newest-first starting at 3.11.15.
9/9 new tests pass (retry succeeds with a newer patch, exhausts
gracefully when every known patch is vulnerable, empty patch list
degrades to None without crashing, retry count is bounded, plus direct
JSON-parsing unit tests for realistic/malformed/empty uv output);
47/47 in the full tests/hermes_cli/test_managed_uv.py file (including
the 3 pre-existing tests for the original bare-minor success path,
confirming no regression there).
A failed `hermes sessions repair` printed "keep state.db and the backup"
and stopped, so a user whose sessions had vanished had no way to discover
that a non-destructive recovery path exists — the reported dead end.
The failure branch now names the next command, read-only step first, seeded
with the backup path it just preserved. Covered by a CLI-surface regression
test that drives the real subprocess against an unrepairable database.
The Codex image backend rejected our own request shape for every account, and
we then translated that rejection into "Image generation is not enabled for the
current Codex account. Switch the image provider to OpenAI API key, FAL, or
xAI." — telling every affected user to abandon a provider that had never
actually been tried. That message is why this reads as a setup failure rather
than a bug: the wire error was replaced with a confident, wrong diagnosis.
Removes the classifier and its exception, so any HTTP failure surfaces
verbatim. The paired request-shape fix (previous commit) is what makes the
400 stop happening; this commit makes the next one diagnosable.
Also fixes error-body truncation: bodies were head-truncated at 500 chars, and
Codex error payloads can carry hundreds of bytes of leading metadata, so the
user got a wall of padding and no message. _summarize_error_body() prefers the
parsed error.message and falls back to a truncated raw body.
Docs: drop the unqualified image-to-image claim for the Codex backend and note
that the hosted tool call cannot be forced, so it is best-effort.
Verified E2E against a local fake Codex backend: success path writes a real PNG
with no tool_choice on the wire; the 400 path now returns api_error carrying
"Tool choice 'image_generation' not found in 'tools' parameter" (148 chars)
instead of the entitlement message. Sabotage run confirms all 4 regression
tests fail when the old behavior is restored.
Refs #19505, #49008, #31335.
The chatgpt.com/backend-api/codex backend 400s on every tool_choice
shape for the hosted image_generation tool — it looks up tool_choice as
a function name and never recognizes hosted-tool entries. Removing the
field from _build_responses_payload() lets the host model decide; the
instructions field nudges it toward the tool.
Salvaged from PR #19979 (originally targeted the old
client.responses.stream call, which no longer exists on upstream/main;
the live request now flows through _build_responses_payload + httpx in
_collect_image_b64).
`_prune_stale_worktrees` runs synchronously before the banner on every
`hermes -w` launch and shells out to git several times per candidate
worktree. On a repo with dozens of accumulated worktrees this dominated
startup: measured 18.5s of a 20.6s cold start, 11.6s on warm repeats.
The `git cherry` patch-equivalence probe was the single largest cost
(9.4s of 11.5s across 24 trees). It is also pure waste on repeat runs: a
tree preserved because it holds unpushed work is re-diff-hashed on every
launch, forever, always reaching the same verdict. On this repo 19 of 24
aged trees were unreapable, so ~11s per launch bought zero reaps.
Two changes, both verdict-preserving:
- Split the loop into a stat-only age filter, a parallel read-only
classification phase (thread pool, bounded to min(8, cpu_count)), and
a serial mutation phase. Only reads are concurrent; unlock/remove/
branch -D stay ordered, so log output and removal order are unchanged.
A pool failure falls back to serial rather than blocking startup.
- Memoize `git cherry` verdicts to
$HERMES_HOME/cache/worktree_merge_verdicts.json, keyed on the exact
`(base_sha, head_sha, max_ahead)` range the verdict was computed from.
Because that key is the complete input to the git call, a cache hit is
identical to recomputation by construction: if either ref moves, the
key changes and real git runs again. Bounded to 1000 entries, written
atomically, and a corrupt/hand-edited cache degrades to recomputation.
Measured on a 44-worktree repo (24 aged candidates):
_prune_stale_worktrees before 11.5s after 2.05s cold / 0.42s warm
hermes -w to banner before 13.9s after 1.79s
Work-preservation is unchanged: all 44 worktrees survived, and every
dirty/unpushed/live-locked guard still fires. Verified the 24 real-tree
verdicts are byte-identical across serial, cold-cache, and warm-cache
runs.
Tests: 75/75 in tests/cli/test_worktree.py (67 existing + 8 new). The
new cache tests were sabotage-verified — swapping the exact-sha key for a
naive path-only key makes two of them fail by deleting a worktree that
had gained unmerged work, which is the data-loss case the key prevents.
* fix(update): bind IsWow64Process2 HANDLE and fall back to GetNativeSystemInfo
The #71218 OS-native probe still rejected correct ARM64 Desktop rebuilds on
Windows-on-ARM when ctypes truncated GetCurrentProcess()'s pseudo-handle and
IsWow64Process2 failed with ERROR_INVALID_HANDLE, falling through to the
lying PROCESSOR_ARCHITECTURE=AMD64 value. Type the HANDLE correctly and use
GetNativeSystemInfo before the env-var fallback.
* test(update): cover WoA IsWow64Process2 handle failure and system-info fallback
Pin the residual #71218 shape where IsWow64Process2 returns FALSE, the env
arch lies as AMD64, and GetNativeSystemInfo must still report ARM64 so the
integrity gate accepts a correctly-built ARM64 Hermes.exe.
`sync_skills()` keys the bundled manifest by frontmatter name but computes
the destination from the bundled path. When upstream renames or
recategorizes a skill, the manifest key still matches while the new dest
does not exist yet, so the loop fell into its "in manifest but not on
disk" branch and misread the skill as user-deleted: the user's copy was
stranded at the old path forever and never received another update.
Three skills hit this in the July 2026 reorg (computer-use,
evaluating-llms-harness, serving-llms-vllm) — silently frozen at their
pre-rename content on every machine that ran `hermes update`.
Recovery only moves a stale copy when it is byte-identical to the origin
hash recorded the last time sync wrote it, which proves the directory is
ours rather than the user's work. User-modified copies are kept in place
with a warning, hub-installed paths are never touched, and a genuine
deletion (no copy anywhere on disk) is still respected.
- tools/skills_sync.py: add _recover_renamed_skill() plus the
_index_active_skills() / _read_hub_install_paths() indexes; call it
before classification and report moves via a new `relocated` key.
- hermes_cli/main.py: surface relocations in both `hermes update` skill
sync reporting sites.
- tests: 4 cases covering relocate, user-modified preservation,
hub-installed exemption, and genuine-deletion respect.
Some OpenAI-compatible endpoints — notably Tencent Copilot
(copilot.tencent.com) — only accept streaming chat requests; any
non-streaming call returns HTTP 400 (code 11101, 'Non-stream chat
request is currently not supported'). The main conversation loop already
streams, so interactive chat works, but every auxiliary task (title
generation, compression, web extraction) used the non-streaming path and
failed on each call.
_provider_requires_stream() detects stream-only endpoints
(copilot.tencent.com built in, plus user-configurable
auxiliary.stream_only_base_urls substring markers in config.yaml).
Matching sync auxiliary calls route through _create_with_progress
(force_stream=True) and async calls through the new
_acreate_with_stream, aggregating the chunk stream — including tool-call
deltas and reasoning deltas — into a complete response via the shared
_ChatStreamAccumulator.
Salvaged from PR #60686 by @kudi88 onto the progress-aware streaming
machinery from #71508, addressing both sweeper-review gaps: the async
path now consumes the stream with 'async for' (awaiting create() and
iterating synchronously raised on AsyncOpenAI streams), and tool-call
deltas are reassembled instead of dropped (MCP passes tools= through
call_llm). Under force_stream there is no silent non-streaming retry —
a stream-only provider rejects those by definition, so the original
error surfaces to the normal recovery chains.
- RelayAdapter.send_exec_approval / send_slash_confirm / send_clarify:
override the base text fallbacks with ONE platform-abstract `prompt` op
(connector renders Discord components / Telegram inline keyboards /
Slack Block Kit / WhatsApp buttons+lists). Option sets mirror the native
adapters exactly (once/session/always/deny with the same
allow_session/allow_permanent/smart_denied gating; once/always/cancel;
choices + Other). Clarify option ids are positional (c0..cN/other) —
choice text is arbitrary UTF-8, callback budgets are 64 bytes.
- Pending-prompt registry: gateway-minted 8-hex prompt ids →
{kind, session_key, extras}; one answer wins, lazy expiry, unanswered
prompts swept opportunistically. Wire timeout_s stays advisory.
- _consume_prompt_response (wired into _on_inbound AND the Discord
passthrough lane): routes answers to the SAME primitives the native
button handlers call — tools.approval.resolve_gateway_approval,
tools.slash_confirm.resolve, tools.clarify_gateway
resolve/mark_awaiting_text — then acks in-channel. Unknown/expired ids
fall through as command-shaped text (typed-reply degradation, the
relay's analog of the native 'approval expired' edit).
- Discord type-3 stub replaced: an hp1:<prompt>:<option> custom_id decodes
to a structured prompt_response (codec mirrored from the connector's
promptCodec); foreign custom_ids keep the legacy best-effort text shape.
- MessageEvent.prompt_response field + ws_transport wire parsing (additive).
- react ack lifecycle: on_processing_start/complete → `react` ops
(👀 → ✅/❌, remove-then-add), op-gated on supported_ops, best-effort by
contract (a react failure never touches the turn).
- Op gating throughout: a connector not advertising `prompt` gets
success=False from send_exec_approval/send_slash_confirm (run.py's text
fallback takes over — same contract as a failed native button send) and
the base numbered-text clarify; `react` silently no-ops.
- docs/relay-connector-contract.md §4: prompt / prompt_response / react
semantics (callback token, budgets, authorization-parity, foreign-id
behavior, per-platform react mappings).
- tests: tests/gateway/relay/test_relay_interactive.py (19) — option-set
rendering + gating matrices, registry consume-once/expiry, resolver
routing for all three kinds (monkeypatched primitives), fall-through
cases, Discord hp1 decode + foreign-id shape, react lifecycle
(success/failure/cancelled), op-gated/best-effort react.
Cross-repo pair: gateway-gateway 'Phase 3 interactive' PR (prompt/react
senders on all four lanes + interaction ingest).
The gateway's pre-agent session-hygiene compression killed the summary
call at a fixed 30s wall-clock deadline (compression.hygiene_timeout_seconds),
regardless of whether the summary model was hung or merely slow. A reasoning
model happily streaming a large summary was cut off mid-generation, the user
got '⚠️ Context compression timed out after 30.0s', and a 300s failure
cooldown left the session oversized — a doom loop for slow-but-healthy
auxiliary models.
Timeouts are now liveness-based instead of wall-clock-based:
- agent/auxiliary_client.py: new thread-local aux_progress_hook. When
installed (only by context compression today), the primary call_llm
attempt streams (stream=True) and aggregates chunks back into a complete
response, ticking the hook per chunk. The configured timeout then acts
per stream read (idle) instead of as a total budget. Providers that
reject streaming fall back to the plain non-streaming call; auth/payment/
rate-limit/transport errors propagate unchanged into the existing
recovery chains. Codex Responses (per SSE event) and Anthropic Messages
(per stream event, via the new create_anthropic_message on_stream_event
callback) tick the same hook from inside their wire adapters.
- agent/conversation_compression.py: CompressionCommitFence gains
touch_progress()/seconds_since_progress(); compress_context() installs
fence.touch_progress as the progress hook around the compress call.
- gateway/run.py: the hygiene wait loop treats hygiene_timeout_seconds as
an INACTIVITY budget — while the fence reports fresh progress the wait
extends, bounded by the new compression.hygiene_total_ceiling_seconds
(default 600s, clamped >= the idle budget) so a degenerate trickle
stream still dies. The timeout warning now says the summary model
produced no output, which is the only case that still triggers it.
- config/docs: hygiene_total_ceiling_seconds added to DEFAULT_CONFIG and
configuration.md; hygiene_timeout_seconds documented as inactivity-based.
Tests: tests/agent/test_aux_progress_streaming.py (hook plumbing, stream
aggregation incl. tool-call deltas and reasoning deltas, rejection
fallback, ceiling kill, fence progress surface); two new gateway tests
prove a slow-but-streaming worker survives past the fixed timeout
(sabotage-verified: fails with the old fixed deadline) and a
forever-trickling worker is still cut off at the ceiling.
Two tests fail deterministically on main depending only on which SQLite the
test interpreter links — nothing about the code under test. Both are green in
isolation and red in the suite / on an older library, the worst diagnostic
shape.
Root cause A — WAL is not always WAL. Hermes refuses journal_mode=WAL on
SQLite builds carrying the upstream WAL-reset corruption bug (3.7.0–3.51.2,
excluding backports 3.50.7 / 3.44.6) and falls back to DELETE. On such a
build NO -wal sidecar is ever created, so
test_wal_checkpoint_truncates_wal_file asserts on a file that cannot exist.
Invisible locally when the repo .venv and the Hermes managed runtime link
different versions (observed: .venv 3.50.4 → DELETE, runtime 3.53.1 → WAL),
so the same test passes for one interpreter and fails for the other.
- tests/conftest.py: add a `requires_wal` marker plus a
pytest_collection_modifyitems hook that skips such tests when the linked
library will fall back to DELETE. The skip reason names the actual
version so it is diagnosable rather than mysterious.
- pyproject.toml: register the marker.
- test_kanban_db_repair.py: mark the -wal-sidecar test.
Root cause B — process-global warn-once dedup. The WAL-fallback warning is
emitted at most once per (process, db_label). Any earlier test in
test_kanban_db.py that opens a kanban.db consumes that one-shot, so
test_connect_falls_back_to_delete_on_locking_protocol sees zero warnings and
fails — but only as part of the file, never alone.
- test_kanban_db.py: clear both dedup sets in the test that asserts on the
warning, with a comment explaining the isolation trap.
The gate deliberately does NOT import hermes_state. That module computes
DEFAULT_DB_PATH from get_hermes_home() at import time, so importing it during
collection — before the per-test _isolate_hermes_home fixture redirects
HERMES_HOME — permanently caches the developer's REAL ~/.hermes/state.db for
the whole session. The first version of this change did exactly that and made
tests read a live 31,881-session production database (test_console_engine
asserted "Total sessions: 2" and got 31881). The version predicate is
duplicated instead, and tests/test_conftest_wal_gate.py pins the two
implementations in agreement across every documented upstream boundary plus
guards against the import coming back.
Verified: on SQLite 3.50.4 the sidecar test SKIPS naming the version; on
3.53.1 it RUNS and passes, so coverage is not lost where WAL works. Clean
main fails exactly these 2 tests under
`scripts/run_tests.sh tests/hermes_cli/ tests/test_hermes_state.py`
(9926 passed, 2 failed); with this change the same scope is green.
Tests: 726 passed across the two kanban files, test_hermes_state.py, and the
new gate tests.
Three foot-guns in the canonical test runner, each of which cost real
debugging time by making an unverified run look verified.
1. Zero collection across the whole run reported success-shaped output.
Per-file rc=5 is rewritten to rc=0 so a platform-gated file (every test
skipped on this OS) doesn't fail the suite — correct, but it also meant a
run where NOTHING was collected anywhere printed
"0 tests passed, 0 failed (100% complete)" and, with no failures
recorded, could exit 0. Now the run-level guard counts every collected
outcome (passed/failed/skipped/errors/xfailed/xpassed): an all-skipped
file still passes, but zero-collected-anywhere prints an explicit
"✗ NO TESTS RAN — this is NOT a pass" block naming the likely causes and
returns 1.
2. A venv without pytest was selected merely for existing. The probe
accepted any directory with bin/activate, so in a checkout/worktree
without a local .venv it picked the RELEASE venv
(~/.hermes/hermes-agent/venv, no pytest). Every file then died with
"No module named pytest" and the run reported 0 tests. Candidates are now
import-checked for pytest — the same guard the HERMES_PYTHON fallback
already applied — and a skipped candidate is named on stderr.
3. Pytest node ids were silently discarded. This runner is file-granular,
so `tests/foo.py::TestBar::test_baz` isn't an existing path: discovery
dropped it and the run ended "No test files to run" while the selector
looked accepted. Node ids are now translated to the FILE plus an inferred
`-k` on the leaf name (parametrized ids reduced to the function name),
with a note explaining the translation. An explicit caller `-k` wins over
the inferred one.
Tests: 4 behavior contracts in tests/test_run_tests_parallel.py. Verified
by sabotage — reverting the runner fails 3 of the 4 (the fourth pins the
pre-existing all-skipped tolerance so fix 1 can't regress it).
- gateway/relay/media.py: RelayMediaClient for the connector's /relay/media
plane (upload local files → re-host reference for send_media; download
re-hosted inbound attachments → local temp paths). Same connector base URL
the WS dials, same per-gateway signed bearer as the upgrade (auth.py) —
no new configuration. stdlib urllib in a thread executor (no new deps);
25MB cap mirroring the connector's MEDIA_MAX_BYTES.
- RelayAdapter: send_image / send_image_file / send_voice / send_video /
send_document overrides route through ONE send_media op (media by
reference: local paths upload first, public URLs pass through). Gated on
supported_ops advertising send_media — legacy connectors keep today's
text fallbacks; connector declines/failed uploads degrade the same way.
Scope/user egress discriminators ride metadata exactly like send.
- Inbound: _localize_inbound_media downloads each media_urls entry to a
local temp path (native-adapter parity — vision/file tools consume
paths); dead re-host refs are dropped, public URLs survive a missing
client. Best-effort, never blocks handle_message.
- docs/relay-connector-contract.md §4: send_media op row + media
ingress/egress semantics (replaces the 'deferred to a later revision'
note). Additive within contract_version 1.
- tests: tests/gateway/relay/test_relay_media.py (15) — kind mapping,
upload-first path handling, op gating (explicit + legacy-empty),
decline/upload-failure fallbacks, scope metadata, inbound localization
matrix, client URL derivation/credential gating. Stub connector grew a
canned send_media result.
Cross-repo pair: gateway-gateway 'Phase 2 media parity' PR (re-host plane +
four ingress lanes + four platform send_media senders).
Follow-up for salvaged PR #70615 — the 2-button smart_deny case
(Allow Once + Deny only) was exercised by an existing test but only
at the flat-label level, not asserting the row pairing. Adds the
missing row-structure assertion using the same capture pattern as
the 4-button and 3-button tests.
Follow-up for salvaged PR #69380. The snippet
'export -p | grep -vE ... > $tmp.$BASHPID || true' attaches the redirect to
the grep pipeline segment, so $BASHPID expands inside grep's pipeline
subshell — a DIFFERENT pid than the parent shell that expands the follow-up
'mv $tmp' operand. The dump landed in an orphaned temp file, mv failed
silently (2>/dev/null), and the shared snapshot never updated again:
exported env / venv activation stopped persisting between commands
(tests/tools/test_local_shell_init.py TestSnapshotEndToEnd caught it).
Wrap the pipeline in a brace group and attach the redirect to the group so
the expansion happens in the current shell, matching mv's expansion. Also
rename the shell-init probe var HERMES_SESSION_ENV_PROBE ->
HERMES_STICKY_ENV_PROBE: it matched the HERMES_SESSION_ prefix the salvaged
fix now intentionally strips from snapshots, and the snippet-shape test is
updated to pin the brace-group contract.
A single long-lived backend serves many sessions through one
_active_environments['default'] LocalEnvironment (the messaging gateway, TUI,
and desktop/web dashboard all collapse the terminal to 'default'). That
environment persists a bash session snapshot file and sources it before every
command. 'export -p' dumped the FIRST session's HERMES_SESSION_ID into the
snapshot, so every LATER session sourced that stale value and its
'echo $HERMES_SESSION_ID' reported a FOREIGN session's id — overriding the
correct per-command Popen env injected by _inject_session_context_env.
Confirmed on staging-fp: session A read its own id, session B (same backend)
read A's id. Reproduced locally with two threads sharing one LocalEnvironment;
the snapshot held 'declare -x HERMES_SESSION_ID=...' from the first session.
Fix: strip the per-session bridged vars (HERMES_SESSION_* / HERMES_UI_SESSION_ID
/ HERMES_CRON_AUTO_DELIVER_*, i.e. gateway.session_context._VAR_MAP prefixes)
from the snapshot at both dump sites in base.py. They are re-injected fresh on
every command, so a snapshot should carry only the user's own shell state, not
Hermes' per-turn session identity.
Complements the _set_session_context session_id fix: that ensures the ContextVar
carries the right id; this ensures the shared snapshot can't override it with a
neighbour's. Adds tests/tools/test_snapshot_session_id_leak.py (regex unit +
real two-session LocalEnvironment integration) and updates the export-shape
assertions in test_base_environment.py for the new 'export -p | grep' dump.
_set_session_context() called set_session_vars() without session_id, so the
HERMES_SESSION_ID contextvar was set to "" (explicitly empty) on every
prompt.submit / session bind. The subprocess-env bridge
(_inject_session_context_env) treats an explicit "" as authoritative and does
NOT fall back to os.environ, so terminal/execute_code commands in a
dashboard/TUI/web session saw an empty $HERMES_SESSION_ID — even though
agent_init had populated it via set_current_session_id().
Every other set_session_vars() call site (gateway/run.py, cron, api_server)
passes session_id; tui_gateway was the only one that dropped it, affecting all
three of its frontends (Ink TUI, desktop, web).
Fix: derive the live id in the existing _sessions lookup loop
(agent.session_id, falling back to session_key — same derivation used at
session finalize) and pass it through to set_session_vars.
Adds tests/tui_gateway/test_session_id_injection.py covering agent id,
session_key fallback, and unknown/ephemeral key.
Reviewer egilewski identified that the 5s lock timeout fell open:
quarantine_zeroed_state_db() logged 'proceeding without the cross-
process lock' and continued to re-check + rename state.db. A slow or
paused startup that still owns the lock can overlap this fallback and
the two processes can again act on the same live file.
Fix: fail closed — return None without moving the file when the lock
cannot be acquired within 5s. Log an error with recovery guidance
(restore from state-snapshots).
Test: test_quarantine_fails_closed_when_lock_held holds the cross-
process lock from a background thread, calls quarantine, and asserts
it returns None without moving the zeroed file.
Reviewer egilewski identified two recovery-loss paths:
Path A — quarantine race (hermes_state.py):
SessionDB checks state.db before quarantine_zeroed_state_db() without a
shared cross-process lock, and Path.rename() may replace an existing
destination. Two writable startups can therefore move the first
instance's newly created database over the .zeroed-*.bak, erase the
original damaged-file evidence, and replace the live database with
another empty one.
Fix: add a cross-process lock (msvcrt on Windows, fcntl on POSIX)
around quarantine_zeroed_state_db() with a 5s bounded timeout. Under
the lock: re-check is_zeroed_state_db (another process may have already
quarantined it and created a fresh DB), use a PID-suffixed unique
destination, and non-clobbering rename with counter fallback.
Path B — size-cap pruning gap (hermes_cli/backup.py):
_too_large() runs before failed-database tracking. With keep=1 and a
size cap, an oversized state.db is omitted while failed_dbs stays empty,
so automatic pruning deletes the older complete snapshot that may
contain the only recoverable database.
Fix: track oversized DB files in a new oversized_skipped list (both in
the directory walk and top-level file loop). The manifest now records
oversized_skipped. Pruning is suppressed when failed_dbs or
oversized_skipped is non-empty, preserving the older complete snapshot
as recovery source.
Tests:
- test_concurrent_quarantine_no_clobber: two threads racing on the
same zeroed state.db — verifies quarantine backup survives with
original bytes and live DB is valid.
- test_oversized_db_suppresses_pruning: keep=1 + oversized state.db
verifies the older complete snapshot is not pruned.
All 33 tests pass (3 zeroed_state_db + 25 TestQuickSnapshot + 5
quarantine_forensic_logging).
Hardening for the Windows zeroed-state.db class (#68474):
- Surface critical stdout when pre-update/quick snapshot cannot copy a
present *.db (was log-only; update still looked successful).
- Detect all-NUL SQLite header on SessionDB open, quarantine the bytes,
and open a fresh DB with recovery guidance to state-snapshots.
Does not claim storage-stack root cause.
The #71119 integrity gate rejected CORRECT ARM64 desktop rebuilds on
Windows-on-ARM (#69179 follow-up): platform.machine() reports the PROCESS
architecture, and the update chain runs an x64 hermes-setup.exe / x64
Python under ARM64 emulation, so the gate compared the ARM64 Hermes.exe
against a phantom 'AMD64 host' and aborted the update.
_windows_native_machine() now asks the OS via IsWow64Process2 (whose
nativeMachine is truthful from emulated processes), falling back to
PROCESSOR_ARCHITEW6432/PROCESSOR_ARCHITECTURE (pre-1511 Win10) and then
platform.machine(). All three consumers — the integrity gate, the
packaged-exe arch preference, and backup validation — go through it.
Sabotage-verified: the three new regression tests fail against the old
process-arch behavior and pass with the fix.
