Connecting Discord asked five questions when the platform needs one. The card
listed a home channel ID you need Developer Mode to copy, an allow-all-users
security toggle, a reply-threading preference, and a home channel display name
— all with working defaults, none discoverable from the form.
Drops them from the setup surfaces entirely: the dashboard/Desktop channel
cards and the `hermes setup gateway` wizard. Discord is now bot token +
allowlist. Matrix drops from 11 fields to 7, Mattermost from 6 to 3.
Suffix-matched (`*_HOME_CHANNEL*`, `*_ALLOW_ALL_USERS`, `*_REPLY_TO_MODE`,
`*_REQUIRE_MENTION`, `*_AUTO_THREAD`, `*_FREE_RESPONSE_*`, `*_PROXY`) so plugin
platforms nobody enumerated get the same treatment. Allowlists deliberately
stay — the gateway denies everyone until one is set, so that IS the decision a
new user has to make. Required credentials are never hidden.
Nothing is removed from the product. The vars still work through
`hermes config set`, .env, and config.yaml, the gateway reads them unchanged,
and dropping them from the cards hands them back to the Keys page rather than
orphaning them (Keys hides only what a Channels card owns).
The include/exclude filter matched exact names only — glob-style entries
('*_radar_*') silently matched nothing, so a Cloudflare flat-mode config
meant to trim 3,320 tools to ~1,900 actually registered 3,319. Unmatched
patterns produced no warning.
- matches_name_filter(): exact membership first (O(1) for literal lists),
then fnmatch.fnmatchcase for entries containing * ? [ — same pattern
semantics as approvals.deny. Entries without metacharacters stay
strictly literal ('docs' never matches 'docs_search').
- _should_register() uses it for both include and exclude (symmetric)
- hermes mcp tools picker (mcp_config.py) pre-selection uses the same
matcher so the UI agrees with runtime registration
E2E against the live Cloudflare capture with the real exclude list:
3,320 -> 1,905 surviving (1,415 excluded); radar/DLP gone,
purge_cache/dns_records kept. 220/220 mcp_tool tests (4 new).
Teknium review changes on the tiered policy:
1. threshold_pct default 10 -> 5 (listing budget = min(5% of context,
listing_max_tokens)); unknown-context fallback 20K -> 10K.
2. Tier 2 no longer leaves the model blind: when even names-only doesn't
fit, the bridge description now carries a one-line-per-server summary
('cloudflare (3320 tools)') plus an instruction to search FIRST rather
than substitute a generic tool or claim the capability is missing —
the measured tier-2 failure mode (core-tool substitution) at zero
meaningful token cost (~50 tokens/server).
3. Listing degradation is now PER SERVER, largest first: one oversized
server (Cloudflare) collapses to its summary line while small
co-attached servers (Linear) keep their full per-tool listings
('mixed' form). Previously global: attaching Cloudflare next to
Linear silently cost Linear its listing. Greedy fit is deterministic
(size then label) so the rendered block stays byte-stable per catalog
— prompt-prefix cache safe.
E2E on real captures (defaults, 200K ctx): linear alone -> tier 1 full;
unreal alone -> tier 2 groups (5% budget) / tier 1 names at 1M;
cloudflare alone -> tier 2 groups; linear+cloudflare -> tier 1 MIXED
(linear fully listed, cloudflare summarized). 48/48 tests.
Tier 0: no MCP/plugin tools -> everything eager (pass-through).
Tier 1: deferred tools whose catalog listing fits min(threshold_pct%
of context, listing_max_tokens) -> bridge + skills-style
listing, degrading to names-only over budget.
Tier 2: listing over budget even names-only (Cloudflare's flat API
surface: 3,320 tools, names alone ~32K tokens) -> bare bridge,
discovery through tool_search only.
The old activation threshold (defer only when schemas > threshold_pct
of context) let mid-size catalogs ride eager and pay full schema cost;
with servers like Cloudflare (~597K tokens of schema, would not even
fit a 200K window) the binary gate is the wrong shape. Activation is
now driven purely by deferrable-tool presence; threshold_pct is
repurposed as the listing budget's context-relative leg.
- AssemblyResult gains tier + listing_form for observability
- listing_max_tokens default 4000 -> 20000 (cap 60000) so an 830-tool
catalog keeps a names-only listing while Cloudflare-scale drops to
bare bridge
- E2E verified against real captures: Linear 24 tools -> tier 1 full,
Epic UE 5.8 830 tools -> tier 1 names-only, Cloudflare 3,320 tools
-> tier 2 (both 200K and 1M context)
Deferred MCP/plugin tools become invisible once the tool_search bridge
activates — live benchmarking (48 runs, Claude Haiku 4.5) showed models
substituting visible core tools (terminal/web_search/browser) for deferred
capabilities or declaring them nonexistent instead of searching: 16/24
task success vs 24/24 with eager loading.
Skills never had this failure mode because every skill keeps a ~21-token
name+description listing line in the system prompt. This ports that exact
pattern to the tool bridge: when tool_search activates, a grouped manifest
of every deferred tool (name + first sentence of description, clipped to
60 chars, grouped per MCP server / toolset) is embedded in the tool_search
bridge description.
- tools/tool_search.py: build_catalog_listing() with deterministic
ordering (byte-stable across assemblies -> prompt prefix stays
cacheable); token-budget fallbacks full -> names-only -> legacy bare
count; bridge_tool_schemas(listing=...) embeds it and instructs the
model to skip tool_search when the exact name is visible (one fewer
round-trip per use)
- config: tools.tool_search.listing auto|on|off (default auto),
listing_max_tokens (default 4000, clamped 200..20000); legacy bool
shapes keep working
- tests: 8 new tests (config parsing/clamps, short-desc clipping,
deterministic rendering, budget fallbacks, bridge embedding, assembly
on/off paths); full file green (47 passed)
- docs: tool-search.md config table + rationale
- scripts/tool_search_livetest2.py: benchmark harness v2 with real
per-call token accounting (normalize_usage spy) and a third 'listing'
mode for A/B/C comparison
The sidebar strip and the Channels page could contradict each other on
the same page load — "Gateway running" next to "The gateway is not
running." /api/status and /api/messaging/platforms each open-coded their
own liveness ladder: status probed GATEWAY_HEALTH_URL and scoped its
PID/state reads to the requested profile, messaging did neither and used
the uncached raw PID probe.
Three deployments hit the split: a cross-container gateway (no local PID,
only the health probe can see it), a profile-scoped dashboard (messaging
borrowed a DIFFERENT profile's runtime state, reporting a false
"connected" that hides a real outage — #71211), and a launch-service
managed gateway with no PID file.
Adds resolve_gateway_liveness() in gateway/status.py as the single ladder
(cached PID -> HTTP health probe -> runtime-status PID with
expected_home) and routes both endpoints, /api/messaging/platforms/{id}/test,
and the kanban dispatcher-presence probe through it. Probe callables are
injectable so the existing monkeypatch seams keep working, and
GatewayLiveness.probe_error distinguishes "down" from "couldn't tell" so
the kanban warning keeps failing OPEN instead of crying wolf.
Closes#71211.
Previews correct themselves on read, but a title already written to the DB
stays wrong. This regenerates those titles from what the user actually typed,
dry-run by default.
Two guards, both hit on a real store: a candidate that isn't title-shaped is
rejected rather than replacing a serviceable title with command output, and a
unique-title collision dedupes through the lineage the way the live auto-titler
does instead of leaving the leaked title in place.
Reported in Discord by @spherohero: `sessions recover --allow-partial` copied
20,817 of 20,824 message rows, then orphan cleanup deleted every one of them
because no session row was salvageable. Final output: 0 sessions, 0 messages.
The salvage worked and then threw the result away -- the exact opposite of
what --allow-partial exists to do.
_cleanup_partial_orphans() removed dependent rows whose session_id had no
matching sessions row. That is correct when a few sessions are lost; it is
catastrophic when the sessions b-tree is damaged worse than messages, which
is the common shape (sessions is small and hot, messages is large).
Reproduced exactly: 500 readable messages, unreadable sessions -> 500
copied, 500 removed, empty output.
Now _reconstruct_missing_sessions() runs FIRST, inside the same transaction,
synthesizing a minimal row per orphaned session_id (only id/source/started_at
are NOT NULL). started_at comes from the earliest surviving message.
Placeholders carry source='recovered' and an explicit title so a fabricated
session can never be mistaken for an original. Same repro now retains all
500 messages under 1 reconstructed session.
Reconstruction is reported as LOSS, not a clean recovery: session metadata
(title, model, timestamps, cost) is genuinely gone even though the
conversation text survived. loss_detected=True, partial=True,
complete=False, with a warning naming the counts.
Also fixes total_removed_or_relinked, which summed every dict value and would
now have counted retained messages as removed.
Sabotage-verified: removing the reconstruction call restores the wipe and
fails the test. 942 targeted tests green.
Second round of @helix4u review on #71779. Both findings reproduced before
fixing.
1. My previous fix turned a crash into SILENT DATA LOSS. Returning
status="missing" for a present-but-unusable state_meta looked like a safe
degrade, but _verify_recovered_database only escalates "failed"/"partial"
into a warning + loss_detected. Measured on the branch: a run that dropped
a real metadata table reported warnings=[], loss_detected=False,
partial=False, complete=True. Strictly worse than the ValueError it
replaced -- that at least failed loudly.
Now "failed" when the table exists but lacks key/value, "missing" only
when genuinely absent. The damaged case yields
warnings=['state_meta copy status is failed'], loss_detected=True,
partial=True, complete=False, while staying verified=True so the output
is still installable-with-review.
2. The race test I wrote had its own scheduling race: after the guard
released the lock, the racer could win before the main thread set the
release event, failing on a correct implementation. Rewritten per
helix4u's design -- copy runs in a worker parked inside the patched
copy, a second worker attempts connect_tracked(), assert it stays blocked,
release, assert it then opens. Deterministic and ~1.1s instead of 10s;
12/12 stable.
Sabotage-verified. Note the third scenario only failed after adding a
unit-level test: recover_session_database short-circuits on the inspection
result when state_meta is entirely absent, so the helper's absent-branch is
unreachable end-to-end and a regression there was invisible. Both statuses
are now pinned directly.
939 targeted tests green.
Post-merge follow-up to #71770. Both defects were found by @helix4u in review
and reproduced against merged main before fixing.
1. Check/use race in _copy_source_bundle (my bug, from the #71770 follow-up
commit). It called has_live_connection(), released the registry lock, and
only then ran shutil.copy2() over the bundle. A tracked connection could
open in that window; the copy's close() then cancels its POSIX advisory
locks -- the exact class #71724 closed. Measured on main: a racer thread
opened a connection mid-copy after blocking 0.000s.
Adds sqlite_safe_read.offline_file_access(), a context manager that holds
the connection-lifecycle lock across an entire multi-step raw access, and
routes the bundle copy through it. Same racer now blocks 10.0s until every
raw descriptor is closed. Any future raw read of a database file (hashing,
moving a bundle aside) should use this rather than a bare pre-check.
2. _copy_state_meta_salvage assumed a 'key' column. A damaged state_meta can
keep 'value' and lose 'key'; columns.index("key") then raised ValueError
and aborted the whole partial recovery. The mirror case (key without
value) would have copied key-only rows and reported the table complete.
Now requires both, matching the non-partial _copy_state_meta, so an
unusable optional table is recorded as missing/failed and --allow-partial
still recovers sessions and messages.
Both regression tests verified by sabotage: reinstating the bare pre-check
fails the race test, removing the key/value requirement fails the other.
937 targeted tests green.
Adds build_aux_picker_rows() + format_aux_picker_entries() to
hermes_cli/inventory.py and converts both auxiliary-task pickers to them,
so custom providers, exclusions, and picker visibility can no longer be
dropped one call site at a time.
The two salvaged commits each fixed one kwarg at these same two sites
(user/custom providers, then for_picker). Both were per-site patches, so
the next aux picker would have reintroduced the gap. The seam makes the
correct behaviour the default a new caller cannot forget:
- user `providers:` and saved `custom_providers:` entries
- `model_catalog.excluded_providers`, matching /model
- exhausted-credential-pool providers stay visible
- only the active custom endpoint is probed, so an offline saved local
server can't hang the picker
- the virtual `moa` row is excluded (auxiliary_client unwraps it to the
aggregator anyway, so offering it is a silently-rewritten choice)
The vision picker also gains the current-selection marker it never had.
Also threads for_picker through build_models_payload, which had no way to
express it despite list_authenticated_providers supporting it.
credential pool has entries but is entirely rate-limited (exhausted): rate
limits are per-model for many providers (e.g. Google Gemini), so an exhausted
key for model-A may still serve model-B, and the user should stay able to
select it. That fix wired `for_picker=True` into `list_picker_providers`.
The two sibling interactive pickers reached by `hermes model` still called
`list_authenticated_providers()` WITHOUT `for_picker=True`, so they kept
hiding exhausted-pool providers:
- `_aux_select_for_task` (hermes_cli/main.py) — the auxiliary-task
provider/model picker.
- `_configure_vision_provider_model` (hermes_cli/tools_config.py) — the vision
provider/model picker.
Both persist a per-task config the user runs *later* (once the momentary
cooldown clears), so hiding an exhausted-pool provider here is exactly the
Tests: two regressions assert each picker requests `for_picker=True` (they
omitted it before the fix); the existing picker/exhausted-pool suite still
passes (6 total).
2107b86024 flipped compression.in_place to True but left both explanatory
comments reading "Default False during rollout". The contradiction is
load-bearing: it is why two recent PRs (#71747, #48951) were built on the
premise that agent.session_id still rotates at every compaction.
Comment-only; no behavior change.
Follow-up to the partial-recovery salvage. _copy_source_bundle() raw-copied
the source state.db and its -wal/-shm sidecars with no live-connection check.
Copying a database file is an open()/close() on it, and close() cancels every
POSIX advisory lock the process holds on that file -- including a running
VACUUM's EXCLUSIVE lock (fe431651c5). hermes_state._backup_db_file already
refuses that situation; this path did not, leaving two policies for one
hazard. Verified against the merged guard: with a tracked connection open,
_backup_db_file returned None (refused) while _copy_source_bundle copied
anyway.
Recovery normally runs as its own short-lived CLI process against an
offline/quarantined file, so this should never fire in practice. It is a
consistency fix, not a live corruption path -- but it is exactly the drift
that reintroduces the bug later.
Regression test verified by sabotage: removing the guard fails it.
Addresses review findings on the previous commits. Three of them were real
defects I reproduced against my own head before fixing.
1. Check/use race (BLOCKING). read_header_bytes_preopen() checked
has_live_connection() under _live_lock, released it, then did the raw
open/read/close outside the lock; connect_tracked() opened before
registering. A thread could pass the "nothing is live" check, another
could open a connection and BEGIN IMMEDIATE, and the first thread's
close() then cancelled its POSIX locks -- the exact bug this guard
exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED).
_live_lock now spans all three lifecycle transitions: open+register,
unregister+close, and check+open+read+close.
2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's
read-only path opens file:/…/state.db?mode=ro; that string was fed to
Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of
the real Path could match, so read-only connections were invisible to
the guard and their locks cancellable. Reproduced with no forced
scheduling. Keys now come from PRAGMA database_list (canonical path),
with an explicit tracking_path override.
3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception
and retried an untracked plain connect, so any error silently disabled
the guard. Now only ImportError (scaffold installs without hermes_cli)
falls back; real failures propagate.
4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db()
and _backup_db_file() raw-read live databases; they now REFUSE when a
connection is live rather than warning. Losing a forensic copy beats
corrupting the database being rescued.
Custom factories are no longer rejected (that broke legitimate callers) nor
silently untracked -- the tracking close() is mixed into whatever factory is
in play, including when an opener substitutes its own after the fact.
WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was
confounded -- the clean WAL result came from 3.53.1, which carries both the
WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the
bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and
DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the
WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL
until a fixed runtime ships is the conservative call, and the WAL policy does
not belong in this root-cause fix.
Six sabotage runs confirm each new test fails when its defect is reinstated
(including two that initially did NOT -- the race test was rewritten to pause
inside the byte read, and a separate test added for the opener-substituted
factory path). 1124 targeted tests green.
Completeness pass over the previous commit.
Registry leak (would have silently disabled the guard): the first version
incremented on open but decremented only in SessionDB.close(). kanban's
connect() hands raw connections to callers who close them directly (4 sites),
so its counter only ever went up — after enough kanban operations every
byte-probe on that path would be refused forever, disabling zeroed-file and
header detection. Replaced manual track/untrack with a TrackedConnection
subclass that untracks in close(), the one method every close path goes
through. Verified across plain close, contextlib.closing, double close,
nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not
a close) correctly stays tracked.
Also: a caller-supplied factory now wins instead of raising TypeError on a
duplicate kwarg, since tracking is an optimisation for the probe guard, not a
precondition for opening the database.
Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert.
Audit notes:
- DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains
correct there; those users are protected by the raw-read fix, not by the
journal mode.
- The remaining whole-file reads are on genuinely offline artifacts:
_backup_db_file (DB won't open; bytes preserved for forensics),
_backup_corrupt_db (quarantine path, now warns if a connection is live),
and backup verification of snapshots. backup._safe_copy_db already uses
the SQLite backup API rather than a byte copy.
- The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped,
not process-scoped), so this is a Linux/macOS correctness fix; the change is
platform-neutral and safe on Windows.
Sibling tests updated: session recovery no longer expects a DELETE-mode
recovered DB, and the kanban WAL-fallback test patches the connect site that
actually runs now.
`hermes sessions optimize` could corrupt state.db. Root cause is Hermes,
not the SQLite WAL-reset bug (#69784).
close() on ANY file descriptor for a SQLite database cancels every POSIX
advisory lock the process holds on that file, including a running VACUUM's
EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed
live databases in several hot paths: the zeroed-state.db detector runs on every
SessionDB construction (and the gateway builds those constantly), and kanban's
post-commit invariant check ran after every COMMIT. While VACUUM rewrote the
file, those probes dropped its lock and let other processes write into it.
A/B against the real code, only variable being the raw read:
SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode
raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt
no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt
SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt.
After this change: 0/4 corrupt, 0 vacuum errors.
Because the upgraded runtime corrupts too, replacing the embedded SQLite does
not fix this class; and because DELETE is where it reproduces, #70055's
"force DELETE on vulnerable builds" mitigation steered users into the failing
mode. That gate is reverted here: vulnerable builds get WAL again and still
warn so operators can upgrade.
- add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the
existing connection instead of open()+seek(28); byte-level probes are
restricted to before any connection exists and refused once one is live,
with an explicit force= escape for offline artifacts (snapshots, archives)
- track live connections in SessionDB and kanban's connect so that guard is
enforced rather than merely documented
- kanban's torn-extend check now only applies under a rollback journal; in WAL
a committed page may still legitimately sit in the -wal file
- revert the force-DELETE WAL gate and update the tests that pinned it
Regression tests assert the behavioural contract (an external process stays
locked out across Hermes' inspection calls) and were verified to fail when the
old raw-open behaviour is restored.
The background write guard decided ownership from `isinstance(usage_rec, dict)`,
so a local skill with NO usage record passed. That successful write called
bump_patch(), which created a `created_by: null` record — and the identical
write was refused from then on. "Allowed exactly once, then never" is a race
with our own bookkeeping, not a policy. Reproduced on main: patch #1 succeeds,
patch #2 with the same arguments is refused.
Option B from the issue. Option A (split `session_review` from
`scheduled_curator` and let the session fork patch user-owned skills it
consulted) would widen autonomous write permission onto skills the user owns
with no user present to consent — wrong direction for a no-user-present actor.
- skill_manager_tool: missing and explicit-null records now resolve
IDENTICALLY, both fail closed. The refusal names the reason and points at
`hermes curator adopt <name>`.
- background_review: both review prompts told the reviewer to patch any skill
consulted in the session and claimed pinned skills could be improved, while
enforcement refused both. Prompts now list pinned, external, and user-owned
skills as protected, and tell the reviewer to RECOMMEND adoption instead of
attempting a write that will be refused.
- skill_usage: document that `created_by` is a curator-management policy flag,
not a provenance claim, and add `is_curator_managed()` so call sites read as
the question they ask. Field name retained — it is on disk in every
`.usage.json` and renaming would strand those records.
- curator CLI: `hermes curator list-unmanaged` itemizes unmanaged skills with
the reason each is unmanaged (completes the #67139 spec).
Foreground writes are untouched: a user-directed edit to a user-owned skill
still works, including on pinned skills.
Sibling tests: 9 failures in test_skill_manager_tool.py were fixtures that
created record-less skills to exercise OTHER guards (consolidation-delete,
read-before-write) and relied on ownership falling through. Fixed at the
fixture, since the real curator only ever operates on managed sediment. One
test asserted the old "manually authored" wording; rewritten to assert the
behavior contract instead of the string.
Validation: 274 targeted tests + all 7 background-review files (60 tests) pass.
E2E on a temp HERMES_HOME (30 checks) covers the flip, foreground writes,
adoption unblocking, pin semantics, prompt/enforcement parity, and the new verb.
Each new test sabotage-verified: revert the fix, confirm it goes red.
Fixes#67140
The residual Windows-on-ARM integrity-gate fix added GetNativeSystemInfo as a
second "OS-native" probe behind IsWow64Process2. Microsoft documents the
opposite behavior: "the API GetNativeSystemInfo also returns emulated
processor details when run from an app under emulation."
On the exact hosts the probe was added to rescue -- an x64 hermes-setup.exe
emulated on an ARM64 Surface -- it therefore reports AMD64, making it a
duplicate of the PROCESSOR_ARCHITECTURE rung already below it rather than a
fallback for it. Its test only passed because the kernel32 fake was hand-fed
ARM64, asserting behavior real Windows does not exhibit.
Replace it with GetMachineTypeAttributes, which answers the question the gate
actually asks -- "can this host load a PE of machine X?" -- instead of
inferring it from an architecture name. It is also the only documented API
that reports AMD64-on-ARM64 emulation support. The gate prefers it and falls
back to the existing name-based mapping on pre-Windows-11 hosts.
The load-bearing half of the previous fix (typing GetCurrentProcess as HANDLE
so IsWow64Process2 stops failing ERROR_INVALID_HANDLE) is unchanged.
Co-authored-by: xxxigm <xxxigm@users.noreply.github.com>
Co-authored-by: Teknium <teknium1@users.noreply.github.com>
`hermes curator status` reported only the skills it manages, staying silent
about curation-eligible skills it can never touch. On a 237-skill library that
meant 112 skills were invisible to every automatic transition with no signal
anywhere — the curator looked broken when it was working as designed.
A skill becomes curator-managed only when `created_by: agent` lands on its
usage record, and only the background review fork writes that marker. Two
populations therefore never qualify: records written before the marker existed
(no key at all, authorship unknowable) and every foreground
`skill_manage(create)` (unset by design — those skills belong to the user).
- skill_usage: `list_unmanaged_skill_names()` / `unmanaged_report()` enumerate
the blind spot, tagging each row with `has_provenance_key` so the two causes
are distinguishable. `adopt_skill()` writes the marker on user declaration
and refuses bundled, hub-installed, external, and protected built-ins.
Adoption never resets the inactivity clock.
- curator CLI: status prints an `unmanaged (no provenance marker)` block on
BOTH the managed and no-managed-skills paths; new `adopt` verb takes names or
`--all-unmanaged`, with `--dry-run` and a confirmation prompt on bulk.
- skills_sync: `_backfill_optional_provenance()` matched candidates by
repo-derived path only, so a skill installed at `mlops/chroma` that upstream
later moved to `mlops/vector-databases/chroma` was skipped forever and
`hermes skills repair-optional` could not fix it. Falls back to an
unambiguous name match, still gated on identical content, and records the
ACTUAL install path.
Provenance stays a declaration, never an inference: a high patch count proves
the agent MAINTAINS a skill, not that it authored one, since Hermes edits
user-written skills on the user's behalf routinely. An "looks agent-made"
heuristic would eventually archive hand-written work.
Validation: 347 targeted tests pass. Each new test verified via sabotage run
(revert the fix, confirm the test goes red) — one initially passed for the
wrong reason because the fixture pinned `prune_builtins` off, masking the guard
under test; fixed to force the shipped default on.
The retry loop skipped only the current version, so on a uv whose download
catalog is stale it walked backwards through older patches. In #71250 the
newest indexed 3.11 is 3.11.14 — the installed version — so all five retries
went to 3.11.13..3.11.9, each a real download+install+probe+delete cycle that
the existing downgrade guard was always going to reject, before failing anyway.
Only newer patches can carry the SQLite fix. Skipping <= current makes the
stale-catalog case cost one attempt instead of six, and still finds 3.11.15
on the first retry when the catalog knows about it.
Fixes#71250.
`hermes update` could not repair the embedded Python runtime's
vulnerable SQLite (WAL-reset bug range 3.7.0-3.51.2, except backports
3.50.7/3.44.6) on installs where `uv python install <minor>` (bare
request, e.g. "3.11") resolves to an older cached/indexed patch that
still links a vulnerable SQLite build, even though a newer
non-vulnerable patch on the same minor line is available and known to
uv. The smoke test correctly rejected the vulnerable candidate every
time, but the provisioner gave up immediately after one attempt,
leaving the repair permanently stuck in the same failing loop on every
`hermes update` run.
Implements option D from the issue (query the index, retry with an
explicit newer patch), which the reporter identified as cleanest:
- New `_list_available_patches()`: runs `uv python list <minor>
--all-versions --only-downloads --output-format json --no-config`,
filters to cpython/default-variant entries (excluding pypy/graalpy),
and returns known patch versions newest-first. Fails safe (returns
[]) on any network/parse error.
- Refactored the single install+find+probe cycle out of
`_install_safe_python_generation()` into `_attempt_install_generation()`,
reusable per attempt with its own generation directory (so a
rejected candidate's files are fully cleaned up before the next
attempt, matching the existing --reinstall semantics).
- `_install_safe_python_generation()` still tries the bare minor-line
request first (preserves the original comment's rationale: for a
given exact patch, python-build-standalone may have no artifact with
fixed SQLite at all). If that resolves vulnerable, it now queries
`_list_available_patches()` and retries with explicit newer patches,
newest-first, bounded to `_MAX_PATCH_RETRIES` (5) attempts -- each
attempt is a real download+install+probe cycle, so the cap keeps
worst-case repair time bounded.
Sanity-checked `_list_available_patches()` against the real `uv`
binary (0.11.7) in this environment: correctly parses live `uv python
list --all-versions --output-format json 3.11` output, returns 14
patches sorted newest-first starting at 3.11.15.
9/9 new tests pass (retry succeeds with a newer patch, exhausts
gracefully when every known patch is vulnerable, empty patch list
degrades to None without crashing, retry count is bounded, plus direct
JSON-parsing unit tests for realistic/malformed/empty uv output);
47/47 in the full tests/hermes_cli/test_managed_uv.py file (including
the 3 pre-existing tests for the original bare-minor success path,
confirming no regression there).
A failed `hermes sessions repair` printed "keep state.db and the backup"
and stopped, so a user whose sessions had vanished had no way to discover
that a non-destructive recovery path exists — the reported dead end.
The failure branch now names the next command, read-only step first, seeded
with the backup path it just preserved. Covered by a CLI-surface regression
test that drives the real subprocess against an unrepairable database.
* fix(update): bind IsWow64Process2 HANDLE and fall back to GetNativeSystemInfo
The #71218 OS-native probe still rejected correct ARM64 Desktop rebuilds on
Windows-on-ARM when ctypes truncated GetCurrentProcess()'s pseudo-handle and
IsWow64Process2 failed with ERROR_INVALID_HANDLE, falling through to the
lying PROCESSOR_ARCHITECTURE=AMD64 value. Type the HANDLE correctly and use
GetNativeSystemInfo before the env-var fallback.
* test(update): cover WoA IsWow64Process2 handle failure and system-info fallback
Pin the residual #71218 shape where IsWow64Process2 returns FALSE, the env
arch lies as AMD64, and GetNativeSystemInfo must still report ARM64 so the
integrity gate accepts a correctly-built ARM64 Hermes.exe.
`sync_skills()` keys the bundled manifest by frontmatter name but computes
the destination from the bundled path. When upstream renames or
recategorizes a skill, the manifest key still matches while the new dest
does not exist yet, so the loop fell into its "in manifest but not on
disk" branch and misread the skill as user-deleted: the user's copy was
stranded at the old path forever and never received another update.
Three skills hit this in the July 2026 reorg (computer-use,
evaluating-llms-harness, serving-llms-vllm) — silently frozen at their
pre-rename content on every machine that ran `hermes update`.
Recovery only moves a stale copy when it is byte-identical to the origin
hash recorded the last time sync wrote it, which proves the directory is
ours rather than the user's work. User-modified copies are kept in place
with a warning, hub-installed paths are never touched, and a genuine
deletion (no copy anywhere on disk) is still respected.
- tools/skills_sync.py: add _recover_renamed_skill() plus the
_index_active_skills() / _read_hub_install_paths() indexes; call it
before classification and report moves via a new `relocated` key.
- hermes_cli/main.py: surface relocations in both `hermes update` skill
sync reporting sites.
- tests: 4 cases covering relocate, user-modified preservation,
hub-installed exemption, and genuine-deletion respect.
Some OpenAI-compatible endpoints — notably Tencent Copilot
(copilot.tencent.com) — only accept streaming chat requests; any
non-streaming call returns HTTP 400 (code 11101, 'Non-stream chat
request is currently not supported'). The main conversation loop already
streams, so interactive chat works, but every auxiliary task (title
generation, compression, web extraction) used the non-streaming path and
failed on each call.
_provider_requires_stream() detects stream-only endpoints
(copilot.tencent.com built in, plus user-configurable
auxiliary.stream_only_base_urls substring markers in config.yaml).
Matching sync auxiliary calls route through _create_with_progress
(force_stream=True) and async calls through the new
_acreate_with_stream, aggregating the chunk stream — including tool-call
deltas and reasoning deltas — into a complete response via the shared
_ChatStreamAccumulator.
Salvaged from PR #60686 by @kudi88 onto the progress-aware streaming
machinery from #71508, addressing both sweeper-review gaps: the async
path now consumes the stream with 'async for' (awaiting create() and
iterating synchronously raised on AsyncOpenAI streams), and tool-call
deltas are reassembled instead of dropped (MCP passes tools= through
call_llm). Under force_stream there is no silent non-streaming retry —
a stream-only provider rejects those by definition, so the original
error surfaces to the normal recovery chains.
The gateway's pre-agent session-hygiene compression killed the summary
call at a fixed 30s wall-clock deadline (compression.hygiene_timeout_seconds),
regardless of whether the summary model was hung or merely slow. A reasoning
model happily streaming a large summary was cut off mid-generation, the user
got '⚠️ Context compression timed out after 30.0s', and a 300s failure
cooldown left the session oversized — a doom loop for slow-but-healthy
auxiliary models.
Timeouts are now liveness-based instead of wall-clock-based:
- agent/auxiliary_client.py: new thread-local aux_progress_hook. When
installed (only by context compression today), the primary call_llm
attempt streams (stream=True) and aggregates chunks back into a complete
response, ticking the hook per chunk. The configured timeout then acts
per stream read (idle) instead of as a total budget. Providers that
reject streaming fall back to the plain non-streaming call; auth/payment/
rate-limit/transport errors propagate unchanged into the existing
recovery chains. Codex Responses (per SSE event) and Anthropic Messages
(per stream event, via the new create_anthropic_message on_stream_event
callback) tick the same hook from inside their wire adapters.
- agent/conversation_compression.py: CompressionCommitFence gains
touch_progress()/seconds_since_progress(); compress_context() installs
fence.touch_progress as the progress hook around the compress call.
- gateway/run.py: the hygiene wait loop treats hygiene_timeout_seconds as
an INACTIVITY budget — while the fence reports fresh progress the wait
extends, bounded by the new compression.hygiene_total_ceiling_seconds
(default 600s, clamped >= the idle budget) so a degenerate trickle
stream still dies. The timeout warning now says the summary model
produced no output, which is the only case that still triggers it.
- config/docs: hygiene_total_ceiling_seconds added to DEFAULT_CONFIG and
configuration.md; hygiene_timeout_seconds documented as inactivity-based.
Tests: tests/agent/test_aux_progress_streaming.py (hook plumbing, stream
aggregation incl. tool-call deltas and reasoning deltas, rejection
fallback, ceiling kill, fence progress surface); two new gateway tests
prove a slow-but-streaming worker survives past the fixed timeout
(sabotage-verified: fails with the old fixed deadline) and a
forever-trickling worker is still cut off at the ceiling.
Reviewer egilewski identified two recovery-loss paths:
Path A — quarantine race (hermes_state.py):
SessionDB checks state.db before quarantine_zeroed_state_db() without a
shared cross-process lock, and Path.rename() may replace an existing
destination. Two writable startups can therefore move the first
instance's newly created database over the .zeroed-*.bak, erase the
original damaged-file evidence, and replace the live database with
another empty one.
Fix: add a cross-process lock (msvcrt on Windows, fcntl on POSIX)
around quarantine_zeroed_state_db() with a 5s bounded timeout. Under
the lock: re-check is_zeroed_state_db (another process may have already
quarantined it and created a fresh DB), use a PID-suffixed unique
destination, and non-clobbering rename with counter fallback.
Path B — size-cap pruning gap (hermes_cli/backup.py):
_too_large() runs before failed-database tracking. With keep=1 and a
size cap, an oversized state.db is omitted while failed_dbs stays empty,
so automatic pruning deletes the older complete snapshot that may
contain the only recoverable database.
Fix: track oversized DB files in a new oversized_skipped list (both in
the directory walk and top-level file loop). The manifest now records
oversized_skipped. Pruning is suppressed when failed_dbs or
oversized_skipped is non-empty, preserving the older complete snapshot
as recovery source.
Tests:
- test_concurrent_quarantine_no_clobber: two threads racing on the
same zeroed state.db — verifies quarantine backup survives with
original bytes and live DB is valid.
- test_oversized_db_suppresses_pruning: keep=1 + oversized state.db
verifies the older complete snapshot is not pruned.
All 33 tests pass (3 zeroed_state_db + 25 TestQuickSnapshot + 5
quarantine_forensic_logging).
Address #68805 review: when failed_dbs is non-empty, skip
_prune_quick_snapshots so the incomplete snapshot does not delete
the older snapshot containing the last good database. The updater's
keep=1 would otherwise evict the recovery source.
Hardening for the Windows zeroed-state.db class (#68474):
- Surface critical stdout when pre-update/quick snapshot cannot copy a
present *.db (was log-only; update still looked successful).
- Detect all-NUL SQLite header on SessionDB open, quarantine the bytes,
and open a fresh DB with recovery guidance to state-snapshots.
Does not claim storage-stack root cause.
The #71119 integrity gate rejected CORRECT ARM64 desktop rebuilds on
Windows-on-ARM (#69179 follow-up): platform.machine() reports the PROCESS
architecture, and the update chain runs an x64 hermes-setup.exe / x64
Python under ARM64 emulation, so the gate compared the ARM64 Hermes.exe
against a phantom 'AMD64 host' and aborted the update.
_windows_native_machine() now asks the OS via IsWow64Process2 (whose
nativeMachine is truthful from emulated processes), falling back to
PROCESSOR_ARCHITEW6432/PROCESSOR_ARCHITECTURE (pre-1511 Win10) and then
platform.machine(). All three consumers — the integrity gate, the
packaged-exe arch preference, and backup validation — go through it.
Sabotage-verified: the three new regression tests fail against the old
process-arch behavior and pass with the fix.
Builds on @ms-alan's label fix: the negative figure had a second cause
that relabelling alone leaves in place.
`hermes sessions optimize-storage` reported "reclaimed -3820.1 MB" on a
database that had in fact shrunk 60% (25069 MB -> 9975 MB). Both figures
came from os.path.getsize(). In WAL mode a VACUUM's rewrite lands in the
-wal file, and the checkpoint that folds it back is REFUSED
(SQLITE_BUSY) while any other connection holds a read-mark — e.g. the
live gateway. So the main file still carried its pre-VACUUM size AND kept
growing while the command ran, making the after-figure larger than the
before-figure. The TRUNCATE checkpoint in close() lands after the caller
has already measured and printed.
- hermes_state.py: add SessionDB.logical_size_bytes() — page_count *
page_size, the size the file settles at once the WAL is folded back.
Correct immediately, readers or not; returns None when the connection
is gone so callers fall back to stat().
- hermes_state.py: best-effort TRUNCATE checkpoint after the optimize
VACUUM so the file settles promptly when nothing else holds the DB.
Documented as insufficient alone (busy under a live gateway) so the
stat() approach is not reintroduced.
- hermes_cli/main.py: both `optimize-storage` and `optimize` report via
logical_size_bytes(); fixing only the reported command would have left
the same bug class next door.
- hermes_cli/main.py: lift the contributor's inline label to a
module-level _size_delta_label() shared by both sites, so the "grew by"
wording is consistent and unit-testable without reading source.
The two halves are complementary: page accounting removes the phantom
negative, and "grew by" still covers a genuine increase when concurrent
session writes outweigh what the rebuild freed.
Verified: sabotaging logical_size_bytes() back to stat() fails the new
test with a 22 MB overstatement, reproducing the report. The test asserts
its own precondition (stat() must actually be lagging) so it cannot
silently stop exercising the bug.
Tests: 464 passed (test_hermes_state.py + the new label tests).
Co-authored-by: chenbin <h-chenbin@voyah.com.cn>
Closes#70146
When hermes sessions optimize-storage runs and the database grows
(rather than shrinks), the old code always printed '(reclaimed X MB)'
with a negative value. Fix by using a conditional label:
- positive delta → 'reclaimed X MB'
- negative delta → 'grew by X MB'
Leg 2 of #63078: prompt.submit returns {"status":"streaming"} immediately and
runs _start_agent_build + _wait_agent(timeout=30s) behind it. The deferred
build (MCP discovery with per-server retry backoff, synchronous model-metadata
HTTP, skills scanning) routinely outlives 30s on cold starts; on timeout
run_after_agent_ready emitted an error EVENT and returned without ever calling
_run_prompt_submit — the user's first message was permanently discarded while
the build finished successfully in the background. The desktop's optimistic
row eventually cleared with no visible error: the blank first session.
New _wait_agent_for_prompt replaces the flat cliff for the deferred prompt
path only (_sess()'s RPC-blocking _wait_agent keeps its 30s contract):
- The pending prompt stays attached to the (already off-RPC) run thread and
is delivered the moment the still-running build completes — a slow build is
no longer message loss.
- The wait runs in 5s slices so a cancel (session.interrupt / churn) is
honored promptly; the cancelled path returns None and defers to the
caller's cancel branch (the #65567 emit) for user-visible messaging.
- Past 30s the client gets ONE keyed notification.show ('Still starting the
agent…', key=agent-build-slow, desktop toast / TUI status bar), cleared on
delivery — patient, never silent.
- Permanent failure only when the build itself fails: agent_error set at
ready, the build thread died without signalling ready (fail fast via the
new _agent_build_thread handle instead of sitting out the cap on a corpse),
or the bounded cap expired on a genuinely hung build. The cap defaults to
600s and is tunable via agent.build_wait_timeout in config.yaml (no new
env vars); the error message states the message was not sent.
Tests: slow-build delivery with zero error events; the keyed progress notice
shown once and cleared; build-failure surfacing exactly one error event with
the real reason; dead-thread fail-fast; cancel honored mid-wait; config
override + fallback semantics; cap expiry message. The compute-host fallback
test stubs the new waiter alongside _wait_agent.
The post-update state.db integrity guard called verify_sqlite_integrity()
with max_bytes=0, which disables the size ceiling and forces a full
PRAGMA integrity_check. That pragma walks every b-tree page in the file,
so its cost scales with database size — measured on a real 30 GB state.db:
143.5s with a cold page cache (worse under an update's memory pressure),
with zero output on screen. The update looks hung right after
"✓ Code updated!" and a CPU sits pegged.
Multi-GB session databases are normal for heavy users, so a
size-unbounded check is never an acceptable default on the update path.
- verify_sqlite_integrity(): max_bytes now defaults to
DEFAULT_INTEGRITY_CHECK_MAX_BYTES (2 GiB) instead of 0. max_bytes=0
remains the explicit opt-in for a full scan.
- The oversized path no longer degrades to a header-only check: it adds a
constant-time structural probe (read-only open + schema_version +
sqlite_master read) so the malformed-schema class is still caught, not
just the #68474 zeroed-file signature.
- Drop the explicit max_bytes=0 at the post-update guard and in
copy_db_and_verify() so both inherit the bounded default.
Measured on the reporter's real 30 GB state.db: 143.5s cold → 0.001s,
still valid=True. Corruption detection verified at multi-GB scale for
both classes (zeroed header, malformed schema) — both still fail closed.
Tests: default-is-bounded invariant, oversized probe catches malformed
schema, max_bytes=0 still forces the full check.
The renderer's session-state cache is memory-only and the backend's
inflight snapshot dies with the backend process, so nothing survived a
full app or machine death mid-turn: reopening the session showed the
transcript up to the last committed turn and silently dropped everything
the crashed turn had streamed.
While a turn runs, the visible tail (user prompt + streamed assistant
rows, tool calls included) is now journaled to localStorage — throttled
off the delta-flush hot path, bounded (24 entries / 7 days), cleared the
moment the turn settles. Session resume folds the journaled tail back
onto the restored transcript. When the backend also has a live text-only
inflight projection for the same turn, the journal overlays its richer
structure onto that row (longer text wins, base row id kept so live
deltas keep landing) instead of treating it as caught up — the ordering
defect that dropped locally recorded tool progress in the original PR.
Co-authored-by: Omar Baradei <omar@kostudios.io>
Follow-up cleanup for PR #71123:
- Extract getattr(args, 'lineage', 'single') == 'logical' to a local
(appeared 3x in the export block)
- Document that the double _collect_delegate_child_ids traversal in
delete_session is an intentional TOCTOU guard inside the write txn
hermes model's custom-endpoint flow is the other write path that produced a
plaintext key, on both the model block and the custom_providers entry. Route
it through the same .env indirection as the Desktop panel, and swap an
existing entry's inline key for the reference when the URL is re-saved.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
