Port from nearai/ironclaw#5149 (the describe-first live-hardening fix in
their progressive tool disclosure work): when a model invokes a deferred
tool through the tool_call bridge without the schema-required arguments,
return the tool's parameter schema instead of dispatching blind.
Pre-fix, a blind call produced an opaque downstream failure
("[TOOL_ERROR] Tool execution failed: KeyError: 'document_id'") that
teaches the model nothing about what the tool expects — IronClaw observed
cheap models looping ~30 identical invalid calls until the iteration
budget died. Post-fix, the model repairs the call in one round-trip.
- tools/tool_search.py: new validate_deferred_call_args() — key-absence
check of schema 'required' fields only; no type checking (coerce_tool_args
already repairs types downstream); fails open on any validator error so
it can never block a legitimate dispatch.
- model_tools.py: probe after the scope gate in the bridge dispatch.
- agent/tool_executor.py: probe in both unwrap sites (concurrent +
sequential) before the underlying tool replaces the bridge; sequential
path flattens the payload to match its {"error": str} wrapping.
- tests: TestDeferredCallSchemaProbe — blind call returns schema (not
KeyError), valid/optional calls dispatch, unvalidatable tools fail open,
out-of-scope rejection unchanged.
Port from openai/codex#33464: GNU rm permutes options, so
`rm build/ -rf`, `rm build/ -r -f`, and `rm build/ --recursive
--force` are equivalent to the flags-first spellings — but every
existing rm pattern required the flag group BEFORE the path, so these
spellings ran with no approval prompt at all (proven live on main).
The hardline floor was NOT affected: protected paths (/, system dirs,
$HOME) match regardless of flag position because the hardline path
matcher does not require flags. The gap was the approval-prompt layer
for arbitrary paths.
New DANGEROUS_PATTERNS entry with a tempered operand run: cannot cross
command separators (; | & newline), quotes, or a bare -- end-of-options
separator (after --, -rf is a literal filename). Flag token must be
whitespace-anchored so the r inside long options like --registry does
not count.
9 positive + 7 negative shapes in tests; approval cluster (868 tests)
green.
Port from anomalyco/opencode#38133/#38134 (patch Unicode matching corpus):
extend UNICODE_MAP with the Zs space-separator family (en/em quad, en/em/
three-per-em/four-per-em/six-per-em/figure/punctuation/thin/hair spaces,
narrow NBSP, medium mathematical space, CJK ideographic space) and the
Unicode minus sign U+2212.
Before: a file containing typographic spacing (French narrow NBSP, CJK
ideographic spaces, math minus) never matched a model's ASCII old_string
via the precise strategies — the edit only succeeded through the
similarity-based context_aware fallback, which (a) can pick the wrong
region (#54572 family) and (b) silently flattens the file's Unicode to
ASCII on replacement. After: these match at unicode_normalized (strategy
7), whose _preserve_unicode_in_replacement keeps the file's typographic
characters in unchanged spans.
All additions are 1:1 mappings, so the existing position-mapping and
preservation logic apply unchanged. Proven live before/after with a
multi-line probe; 59 fuzzy-match + 188 file-tools/patch/skill-manager
tests pass.
Inspired by Claude Code 2.1.214, which added permission prompts for
container-CLI commands (including the Podman shim) carrying
daemon-redirect flags (--url, --connection, --identity, remote mode)
that previously ran without one.
A daemon redirect makes a local-looking command operate on a different
(often remote) daemon, silently acting on production infrastructure.
Any container-CLI invocation carrying a redirect now requires approval
regardless of subcommand:
- -H/--host and --context global flags (value required, global-flag
position only — bare -h help and run-level -h <hostname> stay allowed)
- context use (persistently switches the default daemon)
- podman --url/--connection/--identity and -r/--remote
- DOCKER_HOST=/DOCKER_CONTEXT=/CONTAINER_HOST=/CONTAINER_CONNECTION=
environment prefixes
Sibling-site widening: the existing container lifecycle rules matched
only the verb directly adjacent to the binary name, so a global flag or
a compose -f file flag slipped past the guard, and the legacy hyphenated
compose binary was never covered. They now tolerate global flags — the
same treatment the 'hermes ... gateway' rule already has — and match the
hyphenated compose binary.
Validation: 33 new tests; 339 pass in test_approval.py + new file;
442 pass across the adjacent guard suites; E2E battery of 12 dangerous
+ 17 safe commands via real imports, hot path ~330us/call.
Regression for #71296: dump+source must not execute continuation lines from
HERMES_SESSION_CHAT_NAME/USER_NAME, and the export snippet must unset by
name instead of grepping declare lines.
Widen the cherry-picked /diff base (#4839 by @SHL0MS) into one
cross-surface implementation, folding in the review feedback and the
best ideas from the two sibling PRs (#22703, #53527):
- tools/working_diff.py: shared git collection layer — unstaged
(default), staged, and all (vs HEAD) modes; untracked files folded in
via `git diff --no-index` so new files appear as additions (Codex
/diff parity); shlex-split arguments preserve quoted paths.
- CLI: handler moved to hermes_cli/cli_commands_mixin.py per the
current god-file decomposition (dispatch stays in cli.py), renders
through the rich console with a 400-line terminal-flood guard.
- Gateway: _handle_diff_command in gateway/slash_commands.py + dispatch
in gateway/run.py; fenced ```diff output truncated to 60 lines /
3000 chars before the platform senders apply their own per-platform
message clamps (tool-progress-style layered truncation). Localized
strings in all 17 locale catalogs.
- /diff session (from #53527): cumulative checkpoint-baseline diff of
everything Hermes changed, via new CheckpointManager.session_diff();
docstring records the retained-baseline approximation caveat from
review. Works on both surfaces; degrades with an actionable message
when checkpoints are off.
- Slack: /diff routed via /hermes diff (50-slash cap; keeps
telegram-parity test green and /version native).
- Registry: cross-surface CommandDef with staged|all|session
subcommands; docs: slash-commands reference (CLI + gateway tables +
both-surfaces list) and hermes-agent skill reference.
- Tests: tests/tools/test_working_diff.py (real git repos),
tests/hermes_cli/test_diff_command.py (real git + stubbed checkpoint
manager), tests/gateway/test_diff_command.py (end-to-end handler,
real checkpoint store), TestSessionDiff in
tests/tools/test_checkpoint_manager.py.
Salvaged from the /diff PR cluster #4839 + #22703 + #53527.
Co-authored-by: Ninso112 <ninso112@proton.me>
Co-authored-by: Harshkamdar67 <harshkamdar67@gmail.com>
After N consecutive guardian denials in a session the deny message escalates to a hard-stop instruction. Inspired by ChatGPT Work auto-review circuit breaker.
Cross-surface coverage #23768 missed (per review feedback):
- tools/clarify_gateway.py: _ClarifyEntry carries a multi_select flag
(register() accepts it; signature() exposes it to adapters).
_coerce_text_response now parses multi-select replies — comma- or
space-separated numbers ('1,3' / '1 3'), exact labels, dedup — into a
JSON array string that _parse_multi_select_response decodes into a
list. Out-of-range/unknown tokens reject the reply (native button UI)
or fall back to custom text (awaiting_text/'Other' mode).
- gateway/run.py: _clarify_callback_sync accepts multi_select and
registers it on the pending entry.
- gateway/platforms/base.py: default numbered-list text fallback tells
the user multiple selections are allowed and how to reply.
- tui_gateway/server.py: clarify_callback passes multi_select through
the clarify.request payload as a hint; renderers without checkbox
support ignore the field and remain single-select-compatible.
- tests: 13 new gateway tests (flag storage, comma/space/single-number
parsing, label matching, out-of-range rejection, dedup, end-to-end
resolve, single-select regressions).
Follow-ups to the salvaged #23768 commit, which targeted a pre-79559214
codebase:
- agent/tool_executor.py + agent/agent_runtime_helpers.py: pass
multi_select at both current clarify dispatch points (the PR's
run_agent.py edits landed on dead code paths).
- tools/clarify_tool.py: replace the broad TypeError-retry in
_invoke_callback with inspect.signature detection, so a compatible
callback that raises TypeError internally is not invoked twice
(addresses hermes-sweeper review feedback on #23768).
- tests: cover single-invocation on internal TypeError, legacy 2-arg
callbacks, **kwargs callbacks, and registry handler multi_select
pass-through (schema arg → handler → callback).
hermes update's lazy-refresh pass re-asserts LAZY_DEPS pins whenever the
package is present (active_features() is presence-based). The
tool.trace_upload pin huggingface-hub==1.2.3 sat below transformers'
>=1.5.0,<2 requirement, so every update force-downgraded the shared
package and broke Hindsight local embeddings on daemon startup (#60783).
Keep the exact-pin security posture — no ranges — but move the pin to
1.24.0 (current) and bump uv.lock in lockstep (uv lock --upgrade-package
huggingface-hub: hub 1.4.1->1.24.0, hf-xet 1.3.1->1.5.2, click
8.3.1->8.4.2, drops typer-slim), so the entire tree converges on ONE hub
version. The refresh pass now reports 'current' with zero churn.
Invariant tests (not snapshots): the lazy pin must equal the uv.lock
resolved version, and must sit inside transformers' accepted window.
HfApi surface used by trace upload (whoami/create_repo/upload_file)
verified present with identical kwargs on 1.24.0 in a live venv.
Completes #51690 on top of the salvaged #60378 timeout metadata:
- async_delegation: terminal 'stalled' events now carry structured
stall context (stalled_after_quiet_seconds, stall_threshold_seconds,
stall_phase idle|in_tool, stall_grace_seconds) on both single and
batch paths, persisted in the durable row so restart-restored events
keep it. Mirrors the sync path's timeout_seconds/timed_out_after_
seconds/timeout_phase from #60378.
- list_async_delegations(): exposes seconds_since_progress and live
children_activity (per-child api_calls, current_tool,
seconds_since_activity) sampled from the dispatch's progress_fn
outside the records lock; private monitor bookkeeping and callables
never leak.
- /agents (CLI + gateway): background delegations render per-child
activity rows, quiet-time hints, and the stalling state; gateway
section is new (previously async delegations were invisible there).
New locale key gateway.agents.background_delegations in all 17
catalogs.
Tests: stall-metadata event shape, live-listing projection, gateway
/agents rendering (real registry dispatch, sabotage-verified), sync
timeout metadata fields, non-timeout None contract.
Include last_activity_ts in the progress token sampled from each child.
_touch_activity ticks on every streamed chunk ('receiving stream
response'), every tool transition, and API-call start/completion — so a
child mid-stream on a long response is alive even though api_call_count
only advances when the call completes. Same liveness signal as the
compaction inactivity budget (PR #71508): if tokens are flowing it never
dies; staleness is measured from the last streamed token / tool activity
/ API call.
Replace the wall-clock timeout watchdog (from #60234) with progress-based
staleness detection, on by default with zero config:
- The async registry now accepts a progress_fn per dispatch; delegate_task
wires a sampler over the batch's child agents (api_call_count +
current_tool from get_activity_summary()).
- A single monitor thread sweeps running delegations: a child whose
progress token keeps advancing is never touched, no matter how long it
runs. A frozen token past the stale threshold (450s idle / 1200s
in-tool, mirroring the sync-path heartbeat monitor) marks the record
'stalling' and interrupts the child.
- A stalling child that unwinds within the grace window (120s) finalizes
through the NORMAL path, preserving its partial results. One that never
returns is force-finalized with a terminal 'stalled' completion event so
the owning session hears an outcome and the async slot frees.
- Late runner returns after force-finalization are deduped by the
begin/push/finish finalization split (kept from #60234).
Why not a timeout: delegation.child_timeout_seconds defaults to 0 by
deliberate design (DEFAULT_CHILD_TIMEOUT rationale) — a timeout-based
watchdog never arms for default configs, leaving the reported silent-
profile symptom (#60203) unfixed, and when armed it kills legitimately
slow heavy subagents mid-task. Progress detection distinguishes 'wedged
at first API call' from 'grinding through a 2h review'.
Builds on izumi0uu's finalization-atomicity work from #60234.
Async background delegation can leave gateway sessions holding only a dispatched handle when the detached runner wedges before it can return and enqueue a completion. Enforce the configured child timeout in the async registry so the parent observes a terminal timeout event and the async slot is released.
Constraint: Issue #60203 reports long-lived gateway processes with background child delegates that never produce completion events despite child_timeout_seconds being configured.
Rejected: Relying only on _run_single_child timeout handling | it cannot finalize the async registry when the outer runner thread itself never reaches normal completion.
Confidence: high
Scope-risk: narrow
Directive: Keep background delegation completion owned by the async registry whenever detached workers can outlive the caller's immediate control.
Tested: .venv/bin/python -m pytest tests/tools/test_async_delegation.py tests/tools/test_delegate_subagent_timeout_diagnostic.py tests/tools/test_delegate.py -q
Tested: .venv/bin/python -m ruff check tools/async_delegation.py tools/delegate_tool.py tests/tools/test_async_delegation.py
Tested: git diff --check
Not-tested: Multi-day real gateway degradation; covered with deterministic stuck-runner registry tests.
CI slices 2 and 7 caught three tests broken by always-defer:
- /tools (CLI show_tools + TUI gateway tools.show) now passes
skip_tool_search_assembly=True — it's a discovery/inspection surface,
so users verifying an MCP installed must see deferred tools, not a
collapsed bridge row. This also fixes
test_slash_worker_mcp_discovery (profile MCP tool visible in /tools).
- test_plugins.py::test_plugin_tools_in_definitions: 'visible' becomes
'reachable' — direct schema OR listed in the bridge description;
scope-exclusion assertions unchanged (not direct AND not listed).
- test_discord_tool.py dynamic-schema-rebuild test reads the
pre-assembly list (the rebuilt schema is what tool_describe serves).
Banner/status tool counts intentionally keep the post-assembly view —
they reflect what the model actually sees.
The include/exclude filter matched exact names only — glob-style entries
('*_radar_*') silently matched nothing, so a Cloudflare flat-mode config
meant to trim 3,320 tools to ~1,900 actually registered 3,319. Unmatched
patterns produced no warning.
- matches_name_filter(): exact membership first (O(1) for literal lists),
then fnmatch.fnmatchcase for entries containing * ? [ — same pattern
semantics as approvals.deny. Entries without metacharacters stay
strictly literal ('docs' never matches 'docs_search').
- _should_register() uses it for both include and exclude (symmetric)
- hermes mcp tools picker (mcp_config.py) pre-selection uses the same
matcher so the UI agrees with runtime registration
E2E against the live Cloudflare capture with the real exclude list:
3,320 -> 1,905 surviving (1,415 excluded); radar/DLP gone,
purge_cache/dns_records kept. 220/220 mcp_tool tests (4 new).
Teknium review changes on the tiered policy:
1. threshold_pct default 10 -> 5 (listing budget = min(5% of context,
listing_max_tokens)); unknown-context fallback 20K -> 10K.
2. Tier 2 no longer leaves the model blind: when even names-only doesn't
fit, the bridge description now carries a one-line-per-server summary
('cloudflare (3320 tools)') plus an instruction to search FIRST rather
than substitute a generic tool or claim the capability is missing —
the measured tier-2 failure mode (core-tool substitution) at zero
meaningful token cost (~50 tokens/server).
3. Listing degradation is now PER SERVER, largest first: one oversized
server (Cloudflare) collapses to its summary line while small
co-attached servers (Linear) keep their full per-tool listings
('mixed' form). Previously global: attaching Cloudflare next to
Linear silently cost Linear its listing. Greedy fit is deterministic
(size then label) so the rendered block stays byte-stable per catalog
— prompt-prefix cache safe.
E2E on real captures (defaults, 200K ctx): linear alone -> tier 1 full;
unreal alone -> tier 2 groups (5% budget) / tier 1 names at 1M;
cloudflare alone -> tier 2 groups; linear+cloudflare -> tier 1 MIXED
(linear fully listed, cloudflare summarized). 48/48 tests.
Cloudflare's flat API MCP ships 61 property keys that violate Anthropic's
^[a-zA-Z0-9_.-]{1,64}$ pattern (query-filter params like 'issue_class~neq'
and 'meta.<field>[<operator>]'). One bad key anywhere in the tools array
400s the ENTIRE request — measured live: Anthropic, Bedrock, Google Vertex,
and Azure all rejected an eager 3,320-tool Cloudflare request at validation,
before token limits even applied.
- schema_sanitizer: rename non-conforming property keys deterministically
(bad chars -> '_', 64-char truncation, collision dedup with numeric
suffixes), nested schemas included; required[] remapped alongside
- unrename_tool_args(): reverse map applied in coerce_tool_args at dispatch,
so the MCP server receives the original wire names; recurses into object
values and array items
- deterministic on both sides: the rename map is recomputed from the
registry's original schema at dispatch time, no state carried
E2E on the real capture: 61 -> 0 violations across 3,320 tools; round-trip
verified on get_accounts_intel_attacksurfacereport_issues (5 '~neq' keys).
Tier 0: no MCP/plugin tools -> everything eager (pass-through).
Tier 1: deferred tools whose catalog listing fits min(threshold_pct%
of context, listing_max_tokens) -> bridge + skills-style
listing, degrading to names-only over budget.
Tier 2: listing over budget even names-only (Cloudflare's flat API
surface: 3,320 tools, names alone ~32K tokens) -> bare bridge,
discovery through tool_search only.
The old activation threshold (defer only when schemas > threshold_pct
of context) let mid-size catalogs ride eager and pay full schema cost;
with servers like Cloudflare (~597K tokens of schema, would not even
fit a 200K window) the binary gate is the wrong shape. Activation is
now driven purely by deferrable-tool presence; threshold_pct is
repurposed as the listing budget's context-relative leg.
- AssemblyResult gains tier + listing_form for observability
- listing_max_tokens default 4000 -> 20000 (cap 60000) so an 830-tool
catalog keeps a names-only listing while Cloudflare-scale drops to
bare bridge
- E2E verified against real captures: Linear 24 tools -> tier 1 full,
Epic UE 5.8 830 tools -> tier 1 names-only, Cloudflare 3,320 tools
-> tier 2 (both 200K and 1M context)
Deferred MCP/plugin tools become invisible once the tool_search bridge
activates — live benchmarking (48 runs, Claude Haiku 4.5) showed models
substituting visible core tools (terminal/web_search/browser) for deferred
capabilities or declaring them nonexistent instead of searching: 16/24
task success vs 24/24 with eager loading.
Skills never had this failure mode because every skill keeps a ~21-token
name+description listing line in the system prompt. This ports that exact
pattern to the tool bridge: when tool_search activates, a grouped manifest
of every deferred tool (name + first sentence of description, clipped to
60 chars, grouped per MCP server / toolset) is embedded in the tool_search
bridge description.
- tools/tool_search.py: build_catalog_listing() with deterministic
ordering (byte-stable across assemblies -> prompt prefix stays
cacheable); token-budget fallbacks full -> names-only -> legacy bare
count; bridge_tool_schemas(listing=...) embeds it and instructs the
model to skip tool_search when the exact name is visible (one fewer
round-trip per use)
- config: tools.tool_search.listing auto|on|off (default auto),
listing_max_tokens (default 4000, clamped 200..20000); legacy bool
shapes keep working
- tests: 8 new tests (config parsing/clamps, short-desc clipping,
deterministic rendering, budget fallbacks, bridge embedding, assembly
on/off paths); full file green (47 passed)
- docs: tool-search.md config table + rationale
- scripts/tool_search_livetest2.py: benchmark harness v2 with real
per-call token accounting (normalize_usage spy) and a third 'listing'
mode for A/B/C comparison
- require positive agent ownership proof before any background-review mutation
- fail closed when usage record is missing, malformed, or unreadable
- preserve foreground user-directed mutations and valid agent-owned curator flows
- cover patch, edit, delete, write_file, and remove_file end to end
Closes#67073
The background write guard decided ownership from `isinstance(usage_rec, dict)`,
so a local skill with NO usage record passed. That successful write called
bump_patch(), which created a `created_by: null` record — and the identical
write was refused from then on. "Allowed exactly once, then never" is a race
with our own bookkeeping, not a policy. Reproduced on main: patch #1 succeeds,
patch #2 with the same arguments is refused.
Option B from the issue. Option A (split `session_review` from
`scheduled_curator` and let the session fork patch user-owned skills it
consulted) would widen autonomous write permission onto skills the user owns
with no user present to consent — wrong direction for a no-user-present actor.
- skill_manager_tool: missing and explicit-null records now resolve
IDENTICALLY, both fail closed. The refusal names the reason and points at
`hermes curator adopt <name>`.
- background_review: both review prompts told the reviewer to patch any skill
consulted in the session and claimed pinned skills could be improved, while
enforcement refused both. Prompts now list pinned, external, and user-owned
skills as protected, and tell the reviewer to RECOMMEND adoption instead of
attempting a write that will be refused.
- skill_usage: document that `created_by` is a curator-management policy flag,
not a provenance claim, and add `is_curator_managed()` so call sites read as
the question they ask. Field name retained — it is on disk in every
`.usage.json` and renaming would strand those records.
- curator CLI: `hermes curator list-unmanaged` itemizes unmanaged skills with
the reason each is unmanaged (completes the #67139 spec).
Foreground writes are untouched: a user-directed edit to a user-owned skill
still works, including on pinned skills.
Sibling tests: 9 failures in test_skill_manager_tool.py were fixtures that
created record-less skills to exercise OTHER guards (consolidation-delete,
read-before-write) and relied on ownership falling through. Fixed at the
fixture, since the real curator only ever operates on managed sediment. One
test asserted the old "manually authored" wording; rewritten to assert the
behavior contract instead of the string.
Validation: 274 targeted tests + all 7 background-review files (60 tests) pass.
E2E on a temp HERMES_HOME (30 checks) covers the flip, foreground writes,
adoption unblocking, pin semantics, prompt/enforcement parity, and the new verb.
Each new test sabotage-verified: revert the fix, confirm it goes red.
Fixes#67140
`hermes curator status` reported only the skills it manages, staying silent
about curation-eligible skills it can never touch. On a 237-skill library that
meant 112 skills were invisible to every automatic transition with no signal
anywhere — the curator looked broken when it was working as designed.
A skill becomes curator-managed only when `created_by: agent` lands on its
usage record, and only the background review fork writes that marker. Two
populations therefore never qualify: records written before the marker existed
(no key at all, authorship unknowable) and every foreground
`skill_manage(create)` (unset by design — those skills belong to the user).
- skill_usage: `list_unmanaged_skill_names()` / `unmanaged_report()` enumerate
the blind spot, tagging each row with `has_provenance_key` so the two causes
are distinguishable. `adopt_skill()` writes the marker on user declaration
and refuses bundled, hub-installed, external, and protected built-ins.
Adoption never resets the inactivity clock.
- curator CLI: status prints an `unmanaged (no provenance marker)` block on
BOTH the managed and no-managed-skills paths; new `adopt` verb takes names or
`--all-unmanaged`, with `--dry-run` and a confirmation prompt on bulk.
- skills_sync: `_backfill_optional_provenance()` matched candidates by
repo-derived path only, so a skill installed at `mlops/chroma` that upstream
later moved to `mlops/vector-databases/chroma` was skipped forever and
`hermes skills repair-optional` could not fix it. Falls back to an
unambiguous name match, still gated on identical content, and records the
ACTUAL install path.
Provenance stays a declaration, never an inference: a high patch count proves
the agent MAINTAINS a skill, not that it authored one, since Hermes edits
user-written skills on the user's behalf routinely. An "looks agent-made"
heuristic would eventually archive hand-written work.
Validation: 347 targeted tests pass. Each new test verified via sabotage run
(revert the fix, confirm the test goes red) — one initially passed for the
wrong reason because the fixture pinned `prune_builtins` off, masking the guard
under test; fixed to force the shipped default on.
`sync_skills()` keys the bundled manifest by frontmatter name but computes
the destination from the bundled path. When upstream renames or
recategorizes a skill, the manifest key still matches while the new dest
does not exist yet, so the loop fell into its "in manifest but not on
disk" branch and misread the skill as user-deleted: the user's copy was
stranded at the old path forever and never received another update.
Three skills hit this in the July 2026 reorg (computer-use,
evaluating-llms-harness, serving-llms-vllm) — silently frozen at their
pre-rename content on every machine that ran `hermes update`.
Recovery only moves a stale copy when it is byte-identical to the origin
hash recorded the last time sync wrote it, which proves the directory is
ours rather than the user's work. User-modified copies are kept in place
with a warning, hub-installed paths are never touched, and a genuine
deletion (no copy anywhere on disk) is still respected.
- tools/skills_sync.py: add _recover_renamed_skill() plus the
_index_active_skills() / _read_hub_install_paths() indexes; call it
before classification and report moves via a new `relocated` key.
- hermes_cli/main.py: surface relocations in both `hermes update` skill
sync reporting sites.
- tests: 4 cases covering relocate, user-modified preservation,
hub-installed exemption, and genuine-deletion respect.
Follow-up for salvaged PR #69380. The snippet
'export -p | grep -vE ... > $tmp.$BASHPID || true' attaches the redirect to
the grep pipeline segment, so $BASHPID expands inside grep's pipeline
subshell — a DIFFERENT pid than the parent shell that expands the follow-up
'mv $tmp' operand. The dump landed in an orphaned temp file, mv failed
silently (2>/dev/null), and the shared snapshot never updated again:
exported env / venv activation stopped persisting between commands
(tests/tools/test_local_shell_init.py TestSnapshotEndToEnd caught it).
Wrap the pipeline in a brace group and attach the redirect to the group so
the expansion happens in the current shell, matching mv's expansion. Also
rename the shell-init probe var HERMES_SESSION_ENV_PROBE ->
HERMES_STICKY_ENV_PROBE: it matched the HERMES_SESSION_ prefix the salvaged
fix now intentionally strips from snapshots, and the snippet-shape test is
updated to pin the brace-group contract.
A single long-lived backend serves many sessions through one
_active_environments['default'] LocalEnvironment (the messaging gateway, TUI,
and desktop/web dashboard all collapse the terminal to 'default'). That
environment persists a bash session snapshot file and sources it before every
command. 'export -p' dumped the FIRST session's HERMES_SESSION_ID into the
snapshot, so every LATER session sourced that stale value and its
'echo $HERMES_SESSION_ID' reported a FOREIGN session's id — overriding the
correct per-command Popen env injected by _inject_session_context_env.
Confirmed on staging-fp: session A read its own id, session B (same backend)
read A's id. Reproduced locally with two threads sharing one LocalEnvironment;
the snapshot held 'declare -x HERMES_SESSION_ID=...' from the first session.
Fix: strip the per-session bridged vars (HERMES_SESSION_* / HERMES_UI_SESSION_ID
/ HERMES_CRON_AUTO_DELIVER_*, i.e. gateway.session_context._VAR_MAP prefixes)
from the snapshot at both dump sites in base.py. They are re-injected fresh on
every command, so a snapshot should carry only the user's own shell state, not
Hermes' per-turn session identity.
Complements the _set_session_context session_id fix: that ensures the ContextVar
carries the right id; this ensures the shared snapshot can't override it with a
neighbour's. Adds tests/tools/test_snapshot_session_id_leak.py (regex unit +
real two-session LocalEnvironment integration) and updates the export-shape
assertions in test_base_environment.py for the new 'export -p | grep' dump.
Asked to link to a session, the agent had no way to know the @session
reference syntax exists — every mention in the tool schema described
consuming a link the user dropped, never writing one — so it answered
with the title and timestamp as prose and the desktop had nothing to
render.
Every result now carries a ready-to-copy `link`, and the schema says to
write it inline instead of restating the title around it. The profile
segment is omitted when the active profile can't be named confidently;
a bare id still resolves.
Also skip linkifying a ref a model already wrapped in a markdown link,
which would otherwise rewrite into a nested link.
Follow-up to the cherry-picked #69063: egilewski's review found that the
_dir_has_any_entry(parent) guard treats ANY entry in the mount point's
parent as proof the volume is attached — but unmounting exposes the
UNDERLAY directory's own files (e.g. a .keep placeholder), so a populated
underlying mount-point dir still classified the project as an orphan and
deleted its ref/index/metadata. Reproduced on both main and the PR head.
Attachment evidence is now positive instead of circumstantial:
* _volume_evidence() records the parent directory's (st_dev, st_ino)
identity in the project's metadata while the workdir is observably
live (at _register_project/_touch_project time). A mount point
resolves to the mounted filesystem's root while attached and to the
underlay directory after detach — same path, different directory,
different identity.
* _workdir_is_observably_gone() now requires the parent visible at
prune time to match that recorded identity before the populated-parent
check can classify an orphan. A mismatch means a different directory
(the underlay) is showing through — a detached volume, not an
observed deletion.
* Metadata without a recorded identity (written by older versions) is
never orphan-classified — unsure never deletes; the retention/stale
rule still reclaims genuinely abandoned projects off last_touch.
* The frozen pre-v2 layout has no metadata channel for the identity, so
it keeps the structural checks only (require_parent_identity=False).
* A failed evidence probe on re-registration preserves the previously
recorded identity — stale evidence can only make pruning MORE
conservative.
Windows: st_dev/st_ino of 0 (filesystems without file IDs, some network
shares) is treated as "no evidence recorded", which falls into the
conservative never-orphan path. os.path.ismount and Path.stat are
cross-platform; no POSIX-only calls added.
tests/tools/test_checkpoint_manager.py: adds egilewski's exact
regression (checkpoint history for mnt/volume/project, detach exposes
mnt/volume/.keep, prune with orphan deletion enabled → NOT deleted;
fails on the bare cherry-pick, passes with this fix), plus
no-recorded-identity conservatism and probe-failure identity
preservation. His absent-parent/empty-parent/retention/genuine-deletion/
live-project controls all still pass.
Reported-by: egilewski (review on #69063)
Addresses @egilewski's review: the parent-directory check still deleted
checkpoint history for the most common unmount layout.
Detaching storage removes the parent outright in some layouts
(`/Volumes/Ext/proj` on macOS, `/media/<user>/<label>/proj`), which the first
commit handles. But in the classic static layout — `/mnt/volume/proj`, an
fstab entry, a container bind-mount — unmounting removes the contents and
leaves the mount point behind as an empty directory. `parent.is_dir()` is then
true, the project is absent, and the startup sweep deletes its ref, index and
metadata: exactly the case this PR set out to protect.
Reproduced against the real predicate before this commit:
mount root vanished (macOS) -> False ok
empty surviving mount point -> True <-- history deleted
really deleted (siblings) -> True ok
An empty parent carries no information: it looks identical whether the volume
was detached or the project was deleted. So require the parent to actually say
something — it holds some other entry (we observed a populated directory that
does not contain the project), or it is itself a live mount point (the volume
is attached right now and demonstrably does not hold the project).
The cost is that a project deleted out of an otherwise-empty parent is no
longer reclaimed by the orphan rule. It is not leaked: the retention rule
reads `last_touch` rather than probing the filesystem and still collects it,
so reclamation is deferred, not lost. That is the right direction for a
predicate whose false positive destroys a user's restore points unattended.
`_dir_has_any_entry` stops at the first entry via `os.scandir` instead of
materializing a listing, since a project root can hold a large tree.
tests/tools/test_checkpoint_manager.py: `test_surviving_empty_mountpoint_
keeps_its_checkpoints` pins the reviewed case, and `test_empty_parent_project_
is_still_reclaimed_by_retention` pins the deferral above so the safety valve
cannot silently regress into a leak. Both fail on the previous commit. The
real-orphan control now seeds a sibling so it exercises a populated parent
rather than the ambiguous empty one. 80 passed in the checkpoint suite; the 2
remaining failures (`TestGitEnvIsolation`, `TestClearFunctions`) fail
identically on clean main.
Orphan pruning decides a project is gone from a single probe:
if delete_orphans and (not workdir or not Path(workdir).exists()):
reason = "orphan"
then deletes its ref, index, and metadata — the project's entire checkpoint
history. `Path.exists()` is False for a deleted directory, but it is equally
False for one whose storage is not attached right now: an unplugged external
drive, a share behind a downed VPN, a bind-mount absent from this container,
an offline Windows mapped drive. The project is fine; only our view of it is.
This is not an opt-in maintenance command. `maybe_auto_prune_checkpoints`
runs unattended at startup from both `cli.py` and `gateway/run.py`, with
`delete_orphans=True` by default. So starting Hermes once while the drive is
unplugged silently destroys the restore points for every project on it — the
one thing checkpoints exist to provide, and there is nothing to restore from
afterwards.
Reproduced against the real store: a project registered under an unmounted
path and one on local disk, then a startup prune —
prune: {'scanned': 2, 'deleted_orphan': 1}
unreachable project index still on disk: False
The legacy pre-v2 branch has the same flaw plus a second one: a
`HERMES_WORKDIR` marker that exists but cannot be read leaves `workdir = None`,
which the same condition treats as an orphan. Failing to read a file is not
evidence that a project was deleted.
Require corroboration before deleting: the workdir's parent must be present,
so its absence is something we actually observed. A missing parent means the
volume is not there and we know nothing, so the entry is left alone — and an
unreadable marker never deletes at all. Genuinely abandoned projects are still
reclaimed, both by the unchanged orphan path (parent present, project gone)
and by the retention/stale rule, which runs off `last_touch` rather than a
filesystem probe.
tests/tools/test_checkpoint_manager.py: a project whose whole mount disappears
keeps its history; controls prove a genuinely deleted project is still pruned
and a live project is untouched. The data-loss test fails on main; both
controls pass there. 81 passed across the checkpoint suites (2 failures in
test_checkpoint_manager.py are pre-existing and fail identically on clean
main).
_read_hub_installed_names() reads ~/.hermes/skills/.hub/lock.json with a
strict utf-8 decode. Hub skill descriptions can carry Windows-1252
typographic bytes (em-dash 0x97, smart quotes, bullets) as single high
bytes; read_text(encoding="utf-8") then raises UnicodeDecodeError, which
is a ValueError sibling not caught by the function's
except (OSError, json.JSONDecodeError). It escapes and 500s the whole
/api/skills endpoint, blanking the desktop Skills panel.
Decode with errors="replace" so the offending byte degrades to U+FFFD
and the structurally valid JSON — and every other skill — stays readable.
Fixes#68053
Address P1 from PR review: cmd_prune()'s y/N preview reads
store_status() but the confirmed deletion re-scans both the v2 and
pre-v2 layouts from scratch. A workdir that goes missing while the
human is answering the prompt gets swept in as if it had been shown
and approved.
prune_checkpoints() now accepts orphan_allowlist — a set of v2 project
hashes and/or pre-v2 shadow repo paths. When set, only orphans whose
identity is in the set are deleted; anything newly orphaned since the
scan survives the run. cmd_prune() builds this set from the exact
projects it just displayed and passed confirmation for. --force still
passes None (no preview shown, so nothing to bind to).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
When a background terminal() command backgrounds its own long-lived
child (`node server.js &`, `sleep 300 &`), the grandchild inherits the
write end of the reader thread's stdout pipe. The direct bash child
exits promptly, but the pipe never reaches EOF while the grandchild
lives — so `_reader_loop`'s blocking `read1()` parked the thread
forever, `session.exited` never flipped on its own, and
`notify_on_complete` was silently lost. `_reconcile_local_exit`
(#17327) only runs lazily from poll()/wait(), so nothing autonomous
ever surfaced the exit; each occurrence also leaked a reader thread
and pipe fd for the grandchild's lifetime.
Fix: on POSIX, drain via select() with a short poll interval and stop
shortly after the direct child exits even if the pipe hasn't EOF'd —
the same pattern the foreground path uses in
tools/environments/base.py::_wait_for_process (#8340). Windows pipes
don't support select(), so the blocking path is kept there with the
existing lazy reconcile as the safety net; mocked/iterator stdout
streams (no usable fileno) also keep the historical path.
Fixes#68915
Follow-up hardening on top of the salvaged #69745 guard, addressing both
review findings:
- Drift detection no longer re-reads the file. _reload_target performs ONE
checked read and derives both the drift check and the entry parse from that
same raw snapshot (_detect_external_drift now takes the raw text). The old
second read swallowed OSError as 'no drift', so a read failure between the
two reads let replace/remove/apply_batch rewrite the file from a stale view,
discarding externally added entries.
- Invalid UTF-8 now counts as unreadable: the checked read catches
UnicodeDecodeError and mutations return the preservation refusal instead of
raising (or worse, rewriting bytes we can't round-trip).
- USER.md is covered by the same guard (shared _reload_target path) and now
pinned by an explicit test.
Tests: read-once structural invariant, invalid-UTF-8 refusal with
byte-identical file, user-store refusal.
`_read_file` degraded any read failure to `[]`, conflating "file exists but
couldn't be read" with "empty store". That is a silent, total data-loss bug on
the `add` path.
`add` re-reads the file under lock, appends the new entry, and rewrites the
WHOLE file from the parsed entries. It deliberately skips the drift guard
(#42874: "appending never clobbers existing content") — but that reasoning only
holds when the reload actually saw the file. When `read_text` raises
(an external editor momentarily holding the file on Windows, a permission
change, a filesystem/EINTR blip), `_read_file` returns `[]`, so `add` treats
the store as empty and rewrites the file down to just the new entry — every
prior memory gone — while returning `success: True`.
Reproduced with a transient read failure during `add`:
entries on disk before : 3 (dark-mode pref, deadline, deploy target)
add("A brand new fact") : success=True
entries on disk after : 1 ("A brand new fact") <-- the other 2 wiped
replace/remove/apply_batch were shielded only incidentally — an empty view
means `old_text` never matches, so they abort before writing — but they still
returned a misleading "no entry matched" instead of naming the real problem.
Fix: distinguish unreadable from empty. `_read_entries_checked` returns
`(entries, read_ok)`, with `read_ok=False` only when the file exists but can't
be read; absent/empty stays a clean `([], True)`. `_reload_target` returns a
`_READ_FAILED` sentinel in that case without touching in-memory state, and all
four mutation paths (add, replace, remove, apply_batch) refuse the write with a
clear "retry in a moment" error. This is the same posture as the drift guard
and the pairing/checkpoint fixes: never rewrite a file from a view that isn't
the real one. `_read_file` keeps its `[]`-on-error contract for the read-only
`load_from_disk` caller, which never persists.
tests/tools/test_memory_tool.py: new TestUnreadableFileDoesNotWipeMemory —
add/replace/remove/apply_batch all refuse and leave the file byte-identical on
a transient read failure, plus controls that an absent file is still a clean
empty store and the happy path is undisturbed. The four refusal tests fail on
main. Full suite: 90 passed, 1 pre-existing failure (`test_deduplication_on_load`,
a UnicodeDecodeError unrelated to this change, identical on clean main).
Three durable ledgers used `with _connect() as conn:` where the sqlite3
connection context manager commits/rolls back but never closes, leaking the
db/-wal/-shm file descriptors on every call. On a long-running gateway this
exhausts RLIMIT_NOFILE and fails unrelated components with
`[Errno 24] Too many open files`. Same bug class as the cron execution ledger
(#69567 / PR #69594), which the connection helpers here are modeled on.
Fix: route every ledger operation through a `_transaction()` context manager
that guarantees `conn.close()` on exit. `_connect()` keeps its
schema-on-connect contract (several tests call it directly) and now self-closes
if schema init fails.
Adds per-module regression tests asserting every opened connection is closed,
including the no-op-update and exception-mid-transaction paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On Windows, pipe I/O can deliver non-UTF-8 bytes at chunk boundaries,
causing `UnicodeDecodeError` when the MCP SDK's `TextReceiveStream`
uses `errors="strict"`. Set `encoding_error_handler="replace"` on
`StdioServerParameters` so undecodable bytes become U+FFFD instead
of crashing.
- Strip the salvaged commit's inline encoding kwargs where main had since
gained its own (process_registry, local env, cua doctor, gateway,
commands, gateway_windows — the latter keeps its locale-aware
_schtasks_encoding() from #38186)
- Revert encoding kwargs mistakenly applied to non-subprocess APIs
(exa get_contents, tempfile.mkstemp in webhook.py)
- Guard the ddgs worker Popen (new on main since #55339)
- Update two kwarg-snapshot test assertions for the new kwargs
