Follow-up to the cherry-picked #31856 fix. The contributor's guard defers
idle-TTL eviction until the session store reports the session expired, so the
expiry watcher can tear the agent down and fire MemoryProvider.on_session_end()
with the live transcript. Two gaps remained:
1. Memory-leak regression for mode='none' sessions. _is_session_expired()
returns False forever for the 'none' reset policy, so the naive guard would
never idle-evict those agents — reopening the unbounded-cache leak the idle
sweep (#11565) exists to relieve. Added SessionStore.is_session_finalizable()
(a public predicate: will the expiry watcher EVER finalize this session?) and
gate the deferral on it. mode='none' agents fall through to soft eviction as
before.
2. on_session_end still dropped on the LRU-cap path. Both cache-pressure paths
(_enforce_agent_cache_cap and _sweep_idle_cached_agents) soft-evict via
_release_evicted_agent_soft, which by design does NOT fire on_session_end.
If cache pressure evicts a finalizable-but-not-yet-expired agent before it
expires, the watcher later finds no cached agent and the hook is skipped.
Added _commit_memory_before_soft_evict(): at LRU eviction, if the session is
finalizable and not yet expired, commit end-of-session extraction via the
live agent's own (fully-scoped) memory manager using commit_memory_session()
— extraction WITHOUT provider teardown, so the eviction stays soft and a
resumed turn keeps working. Skipped for mode='none' (no missed boundary to
compensate) and expired sessions (the watcher tears those down directly).
This closes#11205 for ALL eviction paths and reset policies, not just the
idle-sweep + finite-policy case, while preserving the soft-eviction
resumability contract (never calls close() on a live session).
Tests: 5 new cases in test_agent_cache.py (mode='none' still reaped, LRU-cap
commits for finalizable / skips for none, real is_session_finalizable
predicate); all mutation-checked. Contributor's original 2 tests updated to
assert the finalizable path explicitly.
The idle-TTL sweep (_sweep_idle_cached_agents) was evicting agents
as soon as they passed _AGENT_CACHE_IDLE_TTL_SECS, even when the
session hadn't expired yet. In daily-reset mode the reset can fire
hours after the last user message — evicting the agent early means
the session-expiry watcher has no agent in cache to call
on_session_end() with, so memory providers miss the live transcript.
Now the sweep checks the session store before evicting: if the
session still exists and hasn't expired, the agent stays in cache
so the expiry watcher can tear it down properly later.
When the session store is unavailable or throws, falls back to the
original eviction behavior (safe default).
Fixes: #11205
Self-review (ruff+ty lint diff = 0 net-new; 2-agent deep review) surfaced one
Warning + comment-accuracy nits; no Critical:
- W1: the local-probe TTL cache memoized None (probe failure) for 30s, so a
probe that failed during a startup race would suppress a legit retry once
the server came up. Cache only positive results — still fully bounds the
hot-path probe rate (reachable servers cache their value) while an
unreachable one re-probes on the next call. Add a regression test asserting
a None result is NOT cached (retry re-probes); mutation-verified.
- Tighten the platform-guard comment: gateway/TUI/cron already construct with
quiet_mode=True (gated by `not agent.quiet_mode`), so the guard's active job
is CLI dedup vs show_banner, not "filling the gateway/TUI gap" as originally
worded.
Verified not-issues (per review): positive-value 30s cache does not break the
reconcile-after-restart freshness contract (restart = fresh process, empty
cache); cache key is collision-safe; platform guard is correct in both
directions (no runtime path leaves platform None on a non-CLI surface).
Tests: 149 passed. ruff clean; ty 0 net-new vs base.
Desktop/dashboard WebSocket connections drop during long agent operations
(delegate_task subagents, large model outputs) when the uvicorn event loop is
GIL-starved for minutes. Root cause: uvicorn's ws keepalive ping runs on the
SAME event loop as agent turns. A single synchronous GIL-holding call on a
worker thread (a regex/scrub over a large output, or a long subagent turn)
freezes the loop, so it cannot process the incoming pong within ws_ping_timeout
and uvicorn closes an otherwise-healthy connection (#53773: 'event loop stalled
226.3s'; #48445/#50005). Loosening the timeout only raises the threshold — a
multi-minute stall sails past any finite window.
The keepalive ping exists to detect half-open connections (reverse-proxy 524,
dropped tunnels), which cannot happen on loopback: there is no network or proxy
in the path, and a dead local client tears the socket down with a real FIN/RST
that starlette surfaces as WebSocketDisconnect regardless of the ping. So on
loopback the ping provides ~no liveness value while actively killing
recoverable stalls — disable it entirely (ws_ping_interval/timeout=None).
Non-loopback (public) binds sit behind a Cloudflare Tunnel where half-open IS a
real failure mode, so the ping stays at 20/20 to detect it.
Empirically verified (real uvicorn + websockets peer): with ws_ping=None the
server never closes a silent peer during an 8s window; with the pre-fix 2s/2s
window uvicorn closes it. A genuinely-dead client still fires the
WebSocketDisconnect reap path regardless of the ping.
Note: this fixes the local Desktop case (the OP's scenario). A remote Desktop
over an authenticated public dashboard route (McCalebTheSecond's comment) keeps
the ping and needs the deeper GIL-hotspot fix — tracked separately.
Closes#53773
Salvage review of #56431 surfaced one Critical + two Warning issues; fix
them on top of the contributor's cherry-picked commits:
1. Critical — duplicate non-agentic warning on the interactive CLI. The new
agent_init warning fires on every platform, but cli.py show_banner()
already warns on CLI (richer output + /model hint), so a CLI user saw the
warning twice per startup. Guard the agent_init emit to skip platform=="cli"
— it now fills exactly the gateway/TUI gap the PR intended, no duplication.
2. Warning — vLLM error-parse regex under-matched. The patterns required a
literal space before the number, so "max_model_len: 32768", "=32768",
"(32768)", and "... is 32768" all returned None. Broaden both patterns to
accept :/=/(/ 'is' delimiters. Add a parametrized test over all delimiter
variants.
3. Warning — per-call live probe latency on local endpoints. The new
reconcile-on-hit + pre-defaults step-7 probe made every local resolution
fire a synchronous network probe (banner + /model switch + compressor
update_model each within one startup). Add a 30s in-process TTL cache
keyed by (model, base_url) around _query_local_context_length so back-to-
back resolutions reuse one round-trip; not persisted to disk, so the
reconcile freshness contract (re-probe after restart) is preserved. Add an
autouse fixture clearing the cache between tests + TTL coverage.
Tests: 148 passed (was 138). ruff clean.
Replace the webhook delivery-close path's direct reach into private
SessionStore._entries (which also bypassed the store lock) with a public,
lock-held peek_session_id(session_key) accessor. Mirrors the existing
lookup_by_session_id inverse helper. Keeps a getattr fallback for older
stores / test doubles. Adds a unit test for the accessor.
Webhook deliveries created a unique one-shot session (delivery_id baked into
the session key at gateway/platforms/webhook.py:668) but the adapter fired
handle_message via asyncio.create_task WITHOUT ever ending the session
(webhook.py:713, pre-fix). Nothing else closes it: the gateway caches/expires
the agent per session_key but never calls end_session for the webhook path,
and _end_session_on_close teardown doesn't run for these fire-and-forget tasks.
SessionDB.prune_sessions (hermes_state.py:4965) only deletes rows WHERE
ended_at IS NOT NULL. So every webhook session stayed with ended_at NULL ->
unprunable -> unbounded state.db growth. This was the primary driver of the
SQLite lock-contention gateway outage.
Fix: wrap the delivery in _run_delivery_and_close, which awaits
handle_message and then (in finally, so failures still reap) calls
_end_webhook_session -> SessionDB.end_session(session_id, 'webhook_complete').
This mirrors how cron closes its session with 'cron_complete'
(cron/scheduler.py:3065). end_session is first-reason-wins and no-ops on an
already-ended row, so it never clobbers a compression/agent_close reason.
Adds tests/gateway/test_webhook_session_close.py asserting the invariant
(a completed webhook session has ended_at set + is prunable), including the
error-path case, against a real SessionStore + SessionDB.
Add regression tests for vLLM max_model_len error parsing, stale local
cache reconciliation, live probes over llama defaults, and the 64K minimum
guard on persistent cache writes.
(cherry picked from commit 1cb47ef437)
Refine the blank-line handling so a blank line only continues a list run
when the next non-blank line is another list item. This keeps a list ->
paragraph -> list sequence as three separate blocks and matches the
contiguous-list layout for mixed/nested lists (one rich_text block, split
into sub-lists by (indent, ordered)), rather than emitting a separate
block per item.
Adds regression tests for the mixed blank-separated layout and the
list->paragraph->list boundary.
When a Markdown ordered list has blank lines between items (common in
LLM-authored content), the list run loop breaks on each blank line.
Slack numbers each rich_text_list independently, so N items produce N
lists each starting at 1.
Skip blank lines inside the list run as soft separators instead of
breaking, so ordered items stay in one rich_text_list and Slack renders
the correct numbering.
Fixes#57076
setup.runtime_check and setup.status are polled by the Desktop frontend on
connect and periodically (use-status-snapshot → evaluateRuntimeReadiness), but
neither was in _LONG_HANDLERS — so dispatch() ran both inline on the WS reader
thread. Under GIL pressure from concurrent agent turns (terminal I/O, large
output, background-process completions) either can block for seconds:
- setup.runtime_check → resolve_runtime_provider() (config read, auth check,
may probe the provider endpoint)
- setup.status → _has_any_provider_configured() (provider config + credential
scan)
While either blocks the reader thread the WS read loop can't service later
requests; the frontend RPC timeout fires, the client drops the socket, and the
lost setup.runtime_check response reads as ready=false — a false "needs setup"
/ "Settings failed to load" even though the provider is configured.
Route both to the RPC pool (same precedent as #55545's session.list/pet.info/
process.list). The handlers are read-only and pool writes go through the
lock-guarded write_json, so there's no ordering or safety concern.
Test asserts all 5 frontend-polled RPCs are pool-routed.
Co-authored-by: izumi0uu <izumi0uu@gmail.com>
delete_profile stopped only the process named in gateway.pid, but a Desktop
app spawns a headless `serve`/`dashboard` backend per profile that holds the
profile's SQLite connection open and keeps writing sessions/WAL/sandbox files.
That backend is never in gateway.pid, so a CLI `hermes profile delete` run
while the Desktop app is up left it writing into the tree — rmtree's final
rmdir then failed with ENOTEMPTY (#47368 "Bug 2"), and pre-guard it also
resurrected the directory.
- _profile_bound_backend_pids(): find running Hermes backends bound to this
profile via a `--profile <name>` selector or a HERMES_HOME env resolving to
the profile dir. Tightly scoped — current-user only, backend subcommands
(serve/dashboard/gateway) only so an interactive chat is never killed, and
never this process or its ancestors.
- _stop_profile_backends(): terminate them (graceful, then force), best-effort
so it can never make delete worse.
- _rmtree_with_retry(): a few spaced retries absorb the ENOTEMPTY / Windows
file-lock race from a just-terminated writer's in-flight -wal/-shm/sandbox
writes instead of failing the whole delete on a race the next attempt wins.
Complements the recreation guard (deleted profiles no longer reappear) and the
Desktop teardown-before-delete flow; this is the CLI-side convergence fix for a
delete run while a Desktop-managed backend is live.
Part of #47368.
A Z.ai desktop user reported thinking reverting to medium after one turn,
burning ~200% of a week's credits in 4 days despite reasoning_effort: false
in config.yaml. Four compounding bugs:
- _session_info reported reasoning_effort "" for disabled reasoning,
indistinguishable from unset — the desktop adopted it after the first
turn, wiping its sticky "thinking off" pick so every later chat
reverted to the default effort.
- config.set key=reasoning always wrote agent.reasoning_effort to global
config.yaml, so every desktop model-menu selection (preset.effort ??
'medium') clobbered the user's configured value. Now session-scoped
like the messaging gateway's /reasoning, landing on
create_reasoning_override so lazily-built sessions keep it too.
- YAML `reasoning_effort: false`/`off`/`no` (boolean False) was coerced
to "" by every loader's `str(x or "")`, silently re-enabling thinking.
parse_reasoning_effort now treats False/"false"/"disabled" as
{"enabled": False}; loaders (tui gateway, gateway, cli, cron,
delegate) pass the raw value through. The desktop config reader also
crashed on the boolean (false.trim()), aborting voice/STT settings.
- The zai provider profile never sent thinking on the wire, and GLM-4.5+
defaults to thinking ON server-side — so disabling reasoning was a
silent no-op on direct Z.ai, the actual token burner. The profile now
emits extra_body.thinking {"type": "enabled"|"disabled"} for
thinking-capable GLM models, mirroring the DeepSeek profile.
Also: /new (session reset) now carries reasoning_config across the
rebuild like model_override; config.get reasoning prefers the session's
live value and maps a config False to "none"; Settings shows "Off"
instead of a blank select for hand-written false.
The terminal-refresh quarantine filtered in-memory entries on
source == "device_code" but built removed_ids from the deleted
"loopback_pkce" source name, so the revoked device-code entry was
never pruned from the persisted pool in auth.json. Also restores the
_print_loopback_ssh_hint test suite scoped to Spotify (the helper's
remaining caller) instead of deleting it wholesale.
Replace the loopback/PKCE-callback server and manual-paste fallback with
the RFC 8628 device-code flow as the only xAI Grok OAuth login path. The
flow works in headless/SSH/container sessions with no 127.0.0.1 listener,
shrinking the local attack surface.
- Poll the token endpoint with server-provided interval, honoring
slow_down and expires_in; store tokens with auth_mode
oauth_device_code.
- Adaptive proactive refresh skew for short-lived device-code JWTs;
rotated tokens sync back to auth.json, the global root store, and the
credential pool (no refresh-token replay).
- Clear source suppression on successful re-login (CLI + dashboard) and
drop the duplicate dashboard pool entry so exactly one seeded
device_code entry exists.
- Use the shared device_code source name for consistency with the
nous/codex device-code providers.
- Desktop: remove the loopback OAuth flow states and dead type variants;
pkce providers' sign-in URL selection is unchanged.
- Docs (EN + zh-Hans) rewritten for device-code login; drop the deleted
--manual-paste flag from documented commands.
MSYS_NO_PATHCONV is honored by Git for Windows bash only. _find_bash's
final shutil.which fallback can return MSYS2-proper or Cygwin bash,
which ignore it and honor MSYS2_ARG_CONV_EXCL instead. Set both so argv
path conversion stays disabled regardless of which bash flavor spawns.
Also subsumes the cmd /c mangling in #56147.
Per-session /model overrides (_session_model_overrides) were in-memory only,
so a gateway restart silently reverted every session to the global default
model. Persist the non-secret parts (model/provider/base_url ONLY — never
api_key) into the session entry in sessions.json and lazily rehydrate them
on first use after a restart, re-resolving credentials through the normal
runtime provider resolution.
- gateway/session.py: SessionEntry.model_override field with
sanitize_model_override() (allowlist: model/provider/base_url) applied on
both serialization and deserialization; SessionStore.set_model_override /
get_model_override accessors. reset_session() already creates a fresh entry,
so /new keeps its clear-on-reset semantics — a restart cannot resurrect an
override the user reset away.
- gateway/slash_commands.py: write-through at both /model set sites (text
command + picker) after storing the in-memory override.
- gateway/run.py: _rehydrate_session_model_override() called from
_resolve_session_agent_runtime(); in-memory state always wins, credentials
are re-resolved per provider (credential-less fallback on failure). Session
expiry finalization also drops the persisted override.
- tests/gateway/test_session_model_override_persistence.py: restart
round-trip, /new clearing, api_key-never-serialized (including tampered
sessions.json), rehydration + live-state precedence + credential-failure
degradation.
Salvaged from #3659 by @Git-on-my-level, narrowed to the restart-persistence
gap confirmed in triage.
Named providers / custom_providers entries in config.yaml now accept an
extra_headers dict scoped to that endpoint — for reverse proxies, API
gateways, and custom auth schemes (e.g. Cloudflare Access service tokens).
- hermes_cli/config.py: normalize extra_headers on provider entries
(_normalize_custom_provider_entry + providers-dict translation), add
get_custom_provider_extra_headers /
apply_custom_provider_extra_headers_to_client_kwargs helpers keyed on
base_url (case/trailing-slash insensitive, no substring bypass —
mirrors the TLS helpers)
- hermes_cli/runtime_provider.py: surface extra_headers in the resolved
runtime for named custom providers (providers dict, legacy
custom_providers list, and the credential-pool path)
- run_agent.py / agent/agent_init.py: merge per-provider extra_headers
onto the OpenAI client default_headers at construction and on every
_apply_client_headers_for_base_url re-application (credential swaps,
rebuilds), most-specific level wins; OpenAI-wire only (native
Anthropic/Bedrock scoped out)
- agent/auxiliary_client.py: accept model.extra_headers as an alias of
model.default_headers for the global variant
- cli-config.yaml.example: documented commented example
- Header values are treated as secrets and never logged
Salvaged from PR #3526 by @jneeee, reimplemented against current main.
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
Adds a no-code routing layer to the OpenAI-compatible API server so one
Hermes deployment can map different API clients to different
model/provider backends. Clients pick a backend by sending a configured
alias as the OpenAI 'model' field; unmatched values fall back to the
global model. Configured aliases are listed by GET /v1/models.
Precedence (highest first): session /model override > model_routes
route > global config. Route provider credentials resolve through
_resolve_runtime_agent_kwargs_for_provider (same seam as
channel_overrides); per-route api_key/base_url are upstream provider
credential overrides — never caller auth, never logged.
Salvaged and rebased from PR #3176 by @Mibayy onto current main.
Salvaged from PR #3243 by @Mibayy, reimplemented against current main
(the original diff targeted a removed gateway/run.py handler).
- /compact is now a first-class alias of /compress (CLI, gateway,
Telegram/Slack/Discord command lists, autocomplete) — also fixes the
dangling '/compact' references in gateway error messages
(gateway/run.py context-exhausted banners).
- --preview / --dry-run: report what WOULD be compressed (message
counts, token estimate, 'here [N]' boundary) without touching the
transcript. Flags coexist with the existing 'here [N]' / focus-topic
args on both the CLI and gateway surfaces via shared pure helpers in
hermes_cli/partial_compress.py.
- --aggressive (LLM-free hard truncation) is intentionally NOT
implemented: it would need its own transcript-persistence branch
outside the guarded _compress_context rotation machinery (#44794
data-loss class). The flag is recognized and returns an explanatory
message pointing at '/compress here [N]' and /undo instead of being
mis-parsed as a focus topic.
- locales: gateway.compress.aggressive_unsupported added to all 16
catalogs (parity test enforced).
- release.py: AUTHOR_MAP entry for contributor credit.
Salvage of #3459 by @keslerm, reimplemented against the restructured
progress-callback block in gateway/run.py (resolve_display_setting,
needs_progress_queue, thinking-relay). Duplicate PR #3458 by @dlkakbs was
submitted 4 minutes earlier with the same feature — both credited.
Co-authored-by: Dilee <uzmpsk.dilekakbas@gmail.com>
tool_progress: log keeps the chat silent and appends timestamped tool-call
lines to ~/.hermes/logs/tool_calls.log via a dedicated queue drained by an
async writer (RotatingFileHandler 5MB x 3, RedactingFormatter so secrets
never land on disk). Gateway-only by design; thinking_progress relaying and
the webhook gate are unaffected. /verbose now cycles
off -> new -> all -> verbose -> log.
Salvage of the surviving hunk of #3296 by @Mibayy. The PR's gateway
_handle_provider_command hunk targets code removed on main (/provider was
absorbed into /model + /status, which already read model.base_url); the
hermes status mislabel was the remaining live symptom:
_effective_provider_label() only checked the legacy OPENAI_BASE_URL env var,
so a custom endpoint configured canonically in config.yaml still displayed
as OpenRouter.
Replace the plugin-local _IMAGE_MAGIC_MIME table + _sniff_image_mime
body with a delegation to agent.image_routing._sniff_mime_from_bytes,
the canonical magic-byte sniffer already used across the codebase, then
gate its result to the raster formats gpt-image-2's Responses
input_image actually accepts (png/jpeg/gif/webp).
The shared sniffer also recognizes SVG/TIFF/ICO; without the allowlist
those would pass local validation and be rejected server-side with an
opaque HTTP 400. Gating locally fails them cleanly as invalid_image_input.
Adds a regression test for SVG rejection.
Follow-up on top of @CrazyBoyM's #55828.
Follow-up to #56874, which added the Camofox private-page SSRF guard
(_camofox_current_page_private_url) but wired it only into the Camofox
eval path (_camofox_eval). The other Camofox content-read tools —
camofox_snapshot, camofox_get_images, and camofox_vision — still read the
current page's accessibility tree / images / screenshot without the
guard, so on a non-local Camofox backend they can return the content of
an intranet or cloud-metadata page (e.g. 169.254.169.254) that the
terminal itself can't reach.
Apply the same guard, gated on _eval_ssrf_guard_active (non-local
backend, not a local sidecar, allow_private_urls unset) and fail-open on
probe failure, matching the eval-path guard and the main-browser
snapshot/vision guards. camofox_back is intentionally not changed: its
target is unknown until navigation completes, and the subsequent content
read is already guarded.
Adds regression tests covering the three read tools blocking on a private
page, the public-page pass-through, and the guard-inactive no-probe path.
Three CLI reliability fixes:
1. Interrupt reliability: chat() only re-queued the user's interrupt
message when the turn result carried interrupted=True. When the agent
thread raced past its last interrupt check (or finished) before the
interrupt landed, the message was silently dropped — and the stale
_interrupt_requested flag left on the agent instantly aborted the
NEXT turn. Un-acknowledged interrupt messages are now re-queued as
the next turn and the stale flag is cleared (only when the agent
thread actually exited). The clarify-race path also parks the message
in _pending_input instead of dropping it.
2. Slow exit (5+ min): stdlib ThreadPoolExecutor workers are non-daemon
and joined unconditionally by concurrent.futures' atexit hook — even
after shutdown(wait=False). One wedged tool worker (abandoned after
interrupt/timeout) held the process open forever. Promoted
async_delegation's daemon executor to a shared tools/daemon_pool
module and adopted it in tool_executor (concurrent tool batches),
memory_manager (background sync), delegate_tool (child timeout wrapper
+ batch fan-out), and skills_hub (source fan-out). Added a 30s exit
watchdog (HERMES_EXIT_WATCHDOG_S) armed at _run_cleanup start as a
backstop for wedged cleanup steps.
3. Exit jank: after prompt_toolkit tears down the input/status bars the
terminal sat silent for the whole cleanup window, looking hung. Print
'Shutting down… (finalizing session)' immediately at exit start.
E2E: live PTY interrupt of a foreground 'sleep 120' terminal tool now
aborts in ~1s and the typed message runs as the next turn; wedged-worker
+ wedged-cleanup subprocess exits in 5.8s (watchdog) instead of hanging.
Salvage of the surviving piece of #2696 by @tarunravi. The PR's other two
changes (tool progress streaming, SSE None-sentinel fix) were independently
superseded on main by the structured hermes.tool.progress SSE events and the
rewritten queue-drain loop.
Remote OpenAI-compatible frontends can't read server-local file paths, so
MEDIA:<path> tags (browser screenshots, generated images) were dead text.
_resolve_media_to_data_urls() now inlines small (<=5MB) local images as
markdown data URLs across all four response surfaces: chat completions
(non-streaming), session chat, session chat stream final event, and the
Responses API. Non-image, missing, or oversized paths pass through
untouched.
Salvage of #2794 by @CharmingGroot, ported to the relocated
plugins/platforms/email/adapter.py:
- Guard raw_email = msg_data[0][1] against IndexError/TypeError and
non-bytes payloads. UIDs are added to _seen_uids before fetch, so an
exception mid-batch permanently skipped every remaining message in
the batch — now the bad message is logged and skipped instead.
- Message-ID domain generation falls back to 'localhost' when
EMAIL_ADDRESS lacks '@' (now via a shared _message_id_domain() helper
covering all 3 send paths; the PR fixed 2 of 3).
- ChannelOverride + channel_overrides on PlatformConfig
- Resolve model/runtime: session /model, then channel_overrides, then global
- Thread/parent channel lookup; bridge discord.channel_overrides from YAML
- Drop unrelated test and delegate_tool changes from PR scope
delegation.max_concurrent_children is now the single cap for both a
batch's parallelism and concurrent background delegation units.
- _get_max_async_children() delegates to _get_max_concurrent_children();
a leftover max_async_children key logs a one-time deprecation warning
- config v32→33 migration removes the stale key, folding a raised
max_async_children into max_concurrent_children (max wins, no lost
headroom)
- capacity error messages now point at max_concurrent_children
- pool-at-capacity sync fallback now attaches an explanatory note so
the model/user know why the call blocked instead of dispatching async
Previously users who raised max_concurrent_children (e.g. to 15) still
hit the invisible default-3 async cap: the 4th background delegate_task
silently ran inline, blocking the turn with no signal.
CLAUDE_CODE_OAUTH_TOKEN is set and owned by the user's Claude Code
install (subscription OAuth), not a Hermes-managed inference
credential — Claude subscription auth is not a working Hermes provider
path. Blocklisting it broke agent-spawned claude CLIs: with no token in
the child env, claude fell through to the shared macOS Keychain /
~/.claude/.credentials.json store and, on auth failure, cleared it —
logging the user out of their interactive Claude sessions and the
desktop app.
Exempt it from _HERMES_PROVIDER_ENV_BLOCKLIST (it arrives via the
anthropic registry entry, so discard explicitly with rationale).
ANTHROPIC_API_KEY / ANTHROPIC_TOKEN and every other provider credential
remain stripped, and the GHSA-rhgp-j443-p4rf fail-closed passthrough
guard is unchanged for everything still on the blocklist.
Fixes#55878
Follow-up on the salvaged #56392 guard. The cherry-picked change matched
custom:<name> pool entries against the primary by raw base_url string
equality, which (a) can't disambiguate two named custom providers sharing
one gateway base_url and (b) left a latent bare-"custom" entry bypass.
Route the match through get_custom_provider_pool_key(rt[base_url]) compared
against the entry's custom:<name> key, mirroring the sibling guard in
recover_with_credential_pool. Use CUSTOM_POOL_PREFIX instead of the literal.
Add regression tests for the custom same-endpoint (swap) and cross-endpoint
(skip) branches, plus the plain-provider fallback-pool case from #56885.
Two related hardening fixes for auxiliary calls (which include MoA reference
advisors — a pinned-model path where provider fallback is not a meaningful
recovery):
1. Transient-transport retries: the same-provider retry on a connection reset /
timeout / 5xx / 408 was a single attempt, then fallback. For a pinned aux
call a second blip silently loses the call (root of the run2 double-advisor
'Connection error' collapse — a genuine upstream blip). Now retries N times
with exponential backoff, N = auxiliary.transient_retries (default 2 -> 3
total attempts, clamped [0,6]). Compression-on-timeout fast-fail carve-out
preserved.
2. Per-model client-cache isolation: _client_cache_key excluded the model, so
two concurrent auxiliary calls to the same provider/base_url/key but
different models (e.g. an opus + gpt-5.5 MoA fan-out) shared one cache entry
and could race each other's client lifecycle. Model now participates in the
key -> distinct clients, no cross-call races. Same-model reuse unchanged.
- agent/auxiliary_client.py: _transient_retry_count() + backoff loop; model in
_client_cache_key and both call sites.
- hermes_cli/config.py: auxiliary.transient_retries default (2).
- tests: new retry/isolation tests; updated 2 stale-expectation tests to the
corrected behavior (per-model resolve; N-retry escalation).
Backoff base is overridable (_TRANSIENT_RETRY_BACKOFF_BASE) so tests don't sleep.
_get_platform_tools reverse-maps a platform composite to configurable
toolsets with an all-tools subset test. Because get_toolset() merges
registry-registered tools into a toolset, a tool added to a toolset
(delegate_cli -> delegation; desktop-only read_terminal -> terminal) that the
static composite never listed made the subset test fail, silently dropping the
entire toolset on api_server and other inference-based platforms. Compare the
toolset's static membership at all three reverse-map sites.
Fixes#49622.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds include_registry=True kwarg to resolve_toolset/get_toolset. When False,
returns only the static TOOLSETS view with no registry-merged tools — the
composite-authored membership platform reverse-mapping must compare against.
Default True preserves all existing behavior; this is the enabling half of
the api_server toolset-drop fix (#49622).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extends the browser private-network eval guard to the Camofox backend.
On main, _browser_eval() returned early in Camofox mode before running the
shared private-URL literal pre-scan and before re-checking the page URL
after eval, leaving Camofox as a sibling backend that could execute
browser_console(expression=...) against private/internal targets.
- move the eval private-URL literal pre-scan before the Camofox early return
- add a Camofox current-page private-URL probe via the evaluate endpoint
- withhold Camofox eval results when the page is now private/internal
Follow-up to browser private-network hardening in #56173, #56526, #56664.
Salvage of #56764 by @rayjun (rayoo), cherry-picked to preserve authorship.
The fake _SessionStore tracked peer_records but no test read it, leaving
#55300's peer-record behavior unasserted. Add a positive assertion on the
persist path and negative (== []) assertions on the two stale/moved-binding
skip paths, so the peer-record side effect is bound.
Mutation-verified: removing the production _record_gateway_session_peer call
makes the positive assertion fail.
Co-authored-by: João Vitor Cunha <jvsantos.cunha@gmail.com>
The #55300 peer-recording call now fires on the failed-turn compression
split path; the fake _SessionStore in test_compression_failure_session_sync
(carried in with #55721's test changes) lacked that method. Add a
call-tracking no-op so the combined salvage's tests pass.
Co-authored-by: João Vitor Cunha <jvsantos.cunha@gmail.com>
Manual /compress built a temporary AIAgent without the originating
platform / stable gateway session key, so an external context engine
ingested the retained transcript tail as source=cli during /compress
and again as the real platform on resume (duplicate cli,telegram rows).
Pass platform=_platform_config_key(source.platform) + the in-scope
gateway_session_key, mirroring the normal gateway turn. Assigned into
runtime_kwargs (single-valued, authoritative) so they neither collide
into a duplicate-kwarg TypeError nor lose to a stale resolver value.
Fixes#50422.
