switch_model() unconditionally set agent.provider but only set
agent.base_url when the resolved value was truthy. When a real
provider change resolved an empty base_url (e.g. minimax after
copilot), the agent ended up with provider="minimax" but
base_url still pointing at api.githubcopilot.com. That incoherent
pair then got snapshotted into agent._primary_runtime, so it kept
re-applying on every subsequent turn via restore_primary_runtime()
until the process restarted.
try_activate_fallback() and _swap_credential() were audited and
confirmed unaffected: both always derive base_url from an actually
constructed client, never from a possibly-empty resolver hint.
Fix: when base_url is empty AND the provider is genuinely changing,
raise ValueError instead of silently keeping the old provider's URL.
This routes through switch_model()'s existing snapshot/rollback
path, and callers (tui_gateway/server.py's _apply_model_switch)
already catch and surface a clean "switch failed, staying on X"
message. Re-selecting the SAME provider with an empty base_url
(credential-only refresh) still keeps the current URL, unchanged.
Fixes#47828
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Close the remaining end-to-end gaps so the full gpt-5.6 family (sol/
terra/luna + their -pro high-effort modes, 6 slugs) works on every
surface a user can reach them through:
- agent/auxiliary_client.py: the Codex OAuth backend hard-caps context
at 272K for gpt-5.6 exactly as it does for 5.4/5.5, but the default
50% compaction trigger would summarize at ~136K and waste half the
usable window. Extend the existing _is_codex_gpt54_or_gpt55 chokepoint
(single enforced predicate feeding _compression_threshold_for_model)
to match gpt-5.6* on the openai-codex route so those sessions get the
same 0.85 auto-raise. Direct-API/OpenRouter routes (full 1.05M window)
are unaffected; the historical codex_gpt55_autoraise opt-out still
applies. The one-time notice banner is model-dynamic and already
renders the correct slug/cap.
- hermes_cli/config.py, agent/agent_init.py: refresh the autoraise
comments/notice to mention the 5.6 family.
- hermes_cli/codex_models.py: add the -pro variants to DEFAULT_CODEX_MODELS
+ forward-compat so ChatGPT-OAuth (openai-codex) Pro users see the full
family in /model, not just the base tiers.
Supersedes the earlier commit's note that 5.6 was intentionally kept out
of the codex catalog: the slugs are confirmed routable (OpenRouter live
+ codex backend), so they belong there like every other codex-capable
gpt-5.x slug.
E2E verified across all 6 slugs: direct-API ctx 1.05M, codex ctx 272K,
pricing reachable from openai + openai-api routes, codex compaction
override 0.85 (and None on direct-API + when opted out), present in
openai-api picker + codex catalog, /model gpt resolves to sol on both
native routes. Guard tests added for the compaction route matrix.
PR #61587 adds sol-pro/terra-pro/luna-pro to the aggregator lists.
Complete those on the native surfaces the same way this PR completes
the base tiers:
- hermes_cli/models.py: -pro variants in _PROVIDER_MODELS[openai-api].
- agent/usage_pricing.py: alias ("openai", "gpt-5.6-*-pro") onto the
base-tier PricingEntry rows — the -pro high-effort modes bill at the
SAME per-token rates (verified against OpenRouter live pricing
2026-07-09: identical prompt/completion prices for base and -pro);
they cost more per task by consuming more tokens, not a higher rate.
- Context lengths need no new entries: "gpt-5.6-sol" et al. are
substrings of their -pro variants and both lookup tables match
longest-key-first (verified: sol-pro -> 1.05M direct / 272K codex).
- model_switch sort: -pro variants parse as suffix "sol-pro" (rank 1),
so /model gpt still defaults to base sol — pinned by test.
- Not added to DEFAULT_CODEX_MODELS: only confirmed routable via API/
OpenRouter so far; codex live discovery will surface them if ChatGPT
exposes them, same policy as other unconfirmed codex slugs.
Tests: invariant tests extended (pro aliases share base entries, base
sol outranks sol-pro); 191 targeted tests pass.
Phase-2 review findings addressed:
- resolve_billing_route: normalize the "openai-api" picker slug to the
"openai" billing provider — without this the ("openai", <model>)
_OFFICIAL_DOCS_PRICING keys (incl. every pre-existing gpt-4o/gpt-4.1
entry, not just 5.6) were unreachable when the provider is openai-api.
- pricing_version: drop the "preview" tag (GA 2026-07-09 at same rates).
- model_metadata comment: dict order is cosmetic — lookups length-sort
keys at match time; the old comment implied a positional invariant.
- model_switch comment: note "sol" is a series codename, not a generic
quality word.
- tests/hermes_cli/test_gpt56_registration.py: behavior contracts (no
list snapshots) — sol > terra/luna > 5.5 sort invariant, pricing
reachability from both openai and openai-api routes, cache-write
1.25x / cache-read 0.10x input relation.
PR #61578 added the GPT-5.6 series (sol/terra/luna) to the two aggregator
surfaces (OPENROUTER_MODELS, _PROVIDER_MODELS[nous]). This completes the
registration on the remaining surfaces per the standard add-model checklist:
- agent/model_metadata.py: DEFAULT_CONTEXT_LENGTHS 1.05M (direct API, same
as gpt-5.5; more-specific keys precede gpt-5.5 for longest-substring
matching) + _CODEX_OAUTH_CONTEXT_FALLBACK 272K for all three slugs.
Without these the direct-API fallback matched generic "gpt-5" = 400K.
- hermes_cli/codex_models.py: DEFAULT_CODEX_MODELS + forward-compat
templates so ChatGPT-OAuth (openai-codex) pickers surface the series.
- hermes_cli/models.py: _PROVIDER_MODELS[openai-api] (native API picker).
- agent/usage_pricing.py: _OFFICIAL_DOCS_PRICING snapshot — sol 5/30,
terra 2.50/15, luna 1/6 per 1M in/out; cache read 0.10x input, cache
write 1.25x input (OpenAI billing change starting with the 5.6 series).
GA 2026-07-09 at preview rates. Sol Fast mode (Cerebras tier) excluded.
- hermes_cli/model_switch.py: rank "sol" as a flagship suffix so
/model gpt resolves to gpt-5.6-sol, not alphabetical-first luna.
Verified: registry E2E via real imports (both context tables, codex
forward-compat from a gpt-5.5 template, billing-route lookup for
openai/gpt-5.6-sol -> 5.00/M), alias resolution on openai-codex and
openai-api resolves to gpt-5.6-sol; 183 targeted tests pass
(model_metadata, usage_pricing, codex_models, model_catalog).
DeepSeek V4 models (deepseek-v4-flash, deepseek-v4-pro) emit
reasoning_content in a separate delta field before final content,
requiring the same 600s stale timeout floor as R1. Without this,
streams hang for 30–50s with APITimeoutError on providers like
opencode-go while direct calls succeed in ~3s.
Fixes#60338.
Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)
Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by #57229 which
already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
behavior change reverting deliberate e8c3ac2f5; belongs in its own
PR with its own justification if pursued.
Conflict resolution: preserved main's newer timestamp-stripping
(#47868) inside the copy-on-write path.
Review finding: the substring check ('kimi' or 'moonshot' in model)
under-matches bare release slugs like k2-thinking that the repo's
canonical _model_name_is_kimi_family matcher (anthropic_adapter.py)
already covers. Reuse it instead of a second ad-hoc matcher; add a
regression test for the bare-slug case.
Kimi/Moonshot models on OpenRouter honour the same envelope-layout
cache_control markers as Claude on OpenRouter, but the policy fell
through to (False, False) — serving ~1% cache hits on 64K-token prompts
and re-billing the full prompt every turn. Observed within-turn
progression with cache enabled: 1% -> 67% -> 84% -> 97%.
(cherry picked from commit 3b857a35e, agent/agent_runtime_helpers.py hunk only;
the original PR #26014 branch also carried unrelated .dev-workflow artifacts
which are intentionally not included)
Structured review (2a/2b/2c) findings, all fixed:
- MAJOR: detect_local_server_type memo was process-lifetime with no
invalidation, permanently pinning a URL's server type. Now a bounded
1h TTL ((type, monotonic) tuples) so a backend swap on the same port
is re-detected. Test covers ollama->lm-studio swap after expiry.
- MAJOR: legacy disk-row compat was one-way. get_cached_context_length
and _invalidate_cached_context_length now consult the same key-shape
set {canonical, literal, canonical+slash} in both directions, so an
old slashed row is found (and cleared) when the runtime passes the
normalized URL. Tests pin both migration directions.
- MINOR: _localhost_to_ipv4 did whole-string replacement, which could
corrupt a proxy URL embedding http://localhost in its query. Now a
scheme-anchored host-only regex; localhost.example.com and embedded
substrings pass through. Tests added.
- MINOR: _invalidate_cached_context_length now also drops the
in-memory TTL probe rows for the pair, so a resolution inside the
TTL window can't re-persist the value just declared stale.
- Test gaps closed: detect-type cache hit + TTL-expiry re-detection,
ollama-show TTL expiry re-probe, reverse legacy-row lookups.
- Attribution gate: added zhchl@hermes-agent.local -> 8294 (PR #50572
author) to AUTHOR_MAP; the strict CI grep needs bare non-plus emails
literal in release.py.
Gates: ruff clean; targeted suites 202 passed / 0 failed; full
tests/agent 5426 passed with 17 failures identical on clean
upstream/main (pre-existing env-dependent anthropic/bedrock/credpool
tests); mypy delta vs base: 0 new errors; live smoke 6/6 PASS.
#37595 fixed the Windows dual-stack IPv6 timeout only inside
detect_local_server_type. The same 2s-per-probe penalty existed at every
other helper that builds a probe URL from base_url. Extract the rewrite
into _localhost_to_ipv4() and apply it at:
- query_ollama_num_ctx
- query_ollama_supports_vision
- _query_ollama_api_show (server_url derivation)
- _query_local_context_length (server root + LM Studio native URL)
Tests cover the helper's URL forms, non-localhost passthrough, and that
the ollama probes actually POST to 127.0.0.1.
Follow-up hunks completing the probe-cache cluster:
1. _query_ollama_api_show now goes through the existing
_LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) —
it was the one remaining per-resolution POST not covered by the
#56431-era wrapper. Failures are never memoized so a server that
comes up mid-startup is re-probed. Idea credit: #42081 (@Morad37),
reworked to comply with the positive-only rule.
2. Persistent context-cache keys are normalized through
_context_cache_key (trailing-slash strip) so http://host/v1 and
http://host/v1/ share one entry; reads and invalidation honor
legacy un-normalized rows. Idea credit: #37905 (@stevenau21).
Tests: TTL hit collapses to one POST, failure-not-memoized
(mutation-verified: unconditional caching makes it fail), namespace
no-collision vs the sibling probe, slash-variant dedup, legacy-row
read, dual-shape invalidation.
_cli's show_banner() calls _resolve_active_context_length() at every
startup. For non-OpenRouter providers (e.g. minimax-cn, kimi-coding,
custom endpoints) the resolver falls through to step 6 (OpenRouter
live /models fetch), which blocks ~2-3s per CLI launch and adds up
to 7+ minutes when openrouter.ai is unreachable through a proxy that
403s CONNECT (#46620).
Two complementary changes:
1. model_tools.py: read model.context_length from config.yaml and pass
it as config_context_length to get_model_context_length. The
step-0 config override short-circuits the entire resolution chain
including the OpenRouter fetch. No network call is made when the
user has set the value explicitly.
2. agent/model_metadata.py: replace flat timeout=10 with (5, 10)
tuple at all five sites (fetch_model_metadata + four endpoint
probes). urllib3 can otherwise block for 10s per retry stage
through proxies that 403 CONNECT. The tuple bounds connect at 5s
while still allowing slow reads.
Complements the in-flight PR #46685 (which adds HERMES_DISABLE_MODEL_METADATA
env var + same timeout tuple change for fetch_model_metadata). This PR
extends the timeout fix to the other four endpoint probes and adds the
config-override path that addresses the slow-but-reachable scenario
where env-var disable is too heavy-handed.
Refs #46620, PR #46685.
(cherry picked from commit e7faa34199)
On Windows, `localhost` resolves to both ::1 (IPv6) and 127.0.0.1 (IPv4).
httpx tries IPv6 first, hanging 2 sec per probe when the server binds IPv4
only. detect_local_server_type() is called 3+ times during init, each with
a new httpx.Client, compounding to ~14s of dead time.
Replace localhost with 127.0.0.1 inside the function before connecting.
The function is only called for local endpoints (callers guard with
is_local_endpoint()), so IPv6 loopback adds no diagnostic value.
Measured: 19.9s → 4.0s on Windows with a local proxy on 127.0.0.1:8317.
(cherry picked from commit a075d3194b)
Every 5 minutes fetch_endpoint_model_metadata() re-runs the full server-type
waterfall (LM Studio -> Ollama -> llama.cpp -> vLLM), spraying 404s at
endpoints the server never exposes (e.g. /api/v1/models and /api/tags on a
vllm backend).
Add _endpoint_probe_path_cache (base_url -> server type) so the first
successful probe's result is reused for the lifetime of the process.
Subsequent refreshes skip straight to the known-good path.
Fixes#29971.
(cherry picked from commit f3d7a8960a)
Follow-up to the salvaged str(tools) fix. The id()-keyed
_TOOLS_TOKENS_CACHE had no eviction, so a long-lived gateway/desktop
backend could accumulate an unbounded number of stale entries as it
builds transient tool lists. Cap it at 256 with oldest-first eviction
(insertion-ordered dict) and add a regression test asserting the cache
never exceeds the cap.
Estimate tool-schema size without repeatedly stringifying full tool lists, and cache the result per tool snapshot to reduce GIL-heavy work during preflight and compaction.
- Return the boundary snapshot from
_launch_session_boundary_memory_flush as a local value instead of
staging it on self._session_boundary_snapshot. The instance-attr
handoff could leak (no memory manager configured) or mis-fire a
stale snapshot on a later /new if an exception hit between staging
and consumption. A local variable eliminates the class; the helper
also returns None when no memory manager is configured so
new_session takes the inline-switch path.
- Drop the now-dead session_id kwarg from commit_memory_session:
after the redesign no production caller passes it (gateway, TUI,
compression all use the default), and speculative params are
rejected per AGENTS.md. The explicit-old-session need is served by
cli.py's direct engine call + commit_session_boundary_async.
- Drop the dead providers snapshot in commit_session_boundary_async
(only the emptiness check used it).
- Tests updated accordingly (dead-kwarg test removed, snapshot
assertion now covered by return-value contract).
Phase-2 gates: 2a tests/cli 1048 passed + 6 memory files 137 passed;
2b programmatic live smoke 0.38ms non-blocking caller, end→switch→sync
ordering verified; 2c structured 4-angle review — no Criticals, these
warnings fixed.
Deep review of the cherry-picked #16454 found the ad-hoc flush thread
raced new_session()'s inline on_session_switch(reset=True): memory
providers key off internal _session_id state (MemoryManager.on_session_end
takes no session id), so a late off-thread extraction ran against
post-rotation bindings — misattributing the old transcript to the new
session id, double-ingesting the old turn buffer (supermemory), or
double-committing (openviking already async-finalizes in
on_session_switch).
Redesign: new MemoryManager.commit_session_boundary_async queues
on_session_end + on_session_switch as ONE task on the manager's existing
single-worker background executor (the same worker sync_all already
uses). This preserves the strict end→switch ordering providers depend on,
serializes against per-turn syncs FIFO, keeps /new non-blocking, and
degrades to inline (pre-#16454 behavior) when the executor is
unavailable. No ad-hoc threads; no per-provider changes needed.
The context-engine on_session_end half stays synchronous in
_launch_session_boundary_memory_flush (cheap, must land before
reset_session_state rebinds the engine).
Exit durability: _run_cleanup calls the manager's existing
flush_pending(timeout=10) barrier before shutdown, so '/new then quit'
doesn't drop the queued extraction (shutdown_all's own drain is ~5s and
cancels queued tasks). Bounded well inside the 30s exit watchdog.
Tests: ordering invariant with slow (LLM-like) extraction, FIFO
serialization vs sync_all, switch-fires-even-if-end-raises, no-provider
no-op, CLI snapshot handoff + inline-switch fallback, sync engine
boundary, cleanup flush_pending.
Fixes#3356
Build the skills snapshot manifest in one directory walk, avoid importing gateway session context during CLI prompt startup, and reuse direct platform-list matching for snapshot entries.
(cherry picked from commit 1a64c2ed04)
* feat(xai): add grok-4.5 (early access) to catalog, context lengths, and reasoning-effort allowlist
- hermes_cli/models.py: grok-4.5 in _XAI_CURATED_EXTRAS (callable but absent
from models.dev) and _XAI_STATIC_FALLBACK, so the /model picker and
validation surface it on both xai and xai-oauth.
- agent/model_metadata.py: context lengths grok-4.5 -> 500K (per model card)
and grok-build-latest -> 500K (alias); grok-4.5 added to
_GROK_EFFORT_CAPABLE_PREFIXES.
Verified live against api.x.ai /v1/responses (2026-07-08): effort
low/medium/high accepted (server default: high), "none" rejected,
function calling works, full agent turn with terminal tool succeeded.
* feat(xai): grok-4.5 GA — add aggregator catalog entries, refresh comments
grok-4.5 is now GA: models.dev lists it (500K context, effort
low/medium/high) and both OpenRouter and Nous serve x-ai/grok-4.5.
Add it to the OpenRouter fallback snapshot and the Nous static list,
and update the early-access comments.
* chore: regenerate model-catalog.json for x-ai/grok-4.5
Sessions on sub-512K-context models were spending most of their wall-clock
re-summarizing: the 50% trigger left too little post-compaction headroom
(the incompressible floor — system prompt, tool schemas, protected tail,
rolling summary — ate most of the reclaimed space), so compaction re-fired
every 1-2 turns. Three compounding defects fixed:
- Threshold floor: models with context windows below 512K now trigger at
>=75% of the window (raise-only — a higher configured value or per-model
autoraise like Codex gpt-5.5's 85% always wins). Re-derived on
update_model() in both directions.
- No max_tokens on the summary call: the summary budget is prompt guidance
only ("Target ~N tokens"). The wire cap truncated summaries mid-section
on the Anthropic Messages / NVIDIA NIM paths (thinking models burn the
cap on reasoning first), yielding truncated or thinking-only summaries
and compaction loops. Summary token ceiling lowered 12K -> 10K to keep
the guidance within the intended 1K-10K envelope.
- Reasoning traces excluded end-to-end: inline <think>/<reasoning> blocks
are now stripped from assistant content before serialization to the
summarizer, and from the summarizer's own output before the summary is
stored (previously a thinking summarizer model's trace was persisted in
_previous_summary and re-fed into every iterative update, compounding
bloat). Native reasoning fields were already excluded.
Verified E2E with real imports against a temp HERMES_HOME: threshold table
across 64K-1M windows, override interactions (user 0.85 wins, spark 0.70
raised, gpt-5.5 0.85 kept), full compress() round-trip with a thinking
summarizer, and wire-kwargs capture proving no max_tokens is sent.
- OPENROUTER_MODELS: remove openrouter/owl-alpha (free) and
tencent/hy3-preview{,:free}; add tencent/hy3 and tencent/hy3:free
- _PROVIDER_MODELS[nous]: tencent/hy3-preview -> tencent/hy3
- run_agent.py reasoning-prefix list: tencent/hy3-preview -> tencent/hy3
(prefix match still covers -preview if pinned)
- model_metadata: register hy3 context length (262144) alongside hy3-preview
- regenerate website/static/api/model-catalog.json
- update tokenhub curated-list tests to the new IDs
The tencent-tokenhub direct provider still serves hy3-preview and is
intentionally unchanged.
The desktop app's chat panel reuses tui_gateway as its backend, so every chat session was stamped platform="tui". That made the agent read terminal-specific platform guidance while running in the graphical desktop chat surface.
Resolve the misclassification at its source: tui_gateway now picks platform="desktop" when HERMES_DESKTOP=1 and HERMES_DESKTOP_TERMINAL is unset, and keeps platform="tui" for the embedded terminal pane and standalone TUI. Add a PLATFORM_HINTS["desktop"] entry describing the actual chat surface (full GFM markdown, MEDIA: intercept, inline images). Move the embedded-pane clarifier to the platform-hint resolution site so it appends only to the tui hint under HERMES_DESKTOP_TERMINAL=1. Delete the now-dead desktop-hint block from build_environment_hints() that competed with the platform hint.
Standalone TUI sessions produce byte-identical prompts as before; the new desktop hint and clarifier are assembled once per session in the stable tier, so prompt caching is preserved.
* feat(trace): upload sessions to HF Agent Trace Viewer
Salvage trace upload as a smaller CLI-first feature: deterministic Claude Code JSONL export, fail-closed redaction, lazy Hugging Face dependency, and no gateway slash-command wiring.
* chore(trace): drop external porting references from docstrings
Describe the trace-upload design in Hermes' own terms.
* feat(sessions): fold trace upload into 'sessions export --format trace'
Integrates the HF Agent Trace Viewer exporter (PR #36145) onto the
unified export surface instead of a separate 'hermes trace' subcommand:
- --format trace: Claude Code JSONL to stdout/file, or one
<id>.trace.jsonl per session for filtered bulk export; defaults to
the most recent session when no --session-id/filters given.
- --upload pushes to the user's private HF traces dataset (--public to
opt out of private); reads HF_TOKEN with guided setup when missing.
- traces are secret-redacted by default (force mode); --no-redact opts
out after review; redaction failure blocks export (fail closed).
- hermes_cli/trace.py + subcommands/trace.py removed; agent/trace_upload.py
is the single engine. Docs EN + zh-Hans; 4 new CLI tests.
Review findings on the salvaged shim: (a) OpenAI callers may pass stop as
a bare string but Converse's stopSequences requires a list — normalize;
(b) call_llm(stream=True) (MoA aggregator) can reach this client and the
shim silently returned a complete response — keep that behavior (the
streaming consumer's got-final-object path downgrades gracefully) but log
it, and log dropped tool_choice, instead of silently ignoring both.
+2 regression tests.
Follow-up to the salvage of #60217 by @xxxigm.
Auxiliary Bedrock resolution always used the Anthropic Bedrock SDK, which
only works for Claude foundation-model IDs. Non-Claude models such as
openai.gpt-oss-20b-1:0 now use a Bedrock Converse adapter, matching the
main agent's bedrock_converse transport.
Review findings on the salvaged #60332 breaker, fixed as follow-ups:
- restore_primary_runtime() now resets the streak (third provider-swap
path; without it a recovered primary was short-circuited before a
single attempt and could never be re-proven healthy except via /model).
- interruptible_api_call (non-streaming) now carries the same breaker
(guard at entry, bump on stale_call_kill, reset on success). Quiet-mode
/ subagent / headless sessions — the profile most like #58962's
unattended 494-failure session — take this path and had the identical
infinite stale-retry class.
- Partial-stream stub return now resets the streak (chunks were received,
provider demonstrably responsive).
- Consolidated the triple-duplicated counter arithmetic into shared
helpers (_stale_streak/_bump_stale_streak/_reset_stale_streak/
_check_stale_giveup) with one canonical comment block; error message
now says 'consecutive stale attempts' (the counter counts kills, not
turns — a single turn can produce several).
4 new tests (restore resets / no-op restore keeps latch / non-streaming
short-circuit / non-streaming success reset).
Follow-up for the salvaged #60332 circuit breaker. The breaker latches:
once the streak trips, interruptible_streaming_api_call raises before any
stream is attempted, so the on-success reset can never run again. The
error text tells the user to switch models and retry — but neither
switch_model() nor try_activate_fallback() cleared the streak, so a
freshly selected healthy provider kept short-circuiting forever (only
/new recovered), and the automatic fallback chain was wedged the same way.
Reset the streak at both swap sites (after a successful rebuild only;
rollback/exhaustion paths keep the latch). 4 tests.
A session wedged against an unresponsive OpenAI-compatible provider can hit the stale-stream detector on every turn and loop forever, burning the full 180s x retries each turn with no response. Issue #58962 reports 494 consecutive failures over 3+ days on a single session.
The streaming retry path already caps retries WITHIN a turn (HERMES_STREAM_RETRIES, default 2) but has no cross-turn cap. Once a session's conversation state makes every turn stale, it retries indefinitely across turns and never notifies the user.
Add a per-session consecutive-stale-stream counter on the agent:
- incremented on every stale-stream kill in the outer poll loop;
- reset to 0 only when a stream actually completes;
- when it reaches HERMES_STREAM_STALE_GIVEUP (default 5), the next turn aborts immediately with a clear, actionable RuntimeError instead of spending 180s x retries again.
This is distinct from the existing stale-stream work (local-provider hard ceiling #44938, backoff/parse-error #60031): those bound a single hung stream, while this bounds repeated cross-turn staleness and surfaces a user-visible error.
Adds tests/run_agent/test_stream_stale_circuit_breaker.py covering the short-circuit, the success-reset, and the increment.
Replace the hand-rolled ensurepip bootstrap (and five other one-off
pip-install code paths) with hermes_cli.tools_config._pip_install, which
prefers the bundled uv (fast, needs no pip in the venv), falls back to
python -m pip, and bootstraps pip via ensurepip only when missing.
Sites unified:
- hermes_cli/setup.py: _install_neutts_deps, _install_kittentts_deps,
modal SDK install, daytona SDK install
- hermes_cli/memory_setup.py: memory-plugin pip deps (previously dead-ended
when uv AND pip binaries were both absent)
- hermes_cli/dingtalk_auth.py: qrcode auto-install (previously invoked
'python -m uv' which is not how uv ships)
- agent/lsp/install.py: --target LSP server installs
- plugins/google_meet/cli.py, plugins/platforms/matrix/adapter.py,
plugins/platforms/google_chat/oauth.py, plugins/memory/honcho/cli.py
Tests updated to assert the ladder behavior (uv-first, pip fallback,
ensurepip bootstrap) instead of the removed bespoke branches.
--safe-mode promised to disable ALL customizations, but shell hooks
declared in config.yaml's hooks: block registered anyway —
register_from_config() runs independently of plugin discovery and
load_config() does not honor HERMES_IGNORE_USER_CONFIG. Gate it on
HERMES_SAFE_MODE at the single chokepoint so troubleshooting runs fire
zero user-configured code (plugins, MCP, and hooks).
Docs (en + zh) updated; positive + negative tests added.
The salvaged fix added a post-worker _interrupt_requested re-check to the
main OpenAI/Anthropic streaming poll loop. The Bedrock Converse poll loop
(interruptible_streaming_api_call, api_mode='bedrock_converse') has the same
bug class: its worker calls stream_converse_with_callbacks(on_interrupt_check=
...), which breaks out of the event loop on interrupt and returns a PARTIAL
response WITHOUT raising (bedrock_adapter.py). The worker sets result[
'response'] and exits with _interrupt_requested still True, so the in-loop
raise never fires and the poll loop returns the partial — silently swallowing
/stop on Bedrock exactly as it was on the paths the salvaged commit fixed.
Add the identical post-worker re-check before the Bedrock loop's return.
The non-streaming loop (interruptible_api_call) is structurally immune: its
worker's only early return fires off _request_cancelled, which is set by the
main loop immediately before it raises in-loop, so no swallow window exists.
Guard test flips _interrupt_requested True mid-stream (after the pre-flight
check) and asserts InterruptedError is raised; verified RED without the fix
(DID NOT RAISE) and GREEN with it.
Two deep bugs found through systematic analysis of the streaming API
call and fallback credential subsystems:
1. Interrupt signal loss (chat_completion_helpers.py):
When the worker thread exits before the main thread's poll loop
checks the interrupt flag (e.g. _call_anthropic() detects the flag
and returns None), the while loop exits normally and the
InterruptedError is never raised. /stop is silently swallowed.
Fix: re-check _interrupt_requested after the while loop exits.
2. Empty provider bypasses credential guard (agent_runtime_helpers.py):
recover_with_credential_pool() guards against cross-provider pool
swaps with 'if current_provider and pool_provider and current !=
pool_provider'. When agent.provider is '' (valid unset state from
agent_init.py:326), current_provider is falsy, the guard is skipped,
and the pool swaps credentials onto an agent with empty provider.
This is the root cause of the 'provider= model=' empty-string error.
Fix: only skip the guard when pool_provider is empty (unscoped pool),
not when agent provider is empty.
Review findings: (a) an absolute path outside trusted roots passes
through unchanged and gets rejected downstream by skill_view — add a
debug log at the pass-through so the cron 'skill not found' symptom is
diagnosable next time; (b) test_relative_path_unchanged patched
get_skills_dir although the relative branch early-returns before any
root lookup — drop the misleading patch.
The extracted normalize_skill_lookup_name() resolved trusted roots via
agent.skill_utils.get_skills_dir(), but skill_view() enforces
tools.skills_tool.SKILLS_DIR — a separate module attribute that callers
and 60+ existing tests patch directly. With the helper reading a
different symbol than the enforcer, any SKILLS_DIR patch (or future
divergence between the two resolvers) makes normalization disagree with
enforcement and absolute-path loads regress silently. Read SKILLS_DIR at
call time (deferred import, cycle-safe) with get_skills_dir() as the
fallback, and align the new tests to patch the enforced symbol.
Follow-up to the salvage of #59829 by @HexLab98.
Cron jobs may store absolute paths to skills under HERMES_HOME/skills or
external_dirs, but skill_view rejects absolute names for security. Extract
the slash-command normalization into agent.skill_utils and reuse it when
cron loads job skills.
Gateway user replay entries now carry a timestamp (read by the
stale-confirmation expiry check). The transports already sanitize it
(#47868), but handle_max_iterations hand-builds api_messages and calls
chat.completions.create() directly, bypassing the transport — a strict
provider would 400 on the foreign key. Mirror the transport's pop here,
alongside the existing tool_name/codex_* sanitization.
Deleting the matched user message breaks the strict role-alternation
invariant on the exact incident tail this fix targets — user(confirm) →
assistant('OK, restarting') becomes two consecutive assistant messages,
which strict providers reject and which the alternation-repair passes
upstream don't cover. Replace the message content with an explicit
'confirmation EXPIRED, re-confirm before any destructive action'
sentinel instead: the trigger text is still neutralized, the model gets
an affirmative instruction not to act, and the message sequence stays
valid. Adds an alternation-preservation regression test.
Follow-up to the salvage of #59640 by @knoal.
When a high-risk side effect (e.g. host restart via shutdown.exe) runs,
the user's plain-text confirmation phrase is persisted in the conversation
transcript. If the host restart killed the gateway process before the
assistant's tool result was written, the transcript tail ends on the
assistant's text response - and the dangerous confirmation text remains
in the user role.
On the next inbound message - possibly a casual 'are you there?' from
the user minutes later - the LLM sees the stale confirmation and may
interpret the new turn as a fresh re-confirmation, re-executing the
destructive action. This is the failure mode reported in #59607.
Fix:
- Add strip_stale_dangerous_confirmations() in agent/replay_cleanup.py
that removes user messages whose content matches a known dangerous
confirmation pattern AND whose timestamp is older than 60 seconds.
- Add is_dangerous_confirmation() helper with the matched patterns
(i18n-aware: covers 確認強制重開機 from the original incident).
- Wire the stripper into _build_gateway_agent_history() right after the
existing 75ed07ace strippers, so the strip chain is:
strip_interrupted_tool_tails -> strip_dangling_tool_call_tail ->
strip_stale_dangerous_confirmations.
- Update _build_replay_entry() to preserve the timestamp on user
messages (it was previously dropped), since the new stripper needs it.
Complements 75ed07ace (which strips the assistant side of the broken
tail) by handling the user side: a stale plain-text confirmation that
the assistant has not yet responded to in a way the resume logic
recognises.
Failing-test-first discipline: the bug-detection test
test_stale_confirmation_text_is_stripped_on_resume fails on unfixed
code (proves the test catches the bug) and passes after the fix.
Five additional safety tests confirm no regression on:
- fresh confirmations (within expiry) are preserved
- non-confirmation text is preserved
- non-matching histories are untouched
- dangerous-pattern detection works in all cases (case, i18n, None)
- direct unit test of the strip helper
Refs: #59607
* fix(auth): resolve Anthropic OAuth file per-profile + close port-binding platform gaps
Two focused pieces salvaged from PR #57563:
1. _HERMES_OAUTH_FILE was computed at module import time — frozen before
HERMES_HOME/profile overrides, so multiplexed profile turns read and
wrote the DEFAULT profile's .anthropic_oauth.json (OAuth path hijack).
Replaced with a lazy _get_hermes_oauth_file(); all web_server.py call
sites updated.
2. _PORT_BINDING_PLATFORM_VALUES was missing whatsapp_cloud and line —
both bind aiohttp TCP listeners, so a secondary multiplex profile
enabling them would collide with the primary's listener instead of
failing fast at startup.
Original work by @austinlaw076. The rest of #57563 was redundant on
main (adapter routing sweep superseded by #56854's salvage; cron secret
scope landed in fdab380a1; nested-config fallback in from_dict).
* chore(release): map austinlaw076 author email for PR #57563 salvage
* test(hermes_cli): patch _get_hermes_oauth_file instead of removed _HERMES_OAUTH_FILE constant
---------
Co-authored-by: Austin <austin@openvm067.space>
Co-authored-by: Ben <ben@nousresearch.com>
When skill_view loads a supporting file (references/, scripts/,
templates/) instead of the main SKILL.md, the CLI quiet-mode line and
the friendly tool labels now show 'name → file_path' so it's clear
which file was actually read.
Follow-up hardening on the cherry-picked pool-fallback fix. The original
_resolve_codex_usage_credentials wrapped BOTH resolve_codex_runtime_credentials()
and the separate _read_codex_tokens() account_id read in one broad
'except Exception: pass', which had three problems:
1. A transient refresh/network failure (non-AuthError) from the resolver was
silently swallowed and downgraded to pool.select(), which could report
/usage limits for a DIFFERENT pool account than the one actually running.
On main that error surfaced. This is a real behavior regression for the
multi-account/pool case.
2. If the resolver succeeded but only the account_id read raised, the whole
singleton tier was abandoned in favor of a pool token that carries no
ChatGPT-Account-Id header (PooledCredential has no account_id concept),
risking a wrong-account read or 401.
3. 'except Exception' masked genuine programming errors.
Fix: narrow the outer catch to AuthError (the documented 'no creds' failure
mode of both functions), and read account_id in a best-effort inner try so a
partial/missing singleton store can't sink an otherwise-usable credential.
Transient errors now propagate and fail open via the outer fetch_account_usage
guard rather than mis-routing to the wrong account. Adds debug breadcrumbs and
a comment characterizing when the tier-3 pool path actually fires.
Guard tests: a non-AuthError resolver failure must NOT swap to the pool
(fail-open, no snapshot); an account_id read failure keeps the singleton token.
Updated the existing pool-fallback test to use AuthError (the real failure
mode) instead of a generic RuntimeError.
Follow-up on the salvage of #59523. Two low-risk cleanups surfaced by review:
- Extract _ZAI_CODING_OVERLOAD_SHORT_ATTEMPTS as a module constant so
adaptive_rate_limit_backoff() and zai_coding_overload_retry_ceiling()
share one source of truth. Previously both hardcoded short_attempts=3
independently; tuning one without the other would silently desync the
retry ceiling from the backoff schedule.
- Replace the tautological formula-mirroring assert in
test_zai_overload_retry_ceiling_exceeds_short_attempts with a behavior
invariant (ceiling leaves headroom for every long-backoff entry), per the
repo's contracts-over-snapshots testing rule.
