The cross-turn stale-call circuit breaker (_check_stale_giveup in
agent/chat_completion_helpers.py) raises a RuntimeError when the
provider has been unresponsive for N consecutive stale attempts.
This error was classified as FailoverReason.unknown (retryable=True,
should_fallback=False), causing the retry loop to burn all max_retries
against the same dead provider — each retry hitting the circuit breaker
instantly with zero network overhead — before fallback was attempted.
Add a classification rule (section 7b in classify_api_error) that
recognizes the stale-breaker RuntimeError by its signature phrases and
classifies it as FailoverReason.timeout with retryable=False,
should_fallback=True. This makes the retry loop skip retries and go
straight to fallback provider activation on the first hit.
Test: test_stale_breaker_runtime_error_triggers_fallback_not_retry
The #54878 self-heal (SessionStore.get_or_create_session) drops a routing
key pointing at a session already ended in state.db and recovers/recreates
a fresh session_id under the same session_key. The #54947 fix (agent-cache
cache-hit guard in gateway/run.py) treats a cached agent whose snapshot
session_id differs from the current session_id, under the same
session_key, as an intentional /resume-/branch-style switch between two
live sibling conversations, and reuses it unchanged to protect the prompt
cache.
These two fixes compose incorrectly: when the #54878 self-heal just fired,
the cached agent's session_id is not a live sibling — it's the dead session
just routed away from. #54947's "different session_id -> reuse freely" rule
reuses it anyway. The stale agent runs the turn, and the post-run "session
split" sync (agent.session_id != session_id) then writes the routing key
straight back onto the dead session_id, undoing the self-heal. This repeats
on every subsequent message until an interrupt (e.g. /stop) happens to race
in before that post-run sync, silently discarding conversation context.
Reproduced live on the engineering gateway (2026-07-12, routing key
agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines
over ~40 minutes, each followed by the dead session_id being reused and
re-synced back, until an interrupted /stop finally let a fresh session
stick — at which point all prior context was gone.
No open upstream issue tracks this specific interaction as of 2026-07-12
(checked #54878, #54947, #59580, #59597, #61220 — all cover adjacent but
distinct edges of the self-heal / agent-cache system).
Fix: before applying #54947's reuse-on-mismatch rule, check (outside the
cache lock, via SessionStore._is_session_ended_in_db) whether the cached
snapshot's session_id is itself ended in state.db. If so, treat it as a
stale self-heal artifact and evict/rebuild fresh -- same as a genuine
cross-process write -- instead of reusing it. Re-validates the peeked
verdict against the tuple actually held under the lock so a race can't
apply a stale verdict to a different (possibly live) cache entry.
Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new
cases: dead-session eviction, live-sibling reuse preserved [#54947 intact],
cross-process invalidation preserved [#45966 intact], same-session_id dead
edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed,
9040 passed, 11 skipped -- all 14 failures verified pre-existing on
unpatched main (confirmed via git stash + re-run), unrelated to this
change.
GatewayRunner._prepare_inbound_message_text() interpolated
source.user_name — the platform-supplied, user-settable display name —
directly into the message text of every turn in a shared multi-user
session: f"[{source.user_name}] {message_text}". Shared sessions are the
default for any threaded conversation (thread_sessions_per_user defaults
to False) and apply to any group with group_sessions_per_user=False, so
no special configuration is needed to reach this path.
An unescaped display name containing embedded newlines could therefore
masquerade as a new markdown section (a fake "## Override" heading)
inside the live conversation the model reads on every turn — the same
indirect-prompt-injection vector gateway/session.py's
build_session_context_prompt() already guards against for the identical
user_name field via _format_untrusted_prompt_value(), which was never
applied to this sibling call site.
Add neutralize_untrusted_inline_text() alongside the existing helper in
gateway/session.py: it collapses embedded newlines/control characters to
a single inert line without JSON-quoting, so inline "[Name] message"
formatting is preserved byte-for-byte for the common case (unlike
reusing _format_untrusted_prompt_value directly, which would add visible
quote marks to every sender prefix). Wire it into the sender-prefix
construction in gateway/run.py.
Bug 2 of #28156: the picker offered us./global. inference profiles to
EU-region endpoints (unroutable — AWS rejects them regardless of
credentials) and _RECOMMENDED hardcoded us.anthropic.* ids, so non-US
pickers pinned profiles their endpoint can't invoke.
- bedrock_model_routable_from_region(): geo-prefixed profiles are only
offered in their own geography (full AWS prefix set incl. apac./jp./
ca./sa./me./af.); bare ids and global.* pass everywhere; unknown
region shapes hide nothing.
- Recommendations match geo-agnostically on the base model id, so an EU
picker pins eu.anthropic.claude-sonnet-4-6; in-region geo profiles
sort above global.* for the same model (addresses the global.*-first
ordering complaint from the issue thread).
- Dedup generalized from (us., global.) to all profile prefixes.
Rewrites the cherry-picked test to import os locally and monkeypatch the
resolver seams instead of patching bedrock_adapter internals; adds the
inverse assertion (no bearer -> AnthropicBedrock SDK path preserved).
Three fixes for the Bedrock Claude path:
1. Streaming fallback: When AnthropicBedrock SDK raises 'Unexpected event
order' (SDK misparses Bedrock error events as message_start), auto-switch
to native Converse API for the rest of the session instead of failing
after 3 retries.
2. Image base64 decode (#33317): data URL payloads were passed as base64
strings to source.bytes, but boto3 re-encodes at the wire layer. Now
decoded to raw bytes before passing to Converse API.
3. Bearer token routing (#28156): Users with AWS_BEARER_TOKEN_BEDROCK are
now routed through Converse API regardless of model, since the
AnthropicBedrock SDK only supports SigV4 signing.
3 new tests. 121 bedrock_adapter tests passing.
Follow-up hardening on top of #64158 (@DavidMetcalfe):
Backend (the root-cause fix):
- hermes_cli/moa_config.py: add validate_moa_payload() — strict write-time
counterpart to the deliberately tolerant normalize_moa_config(). Flags
half-filled slots, empty reference lists, recursive moa slots, naming the
exact preset/slot.
- hermes_cli/web_server.py: PUT /api/model/moa validates before normalizing
and returns 422 with the specific problems instead of silently swapping the
user's preset for hardcoded defaults (#64156). Also declares
fanout / reference_max_tokens / reasoning_effort on the Pydantic payload so
client round-trips no longer erase hand-set values.
Desktop:
- Replace sanitize-then-send with hold-while-incomplete: the debounced
autosave is deferred (not repaired) while any slot is half-filled, and
flushes once the model pick completes the edit. Mid-edit UI state is never
repainted by a save response (generation guard covers held edits too).
- updateMoaSlot only clears the model when the provider actually changed.
- Explicit preset ops (set default / add / delete) cancel the pending
autosave and invalidate in-flight responses so the two writers can't race.
- Stable row keys (preset+index) so mid-edit rows don't remount; cleared
model shows the 'Model' placeholder instead of vanishing.
Both TS clients' MoaConfigResponse types now declare the round-tripped
fields (fanout, reference_max_tokens, reasoning_effort).
Tests: 12 new backend unit tests (validate_moa_payload contract incl.
validate/normalize agreement), 3 new web_server endpoint tests (422 on
half-filled ref/aggregator, fanout round-trip), 3 new desktop vitest cases
(autosave held while half-filled, flush on completion, same-provider
reselect no-op). E2E validated against a live TestClient with isolated
HERMES_HOME: bug sequence now 422s with config untouched.
Fixes#64156
Follow-ups on the salvaged #20096 profile-routing feature:
- _profile_name_for_source now returns None unless gateway.multiplex_profiles
is on. Routing stamps source.profile, which namespaces session/batch keys,
but the profile-scoped agent run only activates under multiplexing — without
the gate, configured routes with multiplexing off split batch/session keys
into agent:<profile> while the agent still ran from agent:main.
- Widen the profile-aware _text_batch_key fix from Discord to every adapter
that builds batch keys via build_session_key (telegram, whatsapp, matrix,
feishu, wecom, weixin) — routing is platform-generic, so the batch-key
namespace fix must be too.
- Downgrade the no-route-matched log from INFO to DEBUG (fired on every
unrouted inbound message).
- GatewayConfig.to_dict(): serialize profile_routes as plain dicts
(ProfileRoute dataclasses are not JSON-safe).
- Docs: correct the 'independent of multiplexing' claim in
docs/profile-routing.md (routing requires multiplexing), fix the
platform-only specificity row (0, not 1), and document profile_routes in
website/docs/user-guide/multi-profile-gateways.md.
- Tests: pin the multiplex gate (routes ignored when off, active when on,
build_source end-to-end stays in agent:main when off).
Completes the review's ask for "adapter-to-session-key integration coverage
for Discord and a non-Discord platform" on #20096.
Drives a concrete adapter's real BasePlatformAdapter.build_source with an
injected gateway_runner, asserts the matched route's profile is stamped on
the source, and that build_session_key scopes the key under agent:<profile>:
(versus the shared agent:main: namespace). Covers Discord and Telegram — the
Telegram case is the bug-#2 path that previously fell through to default.
Adds a regression anchor: without gateway_runner, profile stays None and the
key lands in agent:main (the silent fallback the fix removes for non-Discord).
Co-Authored-By: Claude <noreply@anthropic.com>
Addresses hermes-sweeper review on #20096.
Problem 1 (profile_routing.py): route matching returned True on a chat_id
hit before the guild_id constraint was consulted, so a route declaring both
guild_id and chat_id matched on chat_id alone. Restored conjunctive (AND)
semantics — every declared discriminator must hold; hierarchical parent_chat_id
matching is preserved. Added a regression test for the guild+chat case.
Problem 2 (base.py / run.py): gateway_runner was injected only when an adapter
pre-declared the attribute, and only Discord did — so build_source never called
_profile_name_for_source for Telegram/Feishu/Slack/etc., despite the
platform-generic claim. Declared gateway_runner on BasePlatformAdapter and made
the plugin-registry injection unconditional, so profile routing now reaches
every platform. Added non-Discord (Telegram) resolution coverage and an
injection-inheritance test.
Also adds docs/profile-routing.md documenting gateway.profile_routes
(matching rules, specificity, profile isolation) — requested in review.
Co-Authored-By: Claude <noreply@anthropic.com>
load_gateway_config only forwarded the top-level multiplex_profiles
key, ignoring the nested gateway.multiplex_profiles form. The latter
is what `hermes config set gateway.multiplex_profiles true` writes,
so users who ran that command got multiplex_profiles=False silently
— no warning, no fallback, profile_routes just stopped matching.
Loader now checks the top-level key first, falls back to the nested
gateway section, and only then defaults to False. Same precedence is
applied to other nested-form keys (profile_routes already did this).
Tests cover: top-level honored, nested honored (regression test for
the silent-fallback bug), default False, top-level overrides nested.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_adapter_credential_fingerprint only looked at adapter.token directly,
but Discord (and similar) adapters store the bot token on their config
sub-object, not on self. Every Discord adapter in a multiplexed
gateway therefore returned None, the same-token conflict check was
silently skipped, and N adapters all polled the same bot token —
producing a per-message race where whichever adapter won the GIL
answered the user.
Adds a config-token fallback (token, then bot_token) so the check
actually fires for config-backed adapters. Direct adapter.token
still takes precedence when both exist.
Tests cover: config-backed token produces a fingerprint, distinct
tokens produce distinct fingerprints, direct token wins over config,
config without token attributes returns None.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three follow-ups to the initial routing PR after code review:
1. Remove dead forum-post hierarchy cache. The `_forum_post_cache`,
`register_forum_post()`, and `resolve_forum_channel()` were never
wired up — no caller in the codebase. Discord's adapter already
sets `parent_chat_id` to the immediate parent (forum channel for a
forum post), so the existing `self.chat_id == parent_chat_id`
branch in `matches()` handles forum posts correctly without a
cache. The hierarchical-resolution branch in `matches()` and the
bounded-LRU infrastructure are removed.
2. Fix docstring specificity numbers (8 → 14, 4 → 6) and rewrite the
"Hierarchical matching" section to describe the actual one-level
parent_chat_id behavior. Removed unused `OrderedDict` and `Set`
imports.
3. Warn loudly when a routed profile doesn't exist on disk. Previously,
a typo in `profile_routes` (e.g. `crypto-tradr`) silently fell back
to the global HERMES_HOME, causing the message to read the default
profile's memory/credentials with no signal to the operator. Now
emits a `logger.warning` with the profile name, source identifier,
and the fallback reason. Bare-exception path also gets `exc_info`.
Tests:
- test_profile_routing.py: +2 tests verifying forum post matching via
direct parent_chat_id (covers the case the removed cache was meant
for). 31 total, all pass.
- test_profile_resolution.py: NEW, 12 tests covering resolution order
(source.profile > routing > active > default), missing-profile
warning, exception handling, and routing consultation. All pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds gateway.profile_routes config that routes specific Discord
guilds/channels/threads (and other platforms) to different profiles.
The routing engine uses hierarchical specificity matching
(thread > channel > guild) with bounded LRU caching for forum post
resolution.
Routing result is stamped on source.profile by BasePlatformAdapter
.build_source() at inbound time. When gateway.multiplex_profiles is on,
the existing _profile_runtime_scope machinery picks up source.profile
and runs the whole turn inside the profile's HERMES_HOME — so memory,
skills, config, and secrets all resolve to that profile automatically.
No new isolation code is added; this PR only adds the routing decision
layer on top of the existing multiplexing infrastructure.
Configuration:
gateway:
multiplex_profiles: true
profile_routes:
- name: server-default
platform: discord
guild_id: "GUILD_ID"
profile: server-profile
- name: special-channel
platform: discord
guild_id: "GUILD_ID"
chat_id: "CHANNEL_ID"
profile: channel-profile
When multiplex_profiles is off, profile_routes is ignored (no behavior
change for single-profile gateways).
Tests: 29 unit tests covering specificity scoring, hierarchical
matching, path-traversal validation, and config parsing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two consecutive incomplete assistant interims with identical visible
content (content + reasoning) are collapsed even when opaque provider
state (encrypted reasoning item ids, message item phases) drifts per
continuation — previously that drift defeated dedup and caused message
storms (#52711). The latest opaque payload is written onto the existing
message in place, so continuation replay still uses fresh provider
state.
Salvage note: the original PR also made the 3-retry continuation cap
cumulative per turn (no reset on progress). That half is intentionally
dropped — a legitimate long turn alternating incomplete/progress would
hard-fail at 3 cumulative, changing semantics beyond the reported bug.
Follow-up to the salvaged #51657: the conversation loop returns the
retry-exhaustion sentinel as BOTH final_response and error, so the
original detector (which required final_response to be falsy) never
fired on real exhaustion turns — the sentinel text was delivered
verbatim into the channel, exactly the #51628 poisoning vector. Detect
the sentinel echo, blank it before empty-response normalization, and
never suppress a turn whose final_response is genuine model text.
Also: dedupe-guard mock fix in the test fixture (has_platform_message_id
must return False, not a truthy MagicMock) and two guard tests
(real answer never suppressed; interrupted/failed never classified).
Map Codex Responses status=incomplete with incomplete_details.reason=content_filter to finish_reason=content_filter so the existing refusal/fallback path runs instead of burning incomplete continuation attempts.
Follow-up on @AlexFucuson9's cherry-picked multimodal flattening:
- http(s) image parts render as '[image: <url>]' so the summary keeps a
referenceable handle after compaction (base64 data: URLs still
collapse to '[image]' — no reusable reference, and leaking them is
the bug being fixed)
- unknown part types (document, future shapes) render as '[<type>]'
instead of being silently dropped
- test docstrings corrected: the original crash premise is stale
(redact_sensitive_text grew str() coercion in #52147); the live bug
is repr-noise + base64 leakage into the summarizer input
msg.get('content') can return a list of parts for multimodal messages
(containing text, images, etc.). The old code passed this list directly
to redact_sensitive_text(text: str), which raised AttributeError on
list.replace(), causing context compression to fail entirely for any
session with attached images.
Fix: detect list content and extract text parts before redacting.
Image parts are replaced with '[image]' placeholder.
Follow-up on the salvaged fallback: silent degradation is how #64333 went
unnoticed (jobs dead on arrival, only errors.log knew). Warn once with a
resync hint, and cover both the skewed and healthy paths with tests.
* fix(auxiliary): route direct-create aux callers through call_llm (#35566)
Five callers (kanban_decompose, kanban_specify, profile_describer, and
goals.py's judge + draft-contract) built raw clients via
get_text_auxiliary_client() and passed extra_body=get_auxiliary_extra_body()
— which only returns Nous portal tags and ignores
auxiliary.<task>.extra_body from config.yaml entirely. That was the
remaining half of #35566 after the call_llm path was fixed.
Routing them through call_llm(task=...) gives each caller the full
auxiliary contract for free: task extra_body, the reasoning_effort
shorthand, transient retries, provider-profile projection, and fallback
chains. goal_judge gains a DEFAULT_CONFIG block (it had none — its
provider/model overrides silently didn't exist as documented keys).
get_auxiliary_extra_body() now has zero non-test callers; kept for
plugin back-compat.
Fixes#35566.
* test: migrate kanban dashboard + CLI specify mocks to call_llm
Two more consumers of specify_task mocked the old
get_text_auxiliary_client symbol (missed in the first sibling sweep —
they live outside tests/hermes_cli's kanban files): the dashboard
plugin's /specify endpoint tests and the /kanban slash-command E2E.
Same migration as the rest: mock call_llm at the source, no-provider
now surfaces via the LLM-error branch.
Ollama's /v1/chat/completions silently ignores extra_body.think (it only
honours it on /api/chat — ollama/ollama#14820), so agent.reasoning_effort:
none never actually disabled thinking on OpenAI-compatible Ollama routes.
Emit the top-level reasoning_effort='none' field (which Ollama respects)
alongside think=False (kept for proxies and the native /api/chat path).
The PR's second half (propagating reasoning_config to the background-review
fork) already landed on main via agent/background_review.py, so only the
provider-profile change is salvaged here, resolved onto the current
GLM/effort-aware profile.
Salvaged from PR #29820 by @Epoxidex.
Five tests for the salvaged #37217 Bug B fix: vendor-field passthrough,
reasoning-key + private-key exclusion, merge-over-existing (fast-mode
speed), no-extra_body regression guard, reasoning-only adds nothing.
Live probes against api.anthropic.com informed the exclusion design:
Anthropic strictly validates the request body (unknown keys 400 with
'Extra inputs are not permitted'), so the passthrough forwards only
caller-configured fields and never the OpenAI-shaped reasoning dict
(translated natively) or _-private plumbing keys.
Catalog entries now follow the same supply-chain rules as pyproject
dependencies:
- n8n: install.ref main -> full commit SHA 7a9ae007 (2026-05-23,
branches/tags can be moved by the upstream owner; SHAs cannot)
- new contract test: every shipped manifest must pin exactly —
git installs need a 40-char SHA, uvx/npx-style launchers need
pkg==X / pkg@X with a digit-leading version (rejects bare names,
ranges, and npm dist-tags like @latest)
- module docstring documents the pin policy (exact version, 2-week
cooldown)
unreal-engine and linear are http transports (server runs elsewhere)
so there is nothing to pin at the transport layer.
Verified: unpinning blender-mcp in the manifest makes the contract
test fail with a named diagnostic; restoring the pin passes.
Adds optional-mcps/blender (ahujasid/blender-mcp, stdio via uvx). The
server advertises 22 tools; 18 front optional asset services with no
upstream trim mechanism, so tools.default_enabled pins the install to
the core surface (scene/object info, viewport screenshot, code exec)
and the rest stay opt-in through 'hermes mcp configure blender'.
Manifests can now declare transport.env (static, non-secret subprocess
env vars), parsed/validated in _parse_manifest and written by
_build_server_config — used here to ship DISABLE_TELEMETRY=true per
the no-telemetry-without-opt-in policy. Runtime already honored
per-server env; manifests just couldn't declare it.
A large Codex request (estimated context >= 10k tokens) disables the no-byte
TTFB watchdog on purpose, and openai_codex_stale_timeout_floor *raises* the
stale timeout (up to 1200s at >100k tokens) so healthy gateway-scale payloads
aren't aborted mid-prefill. When the backend genuinely stalls — no first byte
AND no events, exactly the #64507 symptom — the request is only reclaimed at
that high stale floor, so the session can hang for 13+ minutes with an idle
slash_worker and no ended_at/end_reason while Desktop still shows it as active.
Add a flat, finite hard ceiling on total openai-codex request time that always
applies (min() of the computed stale timeout and the ceiling) regardless of the
TTFB-disable / stale-floor interaction. A stalled large request is now killed
at the ceiling and the retry loop / visible failure path takes over instead of
hanging indefinitely. Tunable via HERMES_CODEX_HARD_TIMEOUT_SECONDS (default
600s; 0 disables the ceiling to restore pre-fix behavior).
Closes#64507
TRUNCATE checkpoint every 50 writes causes B-tree corruption on large
databases (65K+ pages) due to the exclusive-lock I/O pressure from
checkpointing thousands of frames at once.
Switch periodic _try_wal_checkpoint() to PASSIVE mode which does not
require an exclusive lock and cannot corrupt pages under I/O pressure.
Keep TRUNCATE in close() and pre-VACUUM paths where it is safe
(infrequent, controlled conditions).
Also replace silent `except Exception: pass` with logged warnings for
checkpoint failures so operators can detect early corruption signals.
Fixes#45383
_summarize_tool_result() built the web_extract summary straight from the
first `urls` entry. When web_search results are forwarded into web_extract
(a common chain), that entry is a dict ({"url"/"href": ...}) rather than a
URL string. With 2+ URLs the `url_desc += " (+N more)"` step then raised
`TypeError: unsupported operand type(s) for +=: 'dict' and 'str'`, aborting
the pre-compression pruning pass; with a single URL the raw dict repr
leaked into the summary text.
Unwrap the URL from dict entries before use — mirroring the existing
web_extract handling in agent/display.py (`_display_url`),
tools/web_tools.py (`_web_extract_url`) and acp_adapter/tools.py
(`build_tool_title`) — so `url_desc` is always a string and the
concatenation is always str + str.
Adds regression tests covering multiple/single dict URLs, the href key,
malformed dicts, and the plain-string path.
test_normalize_codex_response_salvage_is_xai_scoped broke on main when
two same-day merges crossed: #64764 (#64434 — trust response.status for
reasoning-only turns on UNRECOGNIZED Responses backends) changed what a
bare _normalize_codex_response(response) call returns for
status='completed' reasoning-only output (now 'stop'), while #64768
added this test calling with no issuer_kind and expecting 'incomplete'.
The test's intent is that the xAI reasoning-channel salvage does not
leak into other special-cased backends — pin issuer_kind='codex_backend'
so it exercises exactly that (same pattern as
test_normalize_codex_response_treats_summary_only_reasoning_as_incomplete,
which was already pinned for #64434).
Community reports of GPT 'infinitely thinking' are usually a slow or
overloaded provider plus silent retry machinery: the CLI/TUI/Desktop
spinner shows a generic 'cogitating...' verb for the whole wait and the
gateway heartbeat says only 'Working — N min'.
Add AIAgent._emit_wait_notice(): rewrites the live spinner/status line
(thinking_callback → CLI prompt_toolkit widget, thinking.delta → TUI +
Desktop) and updates the activity tracker (included in the gateway's
'⏳ Working — N min' heartbeat). Wired at the four wait points:
- non-streaming wait loop: after 30s with no response, the line becomes
'⏳ waiting on <model> — Ns with no response yet (provider may be slow
or overloaded; auto-reconnect at Ns)'
- streaming wait loop: same explanation after 30s with no chunks,
including the long-thinking case
- TTFB / stale-stream kills: '⚠ no response from provider in Ns —
reconnecting...'
- Codex continuation retries: '↻ model returned reasoning with no final
answer — asking it to continue (n/3)' instead of silence
Notices are fail-open (display errors never break the wait loop) and
gateway sessions without a display callback still get the improved
activity description.
grok-4.x on the xAI /v1/responses surface sometimes ends a turn with only
reasoning items — no message output item, no tool calls — and those
reasoning items carry no encrypted_content. Two compounding problems:
1. The model occasionally emits its final answer INSIDE the reasoning
channel, delimited by grok's internal "<response>" tag. The answer
exists but is classified reasoning-only → finish_reason=incomplete.
2. An interim assistant message holding only plain-text reasoning replays
as nothing in _chat_messages_to_responses_input, so every continuation
request is byte-identical to the one that just failed. The model
deterministically repeats the reasoning-only response until the retry
budget is exhausted and the turn dies with "Codex response remained
incomplete after 3 continuation attempts".
Fixes:
- _normalize_codex_response (xai_responses only): salvage the
<response>-delimited tail from the reasoning text and promote it to
assistant content; the untagged prefix stays as thinking text.
- Codex-incomplete continuation path: when the interim message has
nothing the input converter will replay (no content, no encrypted
reasoning items, no message items), append a user-role nudge so the
retry actually differs and explicitly asks for the final answer /
pending tool call. Mirrors the existing _get_continuation_prompt
pattern used for length truncation.
Observed live with grok-4.20 on xai-oauth (2026-07-13); sibling of the
grok-composer web_search incomplete-loop fix in transports/codex.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #64434 change makes unrecognized issuers trust
response.status='completed' for reasoning-only turns, so this sibling
test (which exercised the old default-path behavior) now pins the Codex
backend explicitly — the surface where reasoning-only still means
'still thinking'.
Follow-up to the salvaged #64449: the status-trusting branch flipped
github_responses to 'stop' alongside unknown relays. Copilot fronts the
same OpenAI model family as codex_backend and shows the same
reasoning-only 'still thinking' degeneration, so it stays on the
continuation path. Only unrecognized (other:*) backends trust
response.status='completed' as terminal.
The aggregator is MoA's acting model, but the main loop's reasoning
gates key off the virtual moa://local identity and never fire — so with
no per-slot reasoning_effort the aggregator silently ran at the backend
default, ignoring the user's reasoning config entirely (#64187).
New _aggregator_reasoning_config(): slot value > full acting-model
resolution via the shared chokepoint (agent.reasoning_overrides for the
slot's model > global agent.reasoning_effort; YAML False stays
'disabled'). Applied to both aggregator call sites (acting turn +
one-shot /moa synthesis).
Reference advisors intentionally keep slot-or-default: inheriting a
global xhigh into every advisor fan-out would silently multiply cost.
Fixes#64187.
The remote model catalog (website/static/api/model-catalog.json) now labels
exactly one entry per provider block with "default": true — z-ai/glm-5.2 for
both OpenRouter and Nous Portal. That labeled entry is the model Hermes
silently lands on when the user never picked one, and it can be rotated by
editing the manifest alone: no release needed.
- model_catalog.py: get_default_model_from_cache() reads the label from the
in-process/disk cache only — never triggers a network fetch, so hot
resolution paths (agent build, gateway session setup) stay network-free.
- models.py: get_preferred_silent_default_model() resolves catalog label
first, PREFERRED_SILENT_DEFAULT_MODEL constant second (offline/fresh
install). _PROVIDER_SILENT_DEFAULT_OVERRIDES dict replaced by
_SILENT_DEFAULT_PROVIDERS routing through the shared resolver.
fetch_openrouter_models() preserves the "default" badge through live
/v1/models refreshes so the picker shows it.
- scripts/build_model_catalog.py: generator emits the default label so
regeneration can't drop it.
- website/docs/reference/model-catalog.md: schema documents the new field.
- Salvaged from PR #61141 (@HumphreySun98): bare-provider /model switches
(/model nous) route through the cost-safe default instead of curated
entry [0].
- tests: catalog-label precedence, constant fallback, stale-label fallback,
cache-only (no network) guarantee, and a shipped-manifest contract test
pinning the labeled entry to PREFERRED_SILENT_DEFAULT_MODEL.
E2E (temp HERMES_HOME): fresh-install constant fallback, shipped-manifest
label read, release-free rotation (relabeled cache -> new default across
models.py, tui_gateway, and gateway empty-model paths) all verified.
`detect_static_provider_for_model` handles a bare provider name typed as a
model (e.g. `/model nous`) by returning `_PROVIDER_MODELS[provider][0]` — the
first curated entry. For metered aggregators whose curated list is ordered
most-capable-first (Nous Portal), entry [0] is the priciest flagship, so
`/model nous` silently switched to it.
This is exactly the billing footgun `_PROVIDER_SILENT_DEFAULT_OVERRIDES` /
`get_default_model_for_provider` exist to prevent (per their docstring, a
missing model "escalated to Opus and billed 863 requests before the user
noticed"). The non-interactive fallback already routes through that cost-safe
helper; the interactive `/model <provider>` path did not.
Route this path through `get_default_model_for_provider` too. Providers
without a silent-default override are unchanged (the helper returns
`models[0]`), so only overridden providers (currently `nous`) change — from
the flagship to the low-cost default.
Adds regression tests: `/model nous` resolves to the cost-safe default, and
non-overridden providers still resolve to their first catalog model.
Consolidation follow-up on top of @liuwei666888's compressor fix (#52431)
and @Frowtek's ACP render guard (#62588) — completes the class fix across
every display-layer consumer of tool-call arguments:
- agent/context_compressor.py: wrap _summarize_tool_result in a
never-raises backstop (same pattern as get_cute_tool_message and the
ACP guard). The per-branch _str_arg coercions keep summaries
informative; the wrapper guarantees compression can never crash-loop
on a summary branch we didn't anticipate. Also reject non-dict parsed
args (a JSON list/scalar would crash every args.get call).
- agent/display.py: build_tool_preview's process branch sliced
session_id/data without coercion — crashed build_tool_preview and
build_tool_label on the live tool-progress callback (probed live:
TypeError 'int' object is not subscriptable). Coerce like the
sibling branches.
- tests: backstop fuzz matrix (16 tools x 6 hostile value shapes),
fallback-shape contracts, display process-preview regressions,
non-dict args guard.
build_tool_start renders every ACP (Zed) tool call — on the live tool-progress
callback (acp_adapter/events.py) and during session history replay
(acp_adapter/server.py). It called build_tool_title and extract_locations
directly, so a model that emits a malformed argument crashed the render:
- terminal `command` as null/number -> TypeError (len() in build_tool_title)
- delegate_task `goal` as a number -> TypeError (len())
- read_file `path` as a non-string -> pydantic ValidationError building a
ToolCallLocation
A live crash breaks the tool-call event; a persisted one breaks history replay
on every resume of that session. The sibling CLI label builder
get_cute_tool_message was already wrapped for exactly this reason
(agent/display.py: "display must never abort a turn").
Wrap build_tool_start the same way: on any builder failure, fall back to a
minimal, valid start event (tool name as title, resolved kind). The happy path
is unchanged.
Adds tests for the non-string command, path, and goal cases.
* test: deflake async-delegation interrupt test under CI load
test_interrupt_all_signals_running_children failed twice on a loaded CI
worker: the blocker's ev.wait(timeout=5) expired before interrupt_all()
ran, the record finalized on its own, and interrupt_all() found nothing
running (n == 0, interrupted count 0 — the exact CI assertion failure,
reproduced locally by inserting a 5.6s dispatch->interrupt gap).
Raise the internal safety timeouts from 5s to 60s across the file's
gated runners — they exist only as runaway guards (every test releases
its gate explicitly); the pytest-level timeout is the real backstop.
Same flake class as the removed test_crashed_runner_produces_error_
completion (#64431).
* test: fix cross-test completion-event leak (second flake mechanism)
The first CI failure was the 5s guard-timeout race; the rerun exposed a
SECOND mechanism with a different signature: 'completed' == 'interrupted'.
Prior tests (e.g. test_dispatch_rejected_at_capacity) release their gate
and return immediately, but their workers finalize asynchronously — on a
loaded runner the teardown drain races the in-flight _finalize, and the
straggler 'completed' events leak into the NEXT test's queue, where
_drain_one() picks one up instead of the interrupt event. Reproduced
locally: gate release + immediate drain + 0.15s finalize delay leaked 2
events.
Fixes:
- teardown waits (bounded 2s) for active workers to finalize before
draining, so events land in the owning test
- the interrupt test matches its OWN delegation_id via _drain_for()
instead of taking whatever event arrives first
The 0.14 upgrade (#63970) dropped the @assistant-ui/store override, so the
whole cluster de-hoisted from root node_modules into apps/desktop/node_modules
under a single shared tap@0.9.3. The test only looked at the root hoist path
(node_modules/@assistant-ui/tap), which no longer exists, and failed on main.
Resolve tap wherever npm places it (root or workspace-nested), and assert a
single shared version across all install sites — strengthening the invariant to
also catch a split tap install, not just split declared ranges.
The chat_completions path raises EmptyStreamError when a stream yields
no chunks and no finish_reason (#64420). _call_anthropic() had no
equivalent, and the eventless failure surfaces differently by client:
- Real Anthropic SDK: no message_start means no final-message snapshot,
so get_final_message() raises a bare AssertionError — not in the
retry loop's transient set, so it burned no retries and surfaced raw.
- OpenAI-compat shims: may fabricate a contentless Message with no
stop_reason (or return None), flowing out as a 'successful' empty turn.
Track whether any stream event arrived and normalize both shapes to
EmptyStreamError, giving the anthropic path the same transient retry
budget in the shared _call() retry loop. A real completed response
always carries a stop_reason, so the guard cannot fire on legitimate
turns (including eventless mocks with stop_reason='end_turn').
Follow-up to #64420.
