Widen okalentiev's failed_model narrowing (#59561) to
_try_main_agent_model_fallback. The safety-net layer still skipped on a
provider-label match alone, so single-provider users whose aux compression
model and main model share one custom endpoint had ZERO fallbacks: the aux
model timing out exhausted the chain in one hop and compression aborted.
Real incident (0.19.0 debug dump): aux zai-org/glm-5.2 hung 324s and timed
out while main mindai/macaron-v1-venti on the SAME endpoint was serving
448K-token turns — the label-only skip discarded the one viable summarizer,
the session wedged over threshold, and the anti-thrash breaker tripped.
Same convention as the chain fix: model-specific failures (timeout,
connection, rate limit) pass failed_model so only the exact failed model is
skipped; provider-wide failures (auth 401 / payment 402) pass None and keep
the whole-provider skip. Both sync and async call_llm sites pass it.
Sabotage-verified: the new regression test fails on the provider-only skip.
Follow-up to the same-provider fallback fix: narrowing the configured-chain
skip to the exact failed model is only correct for model-specific failures.
Auth (401) and payment (402) errors are provider-wide — every model on the
provider shares the same broken credentials/account — so trying a sibling
model can't recover and merely burns another doomed request before the
aux task fails. Worse, returning that sibling client bypasses the
main-agent-model safety net that a provider-wide skip would have reached.
Only forward failed_model to _try_configured_fallback_chain for
model-specific failures (timeout, connection, rate limit, model-incompatible,
invalid response). Auth/payment keep failed_model=None (whole-provider skip),
preserving the pre-existing safety-net behaviour for credential/billing
failures.
Adds an integration test that a timeout forwards the failed model, and
updates the payment-error test to assert failed_model=None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_try_configured_fallback_chain skipped every fallback_chain entry whose
provider matched the one that just failed. A chain that intentionally lists
several models under the same provider (e.g. two more NVIDIA NIM models
after the primary NIM model times out) was therefore skipped wholesale,
falling straight through to the main-agent-model safety net instead of
trying the other configured models on that provider.
Add failed_model so the skip narrows to the exact (provider, model) pair
that failed. Callers that only know the provider (client-build failures,
where the whole provider is unreachable regardless of model) keep the old
provider-wide skip; the two runtime request-error call sites (call_llm,
async_call_llm) now pass the model that just failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The gateway's pre-agent session-hygiene compression killed the summary
call at a fixed 30s wall-clock deadline (compression.hygiene_timeout_seconds),
regardless of whether the summary model was hung or merely slow. A reasoning
model happily streaming a large summary was cut off mid-generation, the user
got '⚠️ Context compression timed out after 30.0s', and a 300s failure
cooldown left the session oversized — a doom loop for slow-but-healthy
auxiliary models.
Timeouts are now liveness-based instead of wall-clock-based:
- agent/auxiliary_client.py: new thread-local aux_progress_hook. When
installed (only by context compression today), the primary call_llm
attempt streams (stream=True) and aggregates chunks back into a complete
response, ticking the hook per chunk. The configured timeout then acts
per stream read (idle) instead of as a total budget. Providers that
reject streaming fall back to the plain non-streaming call; auth/payment/
rate-limit/transport errors propagate unchanged into the existing
recovery chains. Codex Responses (per SSE event) and Anthropic Messages
(per stream event, via the new create_anthropic_message on_stream_event
callback) tick the same hook from inside their wire adapters.
- agent/conversation_compression.py: CompressionCommitFence gains
touch_progress()/seconds_since_progress(); compress_context() installs
fence.touch_progress as the progress hook around the compress call.
- gateway/run.py: the hygiene wait loop treats hygiene_timeout_seconds as
an INACTIVITY budget — while the fence reports fresh progress the wait
extends, bounded by the new compression.hygiene_total_ceiling_seconds
(default 600s, clamped >= the idle budget) so a degenerate trickle
stream still dies. The timeout warning now says the summary model
produced no output, which is the only case that still triggers it.
- config/docs: hygiene_total_ceiling_seconds added to DEFAULT_CONFIG and
configuration.md; hygiene_timeout_seconds documented as inactivity-based.
Tests: tests/agent/test_aux_progress_streaming.py (hook plumbing, stream
aggregation incl. tool-call deltas and reasoning deltas, rejection
fallback, ceiling kill, fence progress surface); two new gateway tests
prove a slow-but-streaming worker survives past the fixed timeout
(sabotage-verified: fails with the old fixed deadline) and a
forever-trickling worker is still cut off at the ceiling.
OpenAI documents GPT-5.5 / GPT-5.5 Pro as extended-cache-only: in-memory
prompt cache retention is not available for them, and only
prompt_cache_retention: "24h" is supported. Responses requests that omit
the field see near-zero cached_tokens even with a stable prompt_cache_key
and identical prefixes (observed on an OpenAI-compatible Responses relay:
0 cached across repeated identical calls before; 97% cache reads after).
Send the field for the gpt-5.5 model family (bare and namespaced ids like
openai.gpt-5.5) on OpenAI-compatible Responses routes, mirrored in the
auxiliary Codex adapter, and pass it through preflight normalization.
Skipped for xAI, GitHub/Copilot, and the chatgpt.com Codex backend, which
reject or ignore body-level cache fields.
The Vertex AI provider (added same-day, commit c73e74386) was never added to
either of the two provider registries that agent/auxiliary_client.py and the
MoA slot-resolution chain depend on, breaking Vertex outside the main
conversation loop:
1. hermes_cli/auth.py::PROVIDER_REGISTRY had no "vertex" entry. The
plugin-auto-extend loop that normally fills gaps explicitly skips
non-api_key auth types (`if _pp.auth_type != "api_key": continue`), and
Vertex was never hand-declared like "bedrock" is. Because
resolve_provider_client() in agent/auxiliary_client.py gates everything
on `pconfig = PROVIDER_REGISTRY.get(provider)` and returns (None, None)
immediately when pconfig is None, its `elif pconfig.auth_type == "vertex"`
branch was permanently dead code — every auxiliary Vertex call (vision,
title generation, reflection, context compression, MoA reference/
aggregator slots) failed outright, not just a MoA-specific edge case.
2. hermes_cli/providers.py::HERMES_OVERLAYS also had no "vertex" entry, so
hermes_cli.providers.get_provider("vertex") returned None. This backs
_preserve_provider_with_base_url() in agent/auxiliary_client.py, which a
MoA slot's resolved (base_url, api_key) pair needs to keep its "vertex"
identity instead of silently collapsing to "custom" — losing the
identity _refresh_provider_credentials() needs to re-mint an expired
OAuth2 token (~1h lifetime) on a 401, and permanently breaking every
subsequent call in that MoA preset for the rest of the session.
Fix mirrors the existing "bedrock"/aws_sdk entries in both registries
exactly, plus adds a "vertex" branch to _refresh_provider_credentials() (it
had branches for openai-codex/nous/anthropic/xai-oauth but not vertex,
so a 401 fell through to `return False` without evicting the stale cached
client).
- hermes_cli/auth.py: hand-declared vertex ProviderConfig(auth_type="vertex")
in PROVIDER_REGISTRY, matching bedrock's shape.
- hermes_cli/providers.py: vertex HermesOverlay(auth_type="vertex") in
HERMES_OVERLAYS + "Google Vertex AI" label override.
- agent/auxiliary_client.py: vertex branch in _refresh_provider_credentials
that re-mints the token via get_vertex_config() and evicts the stale
cached client.
- 8 new regression tests across tests/hermes_cli/test_vertex_provider.py and
tests/agent/test_auxiliary_client.py: registry membership, end-to-end
resolve_provider_client("vertex", ...) building a working client (proving
the previously-dead branch is now reachable), and the 401-refresh/cache-
eviction path.
Requested in review: builder-level assertions that the gemini-native
branch forwards max_tokens (provider names and the native
generativelanguage.googleapis.com base_url, max_tokens=600), plus a
control showing gemini models on OpenAI-compatible endpoints — including
Gemini's own /openai compatibility endpoint — keep the existing omission
behavior (#34530).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per review feedback from teknium1: reference_max_tokens is an advisors-only
contract. The aggregator is the acting model and must not be capped by the
reference budget. Changed _is_moa from startswith('moa_') to exact match on
'moa_reference'. Added regression test proving aggregator does NOT receive
max_tokens.
Copilot review pointed out that hardcoding kwargs['max_tokens'] would
400 on models requiring max_completion_tokens (GPT-5 family, Copilot).
The existing auxiliary_max_tokens_param() helper already selects the
correct parameter name per model — use it instead of hardcoding.
Test updated to parametrize expected_key so the Copilot gpt-5.5 case
correctly asserts max_completion_tokens instead of max_tokens.
Addresses Copilot review comments on both files.
PR #56756 added reference_max_tokens to cap MoA advisor output and cut
turn latency. The value is correctly threaded through five layers of MoA
code (moa_config → conversation_loop → aggregate_moa_context →
_run_references_parallel → _run_reference → call_llm(task='moa_reference',
max_tokens=800, ...)).
However, _build_call_kwargs() in auxiliary_client.py silently drops
max_tokens for all OpenAI-compatible providers (PR #34845, which fixed
endpoints and NVIDIA NIM keep it. This means reference_max_tokens never
reached the API for the vast majority of providers.
The bug affects every OpenAI-compatible MoA reference/aggregator slot:
Z.AI (coding plan), OpenRouter, OpenAI, GitHub Copilot, and local
providers. Only Anthropic-compat endpoints (MiniMax, /anthropic URLs)
worked — by coincidence, not MoA-aware design.
Fix: thread the 'task' parameter through all six _build_call_kwargs()
call sites. When task starts with 'moa_', max_tokens is always included
in the request kwargs regardless of provider. Non-MoA auxiliary tasks
(compression, titles, vision, etc.) keep PR #34845 behavior unchanged.
Verified end-to-end:
- Z.AI GLM-5.2 with max_tokens=50 → returned exactly 50 tokens
- Z.AI GLM-5.2 with max_tokens=20 → returned exactly 20 tokens
- Z.AI GLM-5.2 uncapped → returned 315 tokens
- 7 new regression tests covering 4 providers, Anthropic wire, non-MoA
tasks, and prefix-matching boundary
- 288 auxiliary_client tests pass (was 281, +7 new), 84 MoA tests pass
- Zero regressions
'auto' is a sentinel meaning "inherit from main runtime / auto-detect",
not a literal model id -- already handled for cfg_model (config-derived)
in _resolve_task_provider_model, but not for the explicit `model` kwarg.
MoA reference/aggregator slots (agent/moa_loop.py's _slot_runtime) forward
a preset's `model:` field as this explicit argument rather than through
auxiliary.<task> config, so a MoA preset configured with `model: auto`
(a natural thing to try given the existing auxiliary.*.model: auto
convention) reached this function as the explicit `model` arg and took
the `model or cfg_model` branch, bypassing the cfg_model-only sentinel
check entirely -- sending the literal string "auto" to the wire as a
model id.
Normalize both the explicit `model` and `cfg_model` the same way, fixing
this at the single chokepoint every caller (MoA included) already goes
through, rather than patching moa_loop.py separately.
Follow-up to srojk34's explicit-provider unwrap (PR #56691):
- Extract _resolve_moa_aggregator() as the single preset->aggregator
resolver shared by _resolve_auto(), _resolve_task_provider_model(),
and resolve_provider_client() so preset lookup/validation can't drift.
- When the main provider is moa, the aggregator model is now the default
for every UNSET auxiliary model: _read_main_model_for_aux() substitutes
the preset's acting (aggregator) model wherever fallback chains
pre-filled from _read_main_model() (router prefill, custom-endpoint
fallback, named-custom default, external-process default,
_try_main_agent_model_fallback).
- Unwrap moa at the resolve_provider_client() chokepoint so direct
callers (vision auto-detect, plugin code) can't dead-end in the
unknown-provider branch, and unwrap the vision auto-detect main
provider before capability probes run against the preset name.
- Real-config tests: temp HERMES_HOME + actual config.yaml exercising
the genuine load_config()/resolve_moa_preset() boundary.
_resolve_task_provider_model() returned an explicit provider="moa" override
(from a caller-passed arg, or auxiliary.<task>.provider: moa in config.yaml)
verbatim, with no MoA-preset unwrap. Only the *implicit* "main provider is
moa" path inside _resolve_auto() unwraps to the aggregator slot (#53827) —
this function never goes through _resolve_auto() at all, so the explicit
case was never covered.
MoA is a virtual provider with no real HTTP endpoint: resolve_provider_client()
looks "moa" up in PROVIDER_REGISTRY (no such entry), falls to the
unknown-provider dead end, and call_llm surfaces a nonsensical "Provider
'moa' is set in config.yaml but no API key was found. Set the MOA_API_KEY
environment variable..." error for a provider that was never meant to be
reached over the wire.
Fix mirrors #53827's aggregator-resolution approach exactly: when either the
explicit `provider` arg or the config-derived `cfg_provider` is "moa",
resolve the named (or default) MoA preset via resolve_moa_preset() and
continue with its aggregator's real provider+model, dropping any explicit
base_url/api_key (the moa:// virtual endpoint and placeholder key belong to
the facade, not the aggregator's real provider). If the preset can't be
resolved (renamed/deleted), degrades gracefully to the pre-fix behavior
instead of raising harder.
- agent/auxiliary_client.py: _unwrap_moa_provider() helper + call sites for
both the explicit-arg and config-derived provider="moa" cases in
_resolve_task_provider_model(). Also tightened base_url/api_key parameter
types to Optional[str] (matching their actual None-accepting behavior),
which incidentally resolved 5 pre-existing ty diagnostics at call sites.
- 5 new regression tests in tests/agent/test_auxiliary_client.py: explicit
arg unwrap, config-derived unwrap, default-preset fallback when no model
is configured, graceful degradation on preset-resolution failure, and a
non-moa regression guard.
* feat(analytics): record auxiliary model usage per task in session accounting
Auxiliary LLM calls (vision, compression, title_generation, web_extract,
session_search, ...) discarded their token usage, leaving dashboard
analytics blind to aux model spend (issue #23270).
- hermes_state.py: session_model_usage gains a task PK dimension
(''=main loop) via v22 table-rebuild migration (SQLite can't alter a
PK); record_auxiliary_usage() writes per-(model,provider,task) deltas
WITHOUT touching sessions counters (gateway overwrites those with
absolute main-loop totals — folding aux in would double-count or be
clobbered). Aux rows never inherit the session's main-loop route.
- agent/aux_accounting.py: ContextVar ambient accounting context
(mirrors the portal_tags conversation context); record_aux_usage()
normalizes usage via usage_pricing.normalize_usage, estimates cost,
and is strictly best-effort. moa_reference/moa_aggregator excluded —
conversation_loop already folds MoA usage+cost into the main delta.
- agent/auxiliary_client.py: _validate_llm_response is the recording
chokepoint — every successful non-streaming aux response passes
through it exactly once, sync and async, including fallback paths
(model read from the response itself stays accurate across
fallbacks).
- run_agent.py: run_conversation publishes/resets the accounting
context; agent/title_generator.py republishes on its bare thread.
- hermes_cli/web_server.py: /api/analytics/usage folds aux rows into
by_model (aux-only models finally appear) and adds a by_task
summary; /api/analytics/models surfaces aux rows on the Models page.
Design per review of PR #62850 by @eeksock (thread-local + separate
auxiliary_usage table): rebuilt on ContextVar (async-safe — thread-local
cross-attributes concurrent coroutines on one event loop) and the
existing session_model_usage table instead of a parallel accounting
path, extended beyond vision to every aux task, and wired the analytics
endpoints so the dashboard actually shows it. Credit to @eeksock for
the approach and @tboatman for the detailed root-cause analysis.
* test(moa): match _validate_llm_response mock to new accounting-hint signature
* test(aux): accept accounting-hint kwargs in remaining _validate_llm_response mocks
Five tests for the salvaged #37217 Bug B fix: vendor-field passthrough,
reasoning-key + private-key exclusion, merge-over-existing (fast-mode
speed), no-extra_body regression guard, reasoning-only adds nothing.
Live probes against api.anthropic.com informed the exclusion design:
Anthropic strictly validates the request body (unknown keys 400 with
'Extra inputs are not permitted'), so the passthrough forwards only
caller-configured fields and never the OpenAI-shaped reasoning dict
(translated natively) or _-private plumbing keys.
The just-merged auxiliary.<task>.reasoning_effort shorthand applied
ensemble-wide to MoA (one value for every advisor) — wrong granularity.
Per-slot preset config supersedes it:
moa:
presets:
deep_review:
reference_models:
- {provider: ..., model: ..., reasoning_effort: low}
- {provider: ..., model: ..., reasoning_effort: xhigh}
aggregator:
{provider: ..., model: ..., reasoning_effort: high}
- Remove reasoning_effort from the moa_reference/moa_aggregator
DEFAULT_CONFIG blocks; _get_task_extra_body now warns-and-ignores the
key on MoA tasks, pointing at the preset config
- Guard tests: MoA aux blocks must not regrow the key; task-level value
is rejected with the pointer warning
- Docs: configuration.md notes the MoA exception and links the MoA page
Every auxiliary task block (vision, web_extract, compression,
title_generation, curator, background_review, moa_reference, ...) now
accepts a reasoning_effort shorthand:
auxiliary:
compression:
reasoning_effort: low
vision:
reasoning_effort: none
_get_task_extra_body() folds it into extra_body.reasoning, which every
auxiliary wire already translates: chat.completions passes it through,
the Codex Responses adapter maps it to top-level reasoning/include, and
the Anthropic auxiliary adapter now forwards it into
build_anthropic_kwargs(reasoning_config=...) (previously hardcoded None).
An explicit extra_body.reasoning on the same task wins over the
shorthand. Invalid levels are ignored with a warning. Empty string
(the shipped default) is a no-op — zero behavior change.
Config: reasoning_effort added to all 16 auxiliary task blocks in
DEFAULT_CONFIG (no version bump — deep-merge handles new keys).
_CodexCompletionsAdapter (agent/auxiliary_client.py) is a second,
independent producer of Codex Responses input — used by auxiliary
calls (context compression, flush_memories, MoA aggregation,
session_search) that route through CodexAuxiliaryClient instead of
the main agent's ResponsesApiTransport.build_kwargs. It calls
_chat_messages_to_responses_input() directly without is_github_responses,
so the previous commit's fix didn't cover it: an auxiliary call made
against a Copilot-backed session could still replay a connection-scoped
codex_message_items id and hit the same HTTP 401.
Detect the Copilot host from the adapter's own client.base_url (same
check the adapter already does further down for prompt_cache_key
opt-out) and pass is_github_responses through, closing the gap.
Still #32716.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A fallback candidate can itself carry a stale credential (e.g. an
expired ANTHROPIC_TOKEN picked up by _try_anthropic). Its 401 previously
propagated out of the fallback call site and aborted the auxiliary task
— for compression: a 60s cooldown + context marker while the session
kept growing past the context cap. Live case: mattalachia debug dump
(Jul 2026), Codex timeout → Anthropic 401 x5 → 296K 'Cannot compress
further'.
Now each fallback candidate call is wrapped: on auth error, refresh the
candidate's provider credentials and retry once; if unrefreshable, mark
the provider unhealthy and walk the discovery chain again so the next
viable candidate serves. Sync + async paths. Non-auth errors still
raise unchanged.
Infer the concrete auxiliary auth provider from the selected client base
URL so provider:auto routes can refresh Copilot/Codex/Anthropic/Nous
credentials after auth errors, instead of skipping refresh because
resolved_provider stayed 'auto'. Adds the copilot branch to
_refresh_provider_credentials and evicts the stale auto-route cache
before retrying.
Fixes#20832. Salvaged from PR #20837, reapplied surgically onto current
main (branch predated the _retry_same_provider_sync/async extraction).
Follow-up to the #55911 salvage: inherit model.api_key only when the aux
base_url resolves to the same hostname as the main model's base_url
(runtime override or config). A misconfigured aux endpoint on a different
host keeps the fail-safe no-key-required placeholder instead of leaking
the main credential cross-host.
When an auxiliary task is configured with provider=custom and an explicit
base_url but an empty api_key, the custom_key fallback chain in
resolve_provider_client() jumped straight to the no-key-required
placeholder without consulting model.api_key from config.yaml. Users
on self-hosted gateways who share the same endpoint and credentials for
both the main model and auxiliary tasks got 401 auth errors.
Add _read_main_api_key() following the same pattern as _read_main_model()
and _read_main_provider(): checks _RUNTIME_MAIN_API_KEY (runtime override)
first, then config.yaml model.api_key. Insert it into the fallback chain
before no-key-required so real credentials are used when available, while
local servers without auth still get the placeholder.
_resolve_task_provider_model returns early on an explicit provider arg,
which skips the config block that consults auxiliary.<task>.base_url /
api_key. Any caller passing provider explicitly (e.g.
resolve_vision_provider_client(provider="custom", ...)) bypasses the
configured custom endpoint and falls through to main-runtime resolution,
silently routing the task to the wrong backend.
Adopt the task's configured base_url/api_key before the early returns,
but only when no explicit base_url was given and the config targets the
same provider (or names none) — a caller forcing a *different* provider
keeps full explicit-arg priority, and an explicit base_url still wins
over config.
Fixes#58515
Two related hardening fixes for auxiliary calls (which include MoA reference
advisors — a pinned-model path where provider fallback is not a meaningful
recovery):
1. Transient-transport retries: the same-provider retry on a connection reset /
timeout / 5xx / 408 was a single attempt, then fallback. For a pinned aux
call a second blip silently loses the call (root of the run2 double-advisor
'Connection error' collapse — a genuine upstream blip). Now retries N times
with exponential backoff, N = auxiliary.transient_retries (default 2 -> 3
total attempts, clamped [0,6]). Compression-on-timeout fast-fail carve-out
preserved.
2. Per-model client-cache isolation: _client_cache_key excluded the model, so
two concurrent auxiliary calls to the same provider/base_url/key but
different models (e.g. an opus + gpt-5.5 MoA fan-out) shared one cache entry
and could race each other's client lifecycle. Model now participates in the
key -> distinct clients, no cross-call races. Same-model reuse unchanged.
- agent/auxiliary_client.py: _transient_retry_count() + backoff loop; model in
_client_cache_key and both call sites.
- hermes_cli/config.py: auxiliary.transient_retries default (2).
- tests: new retry/isolation tests; updated 2 stale-expectation tests to the
corrected behavior (per-model resolve; N-retry escalation).
Backoff base is overridable (_TRANSIENT_RETRY_BACKOFF_BASE) so tests don't sleep.
Follow-up to the salvaged fix: the regression test asserted a frozen
max_tokens == 128_000 literal, coupling it to the Opus-4-8 model table.
Assert against _get_anthropic_max_output("claude-opus-4-8") plus > 2000
instead, so the test survives model-table churn while still catching a
regression to the old `or 2000` fallback.
Two independent MoA auxiliary-call fixes:
#53866 — auxiliary.moa_reference.timeout and auxiliary.moa_aggregator.timeout
were 600s while moa_agent was 120s. Raise both to 900s so a genuinely long
reference/aggregator turn (mixed providers, deep reasoning, long tool chains)
has headroom instead of being cut mid-generation.
#53735 — _CodexCompletionsAdapter (the Codex/Responses auxiliary path used by
the MoA acting-aggregator, compression, web_extract, session_search, etc.)
never set prompt_cache_key, so it stayed cache-cold while the MAIN Responses
transport (agent/transports/codex.py) was warm. Derive the same
content-addressed key via the shared _content_cache_key(instructions, tools)
helper and set it on the aux Responses request, with the same host guards the
main transport uses (xAI carries the key in extra_body; GitHub/Copilot opts out
of cache-key routing).
Tests: 5 new prompt_cache_key cases (set+prefixed, stable across identical
prefix, differs on different instructions, skipped for xai/github hosts).
tests/agent/test_auxiliary_client.py 279 pass; tests/hermes_cli/test_config.py
130 pass.
Upstream #52270 added `_nous_inference_env_override()` but wired it into
only `resolve_nous_runtime_credentials`. Three sibling resolution paths
still ignored the override, so a self-hosted Nous inference endpoint set
via `NOUS_INFERENCE_BASE_URL` was silently dropped whenever credentials
arrived through any of them:
- the credential-pool path (`_resolve_runtime_from_pool_entry`)
- the explicit-provider path (`_resolve_explicit_runtime`)
- the auxiliary side-LLM client (`_pool_runtime_base_url`)
Route all three through the same auth-layer reader so every
`NOUS_INFERENCE_BASE_URL` read shares one normalization path
(trailing-slash stripping, blank -> empty) and the documented
trusted-bypass intent stays in one place. The override is live-only: it
wins for the base URL returned this run but is never persisted to
auth.json or the credential pool, so an ephemeral dev/staging value
cannot poison durable auth state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_resolve_task_provider_model() flattened any explicit base_url to
provider=custom. Correct for bare/custom endpoints, but wrong for
provider-backed routes (anthropic, qwen-oauth, minimax-oauth,
openai-codex, etc.) whose provider branch adds auth refresh, transport,
or request shaping. MoA reference slots resolved through those providers
lost their identity before the aux call, so e.g. a Codex reference hit
chatgpt.com/backend-api/codex without its Cloudflare headers and got
HTML back (surfacing as a spurious rate-limit).
Keep first-class providers intact when paired with a resolved base_url
via _preserve_provider_with_base_url(); bare/custom/auto/unknown and the
direct openai alias still route through custom.
Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>
The auxiliary OpenAI clients were built without overriding the SDK's
default max_retries=2, so every aux call silently made up to 3 attempts
against a slow/hung endpoint — a 120s timeout could stall ~360s before
Hermes saw a single failure. On the critical compression preflight path,
Hermes then added its own same-provider timeout retry on top, roughly
doubling the user-visible stall again before fallback.
- Build both the sync (_create_openai_client) and async (_to_async_client)
aux clients with max_retries=0 (setdefault, so explicit callers still
override). Hermes already owns retry + provider/model fallback policy.
- For task == compression, skip the same-provider transient retry on a
full-budget timeout and fall straight through to fallback. Fast blips
(streaming-close, 5xx) still retry, since those are cheap.
- Add _is_timeout_error to distinguish a full-budget timeout from a fast
connection drop.
Addresses the retry-multiplication root cause of #54465 (the resume-wedge
persistence half landed in #55499).
_try_openrouter() returned (None, None) whenever an OpenRouter credential
pool existed but was exhausted (_select_pool_entry -> (True, None)), making
the OPENROUTER_API_KEY env-var fallback unreachable. Auxiliary tasks
(compression, vision, web_extract) silently failed even with a valid env key.
Now the pool-present branch only returns early when it successfully builds a
client; an exhausted pool falls through to the env-var path. The final
failure (pool exhausted AND no env var) still marks the provider unhealthy.
Fixes#23452.
Co-authored-by: ambition0802 <noreply@github.com>
When the primary provider returns 401 and the auth-refresh path is
unavailable or fails, both call_llm() and async_call_llm() reached the
should_fallback gate without _is_auth_error in the condition, so the
auxiliary task (e.g. compression) was dropped silently — losing message
history. Add _is_auth_error to should_fallback (NOT is_capacity_error) in
both sync and async paths, plus an 'auth error' reason branch.
Auth stays a non-capacity error: it falls back in auto mode via the
is_auto gate, but on an explicitly-configured provider it still respects
the user's choice and raises rather than silently switching providers.
The salvaged context-window screen (#52392) skips fallback candidates that
are too small, and the rate-limit/403 fixes skip candidates that are at
capacity. A third hard failure remained uncovered: a fallback that builds a
client fine but returns a 400 because it structurally cannot run the model.
The canonical case is a configured openai-codex / ChatGPT-account fallback
asked to compress a glm-5.2 conversation:
400 - {'detail': "The 'glm-5.2' model is not supported when using
Codex with a ChatGPT account."}
This is a request-validation error, so should_fallback was False and the
explicit-provider gate blocked it — the auxiliary task (compression) aborted
every turn, dropping middle turns without a summary and churning the session,
which is exactly what destroys the prompt cache.
Adds _is_model_incompatible_error() (400 + capability phrasing, excluding
not-found and billing 400s which the sibling predicates own) and treats it as
a fallback-worthy capacity error in both sync and async call_llm, so the chain
skips the incapable route and continues to the next viable candidate.
The runtime auxiliary fallback chain (_try_configured_fallback_chain and
_try_main_fallback_chain) returned the first reachable candidate without
checking whether the candidate's context window was large enough for the
task. For task='compression' this meant a reachable but undersized
fallback (e.g. 32K) could be selected and then fail, even when a later
larger-context fallback was available.
This adds two small helpers:
_task_minimum_context_length(task)
Returns MINIMUM_CONTEXT_LENGTH (64K) for compression, None for
other tasks (vision, web_extract, etc.).
_candidate_context_window(provider, model, ...)
Thin wrapper around get_model_context_length that returns None on
probe failure so unknown/custom endpoints pass through unchanged
(preserves the existing fallback surface).
Both fallback loops now skip reachable candidates whose resolved context
is below the task minimum and continue iterating. The success path
(first viable candidate wins) is unchanged. Return shape and ordering
for healthy candidates are preserved.
Six regression tests cover:
L2 configured chain skips too-small candidate
L2 chain continues after skipping, returns last viable
L3 main chain skips too-small candidate
L4 unknown-context candidate passes through
L5 non-compression task is not filtered
L6 minimum constant matches MINIMUM_CONTEXT_LENGTH (64K)
3/6 fail on upstream/main without the production change (verified); all
6 pass with the fix. Full test_auxiliary_client.py suite (231 tests)
and related compression tests (130 tests) remain green.
When an explicit aux provider cannot build a client before any request is
sent (missing raw env key, exhausted/unavailable OAuth or credential-pool
auth, resolver returning (None, None)), call_llm raised a misleading
"no API key was found" error and bypassed the configured fallback_chain
entirely. A provider authenticated through Hermes auth / the credential
pool (e.g. ollama-cloud) whose pool entry is exhausted hit this path, so
compression failed instead of routing to the configured fallback.
Adds _try_configured_fallback_for_unavailable_client() and wires it into
both sync and async call_llm before the raise, and into the startup
compression feasibility check.
Salvaged from #51835 by @herbalizer404.
Rate-limit (429) errors on explicit-provider auxiliary tasks were
silently failing instead of triggering the fallback chain. The
is_capacity_error gate only checked payment and connection errors,
excluding rate limits — so when a configured provider like
openai-codex hit its rate limit, auxiliary tasks (kanban_decomposer,
vision, web_extract, approval, etc.) had zero resilience.
Add _is_rate_limit_error() to is_capacity_error at both call sites
(sync and async paths) so rate limits trigger fallback regardless
of whether the provider was auto-detected or explicitly configured.
Fixes#52228
Ollama Cloud (and similar) return 403 with bodies like "this model requires
a subscription, upgrade for access" or "you have reached your session usage
limit, upgrade for higher limits". These are capacity/billing conditions
semantically identical to credit exhaustion, but _is_payment_error() did not
recognize them (403 missing from the status set; keywords missing), so the
configured fallback_chain was never tried and compression failed outright.
Adds 403 to the status set and the subscription/session-usage keywords.
Salvaged from #49076 by @herbalizer404.
