_normalize_main_model_assignment() (POST /api/model/set, the endpoint
Desktop's Settings -> Model page uses to persist the main model slot) has
a fallback for a specific analytics bug: an older session row with no
billing_provider sends the model's bare vendor prefix as "provider"
(e.g. "anthropic" from "anthropic/claude-opus-4.6"), so the code detects
an unrecognized provider paired with a slash-bearing model and treats it
as that stray-vendor-prefix case.
Named custom providers are represented as "custom:<name>" slugs
everywhere else in the codebase (runtime_provider.py, model_switch.py),
but _KNOWN_PROVIDER_NAMES only lists the bare "custom" bucket. So picking
a named custom provider (e.g. "custom:litellm", a LiteLLM proxy fronting
Ollama) together with a slash-bearing model ("ollama/glm-5.2") looked
identical to the stray-vendor-prefix case and got silently rewritten to
provider: openrouter in config.yaml on save -- reassigning the provider
entirely, not just mangling the model id.
Exclude anything starting with "custom" from the fallback, matching the
guard the same function already applies later for the actual
normalize_model_for_provider call.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_MATCHING_PREFIX_STRIP_PROVIDERS includes "custom" so that manually typed
config values like "zai/glm-5.1" repair themselves for their matching
native provider. But "custom" is a generic bucket for arbitrary
user-defined endpoints, not a vendor identity -- unlike zai/gemini/xai,
where a matching alias really does mean "this prefix names the same
backend as the target provider."
_PROVIDER_ALIASES maps "ollama" -> "custom", so a model configured as
"ollama/glm-5.2" against a named custom provider (e.g. a LiteLLM proxy
fronting Ollama, which registers its routes as "ollama/<model>") had its
prefix stripped to bare "glm-5.2" -- a name the proxy doesn't recognize.
_strip_matching_provider_prefix now only strips a literal "custom/"
prefix when the target resolves to "custom"; an alias that merely
resolves to custom (ollama) no longer qualifies, since custom has no
vendor identity for it to redundantly repeat.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root-context rm -f "$log_dir/lock" could follow a raced directory
symlink and unlink a foreign lock outside HERMES_HOME. Clear the
stale lock via s6-setuidgid hermes alongside mkdir, and assert
victim/lock survives the swap-race test.
Without restartable log/run chown, warm volumes that keep a hermes-owned
HERMES_HOME but root-owned logs/gateways would again deny hermes mkdir.
Add a non-recursive stage2 parent heal and cover the poisoned-parent reboot path.
Root-context log/run used to pathname-chown hermes-writable log paths,
which a hermes user can race through a symlink swap via the writable
log control FIFO. Create the leaf with s6-setuidgid hermes mkdir instead;
parent logs/gateways ownership stays a stage2 boot concern (#45258).
A skin that authors a background paints both terminal defaults: OSC-11
for the backdrop, OSC-10 to re-base every default-fg token (markdown
body, borders, anything rendered without an explicit color) onto the
theme's text tone.
The OSC-10 half never fired on a limited-palette terminal.
`normalizeThemeForAnsiLightTerminal` rewrites the foreground tones to
`ansi256(N)`, and `setTerminalForeground` only accepts `#rrggbb` — so
the argument failed the hex test and the write was silently skipped.
The background moved to the skin while default-fg text stayed on the
host profile's foreground.
That split is the reported symptom: prose renders in the terminal's own
near-black while every themed token beside it renders the skin's gray,
so the base text color appears to change between adjacent words. A
resize repaints the affected cells from the screen buffer, which is why
the text "goes black" on resize and why the mix looks scattered rather
than uniform.
Resolve the tone through a new `themeToneHex` before handing it to
OSC-10: `ansi256(N)` maps through the xterm grayscale ramp and 6x6x6
cube, an authored hex passes through, and anything with no paintable
color yields '' (which correctly clears back to the terminal default).
Verified on Terminal.app + the `brooklyn` skin: `theme.color.text` is
`ansi256(238)`, previously dropped, now emitted as
`ESC]10;#444444 BEL` alongside the existing `ESC]11;#f6f9fd BEL`.
The large-session-resume E2E captured initialMockReplyCount immediately
after openSeededSession, which returns once the NEWEST turn is in the
viewport. With FIRST_PAINT_BUDGET=20 (lowered from 60 in this branch),
only the newest ~10 turns mount at first paint; the older turns
backfill in a rAF. The baseline was reading 10 instead of 27, so once
the backfill mounted the full 28 (27 seeded + 1 new), the test saw
"28 ≠ 11" and reported duplicates that were never there.
Wait for the oldest seeded turn to mount before taking the baseline.
This makes the count reflect the fully-mounted transcript regardless
of FIRST_PAINT_BUDGET, so the perf win (smaller first paint) and the
no-duplicate invariant both hold.
Refs #72504
Every fallback/dedup/skip decision asks one question — 'is this candidate
the same backend as the one that failed, along the axis that failure
invalidated?' — but it was re-implemented inline at six sites across four
subsystems, each comparing whatever string was locally convenient. Each
incident fixed one site while the others kept the bug: #22548, #70893,
#59561, #72468, #62984/#54250/#57584.
agent/backend_identity.py now owns the concept: BackendIdentity (provider /
model / base_url axes), FailureScope (MODEL / CREDENTIAL / ENDPOINT — each
failure class invalidates a different axis), and should_skip_candidate().
Unknown axes never manufacture a skip (over-skipping strands failover; a
wrong try costs one RTT).
Migrated sites:
- chat_completion_helpers.try_activate_fallback: replaces the provider+model
early-exit (the #62984 bug: ignored base_url, stranding multi-endpoint
pools) AND _fallback_entry_is_same_backend_by_base_url (deleted)
- auxiliary_client._try_configured_fallback_chain +
_try_main_agent_model_fallback: replace label/model comparisons; auth and
payment map to CREDENTIAL scope, keeping the #59561 carve-out
- hermes_cli/fallback_cmd add: primary-match + duplicate checks now identity-
aware (#54250/#57584): same provider+model on a different explicit
base_url is a pool entry, not a duplicate
_mark_provider_unhealthy stays label-keyed deliberately: its only triggers
are confirmed 402s, which ARE credential-scoped.
Owner-level tests pin each incident's semantics by number; sabotage-verified
(removing the base_url axis fails the #62984 test).
flattenSessionsWithBranches always re-sorted roots by last_active, so a
turn finishing floated background tasks over the hand-picked Pinned list
even though $pinnedSessionIds already stored drag order. preserveOrder
skips that sort for pins (and other non-date-grouped manual lists); default
recents stay recency-sorted for truthful date buckets.
Deferring the FIRST_PAINT_BUDGET -> RENDER_BUDGET backfill while a thread
streams cut a 1374ms streaming-session switch to instant, but it also
means a streaming transcript stays clipped to 60 parts for the duration
of the run. `large-session-resume` asserts the resumed transcript shows
every seeded reply exactly once, and that count is short while the budget
is held down — a genuine behavior change, not a flaky test.
The switch cost is real and still worth fixing, but the fix has to keep
the full transcript mounted (raise the budget in idle callbacks, or
virtualize) rather than withhold it. Session-switch work is happening in
a parallel effort; leaving the invariant intact for them.
Everything else in this branch is untouched: the reflow-gated RO pins
(11.5 -> 59fps drag), the structural/weight signature split, the adaptive
stream flush, the tree-split preview, and the tool-row memo boundaries.
Follow-ups on the salvaged #63359:
- _finalize_child_results carries tool_call_history on subagent_stop
(the #62011/#72403 field landed after the PR branched; the shared
pipeline must emit it for both delegate_task and plugin-launched
children). Lifecycle test updated for the new payload field.
- The lifecycle executor uses DaemonThreadPoolExecutor — a wedged or
abandoned child must never block interpreter exit at atexit-join time
(same rationale as _run_single_child's timeout executor and the
async-delegation pool).
- delegate_task's batch path keeps live-transcript wiring while routing
child construction through the shared
_build_child_preserving_parent_tools helper.
`hermes -c`/`--resume` (continue last session) resolved the globally
most-recently-used session, then cd'd into *its* recorded cwd. So running
`hermes -c` from repo A could land you in repo B's session — the session
you last touched anywhere, not the last one *here*.
Now `_resolve_last_session` scopes to the current workspace first: the git
repo root when CWD is inside a repo (so all sessions across its
subdirs/worktrees group together), else the CWD itself — matching the
`workspace_key` identity `hermes sessions list --workspace` already groups
on. It falls back to the unscoped global MRU when no session matches the
current workspace, preserving the old behaviour for fresh directories.
Adds `workspace_key` param to `SessionDB.search_sessions` and a
`_workspace_key_clause` SQL helper that mirrors `workspace_key()`: a row
matches when its `git_repo_root` equals the key, or (legacy rows without
git metadata) when its `cwd` is at or under it.
Switching to a STREAMING session took ~1.4s to settle while an idle
session settled in ~50ms. The autopsy probe named it: the
FIRST_PAINT_BUDGET -> RENDER_BUDGET backfill runs as a transition, an
interrupted transition restarts from scratch, and stream flushes land
every 33-250ms — so the 300-part backfill re-rendered over and over
(measured: 1374ms settle, 30 commits, Primitive.div x2237 for one switch).
Gate the backfill on the thread being idle. The user lands on the live
tail immediately either way; older turns backfill the moment the run
ends, and 'Show earlier' remains the manual path meanwhile.
Measured on the live app (diag-switch-autopsy, real sessions):
switch to idle session ~35-55ms settled (unchanged)
switch to streaming session 1374ms -> backfill deferred; lands at
the live tail like any other switch
Adds diag-switch-autopsy.mjs (per-switch settle/commits/top-renders) and
live-drive.mjs (status/fps/drag one-liners against the running app).
Driving HER real instance (real profile, real transcripts, streams live)
via CDP instead of synthetic tiles finally exposed the remaining stall.
The timeline on a real 60-frame sash drag:
style recalc 2736ms | script 1027ms | layout 89ms
top callsite: pin @ fallback.tsx — 927ms
Two pin-to-bottom ResizeObservers (the bounded tool window's and the
reasoning preview's) pinned on EVERY resize delivery. A sash drag changes
every message's WIDTH once per frame, so each frame ran scrollTop write ->
scrollHeight read across every tool group: a forced write-read reflow
cascade that the render counters could never see (zero React involvement).
Both pins are now height-gated off the RO entry (reflow-free): only
content GROWTH pins. Width-only deliveries return immediately.
Measured on the live app, same drag, before -> after:
fps 11.5 -> 59-60
p95 101ms -> 18ms
slow>33 60/60 -> 1/60
Also in this batch (each was verified live before the next was attempted):
- thread/list: split messageSignature into STRUCTURAL (ids/roles — keys
boundaries + row identity) and WEIGHT (part counts — budget only), and
memoize groups + row JSX. A streamed part-append re-rendered every
turn's boundary via its resetKey prop; explain() measured 540-865
wasted Block renders per drag/stream sample, now {}.
- message-render-boundary: document the structural-only resetKey contract.
- tool/fallback: memoize ToolFallback's part object + ToolEntry/ToolTitle/
ToolGlyph (151 renders each, 100% wasted, on real transcripts).
- use-message-stream: ADAPTIVE flush floor — next flush waits 3x the
measured cost of the last one (33ms floor, 250ms cap), so multi-stream
load degrades text update rate instead of input latency.
- tree-split: preview sash drags with inline flex on the two seam
wrappers, committing the store ONCE on release (fixed-zone sides get
flexBasis only, so a hidden sidebar can't leave a phantom gap).
- debug/: perf-live LoAF long-frame attribution, explain() cascade walker
with changed-hook indices, diag-real-loop/key-latency/switch-trace
probes that drive the real app over CDP.
Typing during 2 live streams: keystroke->paint p50 3.3ms, p95 18.4ms,
zero frames over 33ms. Session switch p50 ~35ms settled; the remaining
~1.3s outlier tail is streaming-session switches (React work-loop, not
style/layout) — next target.
Widen okalentiev's failed_model narrowing (#59561) to
_try_main_agent_model_fallback. The safety-net layer still skipped on a
provider-label match alone, so single-provider users whose aux compression
model and main model share one custom endpoint had ZERO fallbacks: the aux
model timing out exhausted the chain in one hop and compression aborted.
Real incident (0.19.0 debug dump): aux zai-org/glm-5.2 hung 324s and timed
out while main mindai/macaron-v1-venti on the SAME endpoint was serving
448K-token turns — the label-only skip discarded the one viable summarizer,
the session wedged over threshold, and the anti-thrash breaker tripped.
Same convention as the chain fix: model-specific failures (timeout,
connection, rate limit) pass failed_model so only the exact failed model is
skipped; provider-wide failures (auth 401 / payment 402) pass None and keep
the whole-provider skip. Both sync and async call_llm sites pass it.
Sabotage-verified: the new regression test fails on the provider-only skip.
Follow-up to the same-provider fallback fix: narrowing the configured-chain
skip to the exact failed model is only correct for model-specific failures.
Auth (401) and payment (402) errors are provider-wide — every model on the
provider shares the same broken credentials/account — so trying a sibling
model can't recover and merely burns another doomed request before the
aux task fails. Worse, returning that sibling client bypasses the
main-agent-model safety net that a provider-wide skip would have reached.
Only forward failed_model to _try_configured_fallback_chain for
model-specific failures (timeout, connection, rate limit, model-incompatible,
invalid response). Auth/payment keep failed_model=None (whole-provider skip),
preserving the pre-existing safety-net behaviour for credential/billing
failures.
Adds an integration test that a timeout forwards the failed model, and
updates the payment-error test to assert failed_model=None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_try_configured_fallback_chain skipped every fallback_chain entry whose
provider matched the one that just failed. A chain that intentionally lists
several models under the same provider (e.g. two more NVIDIA NIM models
after the primary NIM model times out) was therefore skipped wholesale,
falling straight through to the main-agent-model safety net instead of
trying the other configured models on that provider.
Add failed_model so the skip narrows to the exact (provider, model) pair
that failed. Callers that only know the provider (client-build failures,
where the whole provider is unreachable regardless of model) keep the old
provider-wide skip; the two runtime request-error call sites (call_llm,
async_call_llm) now pass the model that just failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Teknium: the caps should bound a single agent loop, not accumulate
over the whole session. Rename SessionCapConfig -> LoopCapConfig and the
config section session_caps -> loop_caps; move the counters into
reset_for_turn (invoked per turn via turn_context) so each turn starts
with a fresh budget; retune defaults 200 -> 50 (a single turn issuing 50
web searches / spawning 50 subagents is already pathological). Block
codes session_*_cap -> loop_*_cap and messages updated to drop the
/new-resets-the-budget guidance (irrelevant now that it resets per turn).
Tests flipped: the old persists-across-turn-resets assertion becomes
resets-each-turn.
Add per-session lifetime caps on web_search calls and subagent spawns
(defaults 200/200, matching Claude Code v2.1.212). Unlike the existing
per-turn tool-loop guardrails, these count over the whole session and
reset only when a fresh agent is built (/new, /clear). Hitting a cap
blocks the offending call and halts the turn cleanly.
- agent/tool_guardrails.py: SessionCapConfig + session counters on the
controller (in __init__, not reset_for_turn, so they persist across
turns). before_call() enforces caps first, independent of
hard_stop_enabled. delegate_task batches count each task.
- hermes_cli/config.py: tool_loop_guardrails.session_caps defaults.
- docs + tests (unit + E2E validated against a real AIAgent).
One relay adapter fronts N platforms on one WS, but the capability surface
(MAX_MESSAGE_LENGTH / message_len_fn) was a scalar from whichever descriptor
resolved the handshake — and the transport's read loop OVERWROTE it on every
descriptor frame (last-writer-wins). A Discord chat on a gateway whose
applied descriptor was Telegram's inherited the 4,096-char cap and over-sent
into Discord's 2,000-char API 400 (observed live: 2,543/2,641-char replies
silently lost while inbound kept working).
- ws_transport: accumulate one descriptor per platform in
_descriptors_by_platform (exposed via descriptor_for_platform); the FIRST
descriptor of a connection generation stays the session default instead of
last-writer-wins; the map resets on re-dial.
- BasePlatformAdapter: new max_message_length_for_chat /
message_len_fn_for_chat hooks defaulting to the scalar surface (native
single-platform adapters unchanged).
- RelayAdapter: overrides resolve the chat's platform from _platform_by_chat
(the same map per-frame egress uses) and look up that platform's negotiated
descriptor; falls back to the scalar for unknown chats/transports.
- stream_consumer (streaming budget, _raw_message_limit, fallback-continuation
chunking) + run.py tool-progress limit now resolve per-chat.
Tests: tests/gateway/relay/test_relay_per_platform_caps.py (7) — verified
fail-without/pass-with (all 7 fail with the fix stashed). Relay + stream
consumer suites green (213 + 223).
Follow-up to the salvaged #67409 search aliases: with kimi-k3 now in
the curated kimi-coding list (#68108), a Coding Plan key rendered TWO
rows for one model — curated 'kimi-k3' plus live-discovered bare 'k3'
(merge dedup was exact-string). Add model_alias_canonical() derived
from the same alias table and use it as the merge dedup key, so the
curated public slug wins and live-only models still surface.
Kimi Coding discovers the flagship as wire id `k3`. Picker search used
only that id, so typing "kimi" hid it next to every other kimi-* model.
Add picker-only search aliases without changing the wire id.
Port from nearai/ironclaw#5149 (the describe-first live-hardening fix in
their progressive tool disclosure work): when a model invokes a deferred
tool through the tool_call bridge without the schema-required arguments,
return the tool's parameter schema instead of dispatching blind.
Pre-fix, a blind call produced an opaque downstream failure
("[TOOL_ERROR] Tool execution failed: KeyError: 'document_id'") that
teaches the model nothing about what the tool expects — IronClaw observed
cheap models looping ~30 identical invalid calls until the iteration
budget died. Post-fix, the model repairs the call in one round-trip.
- tools/tool_search.py: new validate_deferred_call_args() — key-absence
check of schema 'required' fields only; no type checking (coerce_tool_args
already repairs types downstream); fails open on any validator error so
it can never block a legitimate dispatch.
- model_tools.py: probe after the scope gate in the bridge dispatch.
- agent/tool_executor.py: probe in both unwrap sites (concurrent +
sequential) before the underlying tool replaces the bridge; sequential
path flattens the payload to match its {"error": str} wrapping.
- tests: TestDeferredCallSchemaProbe — blind call returns schema (not
KeyError), valid/optional calls dispatch, unvalidatable tools fail open,
out-of-scope rejection unchanged.
Two sibling tests asserted the #34452 'No reply:' explainer text for
reasoning-only exhaustion. That terminal now delivers the labeled
reasoning excerpt (strictly more informative — it carries the model's
reasoning, which may contain the answer); the explainer still covers
the truly-empty case. Update the assertions to pin the new contract:
excerpt present, reasoning text included, '(empty)' never delivered.
When the empty-response ladder is fully exhausted (thinking-prefill
continuation, empty-content retries, provider fallback) and the model
produced structured reasoning but never any visible text, deliver a
clearly labeled excerpt of that reasoning instead of a bare '(empty)' —
the reasoning frequently contains the actual answer.
Delivery-only by design: raw chain-of-thought is never promoted to a
normal answer earlier in the ladder (prefill continuation still gets
first crack, retries and fallback still run), transcript persistence
semantics are untouched (the '(empty)' sentinel scaffolding keeps its
replay-safety behavior), and a truly empty exhaustion still returns the
existing terminal.
Idea credit: PR #48795 (@ligl0325) proposed falling back to
reasoning_content on empty content; this lands the safe kernel of that
idea at the one point in the ladder where it is strictly an improvement.
Port from nearai/ironclaw#6129: their sensitive-marker scrubber matched
markers as bare substrings, so tool results containing 'Secretary of the
Treasury' were scrubbed as 'secret' on replay, evicting legitimate content
and forcing the model into a re-fetch loop. Hermes' lowercase/dotted/YAML
config-key redaction patterns (_CFG_DOTTED_RE, _CFG_ANCHORED_RE,
_YAML_ASSIGN_RE) had the same false-positive class: their key classes allow
arbitrary alphanumeric affixes around the keyword, so ordinary document
text like 'Secretary: J.Smith', 'tokenizer: cl100k_base' (HF model cards),
and BibTeX 'author=Smith' got value-masked on the surfaces that run these
passes (browser snapshots, log lines, kanban summaries, CLI-echoed output).
Fix: post-match word-boundary validation of the keyword occurrence inside
the matched key. Boundaries: key edges, non-letters (_ - . digits),
camelCase transitions (clientSecret, secretKey, APIToken), plural 's'
(secrets:, tokens:). Concatenated real-world compounds keep matching via
explicit alternatives (authtoken, authkey, secretkey, accesstoken). ALL-CAPS
keys keep legacy embedded matching (MYTOKEN=...) — all-caps is almost never
prose, same rationale as _ENV_ASSIGN_RE. Same discipline the file already
applies to exact-match body/query keys (ported from ironclaw#2529) and the
deliberate 'auth' exclusion that keeps 'author:' from matching.
Port from openclaw/openclaw#110518 / #110956: some models reuse one call id
for different tool calls in a single batch (native Kimi Responses replays,
Ollama-compatible endpoints, degraded models at long context). Hermes kept
both calls but the pre-API sanitizer then dropped the later call/result pair
per id (#58327), so the second call's output silently vanished from every
replayed payload — the model never saw it and confabulated.
_uniquify_tool_call_ids renames later collisions to a deterministic <id>_d<n>
suffix at ingestion, before validation/dispatch/history build, so both pairs
survive. Composite Responses ids collide on the call half and keep their
response-item half. Deterministic suffixes preserve prompt-cache prefix
stability (no random UUIDs).
