TestThresholdTokensCap and TestLazyContextResolution landed on main after
the #38991 lazy-init base; they construct ContextCompressor under a
get_model_context_length patch and read threshold_tokens after the with
block. With deferred resolution the probe now fires lazily, so resolve
inside the mock (same pattern as the rest of the suite) and give the
lazy-resolution mock a real return_value.
Self-review found the deepcopy->shallow-copy change in
apply_anthropic_cache_control (#57046) had no test pinning the
"prompt caching is sacred / never mutate the caller's list" invariant.
The old test_returns_deep_copy only exercised the single marked message,
never an un-marked shared reference, so a regression that deep-copied too
little would pass the whole suite.
Add two tests: (1) caller list + every element left byte-identical after
the call, un-marked middle messages returned as shared references, marked
messages fresh copies, and mutating a returned marked message does not leak
upstream; (2) structural byte-equivalence vs a reference full-deepcopy
implementation across both native_anthropic modes and two TTLs.
Mutation-verified: neutering the per-message deepcopy makes test (1) fail.
The lazy-init change in #32221 deferred get_model_context_length() out of
ContextCompressor.__init__, but the "Context compressor initialized" log
(emitted only when quiet_mode=False) reads self.context_length,
self.threshold_tokens and self.tail_token_budget. Those property reads
resolve the deferred value, so the synchronous model-metadata probe still
ran inside __init__ on the interactive-CLI path — the exact blocking the
PR removed, just narrowed to the non-quiet path.
Emit the informative line once, on first context-length resolution, so
construction stays non-blocking on every path. Add a regression test
asserting no probe fires in __init__ with quiet_mode=False and that the
init log is emitted exactly once on first access.
Every LLM call built a fresh openai.OpenAI wire client (new httpx pool,
TCP+TLS handshake, measured 19.2ms p50 / 35.5ms p95 per call at ~5 calls
per tool-loop turn) and closed it when the request finished. Cache ONE
reusable wire client on the agent, keyed by the effective client kwargs:
- _create_request_openai_client hands back the cached client when the
effective kwargs are identical; any change (credential rotation,
provider failover, vision default_headers) evicts and rebuilds.
- Only a request that produced a response reports a reuse close reason
(request_complete / stream_request_complete); error unwinds report
*_error_cleanup and really close, so a retry after a request error
always builds a fresh pool.
- Cross-thread aborts poison the slot: a pool whose sockets were
shutdown(SHUT_RDWR) from a stranger thread is never reused (#29507) —
the owner-thread close discards it and the next create rebuilds. The
holder read and the abort are atomic (under the holder lock) at all
three abort sites, so a late abort can never poison the NEXT request's
checked-out client.
- Worker-side interrupt breaks close the half-read SSE stream on the
owning thread before building the partial response; a failed close
poisons the slot (otherwise each interrupt leaked one checked-out
connection until the pool hit PoolTimeout). run_codex_stream gets the
same poison-on-close-failure handling.
- Single checked-out slot (in_use): a concurrent call gets an untracked
client with the old per-request lifecycle.
- release_clients() / close() really close the cached client when idle;
if a worker has it checked out they abort the sockets and detach the
slot, deferring the FD release to the worker's own close.
- MoA facade and Mock passthroughs never enter the cache; max_retries=0
is preserved on all request clients.
Follow-up to PR #64171. The _TOKEN_DELTA_* classification must exactly
cover update_token_counts' keyword surface: an unclassified kwarg is
silently kept only from the first delta of a merged run. Introspect the
live signature and fail with a pointed message when a future kwarg is
added without classification (or a classified field is removed).
Follow-up to PR #64171. The writer loop and flush_token_counts'
caller-drain both set _token_writer_busy BEFORE popping the queue —
that ordering is what makes flush's lock-free fast path (reads queue
then busy, no cond held) sound. _stop_token_writer's leftover drain did
it backwards (clear queue, then set busy), leaving a few-bytecode window
where a concurrent flush could observe 'empty and idle' and return True
with the popped batch still unapplied. Shutdown-only staleness, no data
loss — but the protocol now matches at all three drain sites.
Test: concurrent flush during a stop-drain mid-apply must time out
(False), never report drained.
Follow-up to PR #64171. Two writer-death hardening gaps:
- queue_token_counts only spawned the writer when the thread object was
None, so a writer that died from an unexpected exception could never be
replaced: deltas piled up on the uncapped deque until a reader's flush
drained them synchronously. Respawn on 'not thread.is_alive()' instead.
(atexit re-registration on respawn is safe: unregister removes all equal
bound-method registrations and the drain hook is idempotent.)
- _coalesce_token_deltas ran outside the per-delta try/except in
_apply_token_batch, so a merge bug (e.g. an unclassified future kwarg
summing None + 0) would escape and kill the writer thread. Wrap it and
fall back to applying the raw batch — coalescing is an optimization,
never load-bearing.
Tests: coalesce-failure fallback + dead-writer respawn.
Every API call in the tool loop persisted its token/cost delta by
calling SessionDB.update_token_counts() synchronously on the turn
thread — a BEGIN IMMEDIATE sessions UPDATE plus a session_model_usage
upsert, measured in production at p50 3.3ms / p95 70.4ms per call and
up to 299ms against a cold multi-GB state.db. The tool loop stalls for
that long between calls, N times per multi-tool turn.
SessionDB gains queue_token_counts(): same signature and semantics as
update_token_counts(), but the critical path is a deque append plus a
condvar notify. A lazily started daemon thread applies deltas in
enqueue order through the existing update_token_counts ->
_execute_write path, so the established self._lock / BEGIN IMMEDIATE /
jitter-retry discipline is unchanged. When a backlog forms, adjacent
same-route incremental deltas coalesce into one UPDATE: token and
api-call fields sum, cost fields sum None-preservingly (an all-None
run stays None so COALESCE keeps the stored value), and absolute=True
deltas never merge and act as ordering barriers. Route equality is
required for a merge because those fields feed COALESCE backfill, the
last-non-None-wins status fields, and the per-model usage attribution
key — a merged apply is row-equivalent to sequential applies.
Correctness and durability:
- flush_token_counts() gives read-your-writes to token/cost readers
(get_session, list_sessions_rich, _get_session_rich_row,
list_gateway_sessions, InsightsEngine.generate) — a plain attribute
check when nothing is queued. The writer sets its busy flag before
popping the queue so the lock-free fast path can never miss an
in-flight batch.
- update_session_model / update_session_billing_route /
update_session_meta write the sessions row synchronously, bypassing
the queue, so they flush it first: a still-queued first-of-session
delta carries the pre-switch route, and applying it after the switch
UPDATE would trip the first_accounted_route branch (api_call_count
== 0 plus a route mismatch) and resurrect the old model/provider.
- AIAgent._persist_session flushes at turn finalize and every
error-exit persist point; close() stops and drains the writer before
the WAL checkpoint; an atexit hook (registered on first enqueue,
unregistered on close so closed instances are not pinned until
interpreter exit) drains at shutdown. Worst-case crash loss is the
in-flight call's delta — the same window as the old inline write.
- A flush trusts a live stop-flagged writer (its loop drains before
exiting) and only drains on the caller's thread when the writer is
dead or never started, claiming the same busy flag so concurrent
flushes wait instead of racing an in-flight batch.
- After close() has stopped the writer, queue_token_counts applies the
delta inline instead of parking it on a queue nothing will drain; a
closed-connection failure then raises at the call site, which
already guards for it, exactly like the old synchronous path.
- Writer apply failures are logged and never raise into a turn; the
writer thread survives and keeps applying.
Call sites switched to the queue: the per-call site in
agent/conversation_loop.py and both codex app-server sites in
agent/codex_runtime.py. In-memory per-turn counters
(agent.session_estimated_cost_usd etc.) stay synchronous, so live turn
displays never see the queue.
Tests: tests/agent/test_async_token_accounting.py (19 tests: enqueue
ordering, absolute-as-barrier, backlog coalescing with exact sums,
coalesced-vs-sequential row equivalence, merge unit rules, None-cost
preservation, read-your-writes, flush vs stop-flagged/concurrent
drains, inline apply after writer stop, close/atexit durability,
_persist_session drain, writer failure isolation);
tests/run_agent/test_token_persistence_non_cli.py updated to the
queue_token_counts contract.
The auto-continue recovery note was typed only after run_conversation
returned, so its row sat untyped for the whole turn — and permanently
when the continuation was itself killed, which is the case it exists
for. persist_user_display_kind stamps the type on the live message
before the crash persist writes it, in the same insert as the content.
The flush also carries display_metadata through, which it was dropping.
Expanding an @file:/@folder: reference deleted the token from the message
it was typed in, leaving a hole in the sentence and no anchor for clients
to chip. The attached context block still names each ref, so nothing is
lost by leaving the token where the user put it.
Add distinct, greppable log lines at both fix sites so the condition is
observable in the field instead of only inferable from the absence of the old
'Cannot compress further' spiral:
- chat_completion_helpers: warn when an empty partial-stream stub is replaced
with the placeholder (0 chars recovered, no tool call).
- error_classifier: warn when a malformed-body 400 is classified as
format_error rather than context overflow, with num_messages/approx_tokens.
Extend the litellm-shape classifier test to assert the warning is emitted.
A stream that dies after delivering deltas but with 0 recovered chars (and
no captured tool call) produced a content-less assistant stub. That empty
non-final message is invalid for the Anthropic schema, so every later request
400s with 'all messages must have non-empty content' (INVALID_REQUEST_BODY).
On a large session the 400 was misclassified as context overflow, dropping the
loop into a compression spiral that ends in 'Cannot compress further' — a
misleading context-size error on a session nowhere near its limit.
Root cause: substitute a minimal '[response interrupted]' placeholder so the
stub is never empty.
Misclassification: add _INVALID_MESSAGE_BODY_PATTERNS (checked before the
context-overflow heuristic) to classify these as non-retryable format_error,
and teach the body extractors the litellm/Bedrock errorMessage/errorCode/
errorArgs shape so descriptive proxy errors are not mistaken for generic ones.
Adds RED-verified regression tests for the stub path and three classifier tests
(including a guard that real overflows still compress).
_peel_moa_guidance hand-implemented the inverse of moa_loop's
_attach_reference_guidance from a different module — a drifting separator
or shape would make the peel silently no-op and put the last cache
breakpoint on the turn-varying guidance block (the #72626 bug class).
Move the inverse into moa_loop.peel_reference_guidance directly adjacent
to the attach, keep a thin wrapper in conversation_loop, and pin the
contract with a round-trip test over all three attach shapes.
Also fix the empty-list residue: peeling a guidance-only content part now
drops the whole message (mirroring the appended-user-message shape)
instead of leaving an empty-content user turn behind.
guidance=None is a real prepared shape (all references failed / silent
degraded policy builds prepared_request without attaching guidance), and
the MoA facade sends prepared['messages'] — not api_kwargs['messages'].
Gating the rebase on 'and guidance' left the stale decoration in the
prepared object for the no-guidance MoA sub-path, so #72626 persisted
there. rebase_prepared_request already handles falsy guidance (copies
messages, skips the attach).
The static-prefix reconstruction pattern (build_system_prompt_parts ->
['stable'] -> startswith gate -> fail-open) existed in three copies:
session restore (conversation_loop), compression keep-prompt path
(conversation_compression), and the new failover redecoration helper.
Hoist it into agent/system_prompt.reconstruct_static_prefix and call it
from all three sites.
Also memoize failed rebuilds per stored prompt (_static_rebuild_failed_for):
the redecoration chokepoint runs at the top of every retry attempt, and a
persistent stable-tier mismatch (restored session whose SOUL.md/skills
changed since save) would otherwise re-run the full prompt build — SOUL.md,
context files, memory I/O — on every attempt of every API call for the
life of the session. A legitimately changed stored prompt retries once.
strip_anthropic_cache_control flattened ANY pure-text multi-part content
list with a separator-less join. Decoration only ever produces a single
text part or the 2-part [static, volatile] system split; organic
multi-part text (merged user turns, imported transcripts) got word-jammed
and parts carrying extra keys (citations) were silently dropped — on the
common no-failover path, since redecoration runs on every attempt.
Restrict the flatten to the exact decoration-produced shapes and make
marker removal copy-on-write on part dicts (the per-call message copy is
shallow, so parts alias the persistent history).
Add strip_anthropic_cache_control coverage and policy-change cases
(cache-off→on, on→off, native→envelope, MoA guidance outside marker)
that TestSyncFailoverPreservesCacheDecoration did not exercise.
Three code-reuse fixes applied during salvage:
1. Reuse _relative_time from hermes_cli/main.py instead of duplicating
the relative-time formatting logic in hermes_cli/status.py.
2. Extract _stamp_hygiene_compression_provenance helper in gateway/run.py
to deduplicate the two nearly-identical try/except blocks that stamp
compression timeout/abort provenance in the hygiene path.
3. Add ContextCompressor.record_timeout_failure() method and use it from
the in-agent compress_context timeout callback instead of re-implementing
the (60, 300, 900) cooldown ladder inline. The existing summary-LLM
exception handler already has this ladder — now both paths share one
method.
Three mechanisms to detect and notify when gateway sessions stall silently:
1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
and hermes status show progress during long turns without new message rows.
2. Stall watchdog: when a busy session has pending inbound and the shared
activity clock is idle past agent.session_stall_timeout (default 300),
log a WARNING and notify the user once to try /new. Notify-only; does
not kill the turn.
3. Compaction timeout: fenceless compress_context callers get a progress-aware
host budget (compression.context_timeout_seconds default 120 idle,
compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
cancel via commit fence, skip compaction without dropping messages, and
continue the turn.
Closes#72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).
Cherry-picked from PR #72424 by @fangliquanflq.
claude-fable-5 is a Mythos-class reasoning model (1M context, 128K output,
adaptive thinking per anthropic_adapter.py) but was missing from the
_REASONING_STALE_TIMEOUT_FLOORS table. Without a floor entry it got the
default 180s stale timeout (300s with context scaling), which is too short
for fable-5's thinking phase on large contexts.
Each stale kill bumped the cross-turn circuit breaker streak; after 5
consecutive kills _check_stale_giveup() fired immediately (elapsed: 0.00s),
aborting all calls with "Provider has been unresponsive for 5 consecutive
stale attempts." Users with 191K-token contexts hit this reliably.
Add ("claude-fable", 600) — deep-reasoning tier alongside o1/deepseek-r1/
nemotron-3-ultra. The claude-fable slug matches claude-fable-5 and future
variants via the existing right-anchor regex.
Update _pre_msg_count after adopting the durable transcript so the
post-compression log reflects the correct pre-adoption message count.
Also use the existing _live_child_id() helper in the updated test
instead of hand-rolled child-id extraction.
Follow-up to #72631.
Busy sessions (memory review / shared session writers) kept outrunning the
in-memory snapshot, so rotation-mode compress aborted every attempt with
"changed before lease acquisition" and surfaced as a fake "No changes from
compression". Adopt the durable transcript and continue compressing instead
of returning the stale snapshot unchanged.
Every fallback/dedup/skip decision asks one question — 'is this candidate
the same backend as the one that failed, along the axis that failure
invalidated?' — but it was re-implemented inline at six sites across four
subsystems, each comparing whatever string was locally convenient. Each
incident fixed one site while the others kept the bug: #22548, #70893,
#59561, #72468, #62984/#54250/#57584.
agent/backend_identity.py now owns the concept: BackendIdentity (provider /
model / base_url axes), FailureScope (MODEL / CREDENTIAL / ENDPOINT — each
failure class invalidates a different axis), and should_skip_candidate().
Unknown axes never manufacture a skip (over-skipping strands failover; a
wrong try costs one RTT).
Migrated sites:
- chat_completion_helpers.try_activate_fallback: replaces the provider+model
early-exit (the #62984 bug: ignored base_url, stranding multi-endpoint
pools) AND _fallback_entry_is_same_backend_by_base_url (deleted)
- auxiliary_client._try_configured_fallback_chain +
_try_main_agent_model_fallback: replace label/model comparisons; auth and
payment map to CREDENTIAL scope, keeping the #59561 carve-out
- hermes_cli/fallback_cmd add: primary-match + duplicate checks now identity-
aware (#54250/#57584): same provider+model on a different explicit
base_url is a pool entry, not a duplicate
_mark_provider_unhealthy stays label-keyed deliberately: its only triggers
are confirmed 402s, which ARE credential-scoped.
Owner-level tests pin each incident's semantics by number; sabotage-verified
(removing the base_url axis fails the #62984 test).
Follow-ups on the salvaged #63359:
- _finalize_child_results carries tool_call_history on subagent_stop
(the #62011/#72403 field landed after the PR branched; the shared
pipeline must emit it for both delegate_task and plugin-launched
children). Lifecycle test updated for the new payload field.
- The lifecycle executor uses DaemonThreadPoolExecutor — a wedged or
abandoned child must never block interpreter exit at atexit-join time
(same rationale as _run_single_child's timeout executor and the
async-delegation pool).
- delegate_task's batch path keeps live-transcript wiring while routing
child construction through the shared
_build_child_preserving_parent_tools helper.
Widen okalentiev's failed_model narrowing (#59561) to
_try_main_agent_model_fallback. The safety-net layer still skipped on a
provider-label match alone, so single-provider users whose aux compression
model and main model share one custom endpoint had ZERO fallbacks: the aux
model timing out exhausted the chain in one hop and compression aborted.
Real incident (0.19.0 debug dump): aux zai-org/glm-5.2 hung 324s and timed
out while main mindai/macaron-v1-venti on the SAME endpoint was serving
448K-token turns — the label-only skip discarded the one viable summarizer,
the session wedged over threshold, and the anti-thrash breaker tripped.
Same convention as the chain fix: model-specific failures (timeout,
connection, rate limit) pass failed_model so only the exact failed model is
skipped; provider-wide failures (auth 401 / payment 402) pass None and keep
the whole-provider skip. Both sync and async call_llm sites pass it.
Sabotage-verified: the new regression test fails on the provider-only skip.
Follow-up to the same-provider fallback fix: narrowing the configured-chain
skip to the exact failed model is only correct for model-specific failures.
Auth (401) and payment (402) errors are provider-wide — every model on the
provider shares the same broken credentials/account — so trying a sibling
model can't recover and merely burns another doomed request before the
aux task fails. Worse, returning that sibling client bypasses the
main-agent-model safety net that a provider-wide skip would have reached.
Only forward failed_model to _try_configured_fallback_chain for
model-specific failures (timeout, connection, rate limit, model-incompatible,
invalid response). Auth/payment keep failed_model=None (whole-provider skip),
preserving the pre-existing safety-net behaviour for credential/billing
failures.
Adds an integration test that a timeout forwards the failed model, and
updates the payment-error test to assert failed_model=None.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_try_configured_fallback_chain skipped every fallback_chain entry whose
provider matched the one that just failed. A chain that intentionally lists
several models under the same provider (e.g. two more NVIDIA NIM models
after the primary NIM model times out) was therefore skipped wholesale,
falling straight through to the main-agent-model safety net instead of
trying the other configured models on that provider.
Add failed_model so the skip narrows to the exact (provider, model) pair
that failed. Callers that only know the provider (client-build failures,
where the whole provider is unreachable regardless of model) keep the old
provider-wide skip; the two runtime request-error call sites (call_llm,
async_call_llm) now pass the model that just failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per Teknium: the caps should bound a single agent loop, not accumulate
over the whole session. Rename SessionCapConfig -> LoopCapConfig and the
config section session_caps -> loop_caps; move the counters into
reset_for_turn (invoked per turn via turn_context) so each turn starts
with a fresh budget; retune defaults 200 -> 50 (a single turn issuing 50
web searches / spawning 50 subagents is already pathological). Block
codes session_*_cap -> loop_*_cap and messages updated to drop the
/new-resets-the-budget guidance (irrelevant now that it resets per turn).
Tests flipped: the old persists-across-turn-resets assertion becomes
resets-each-turn.
Add per-session lifetime caps on web_search calls and subagent spawns
(defaults 200/200, matching Claude Code v2.1.212). Unlike the existing
per-turn tool-loop guardrails, these count over the whole session and
reset only when a fresh agent is built (/new, /clear). Hitting a cap
blocks the offending call and halts the turn cleanly.
- agent/tool_guardrails.py: SessionCapConfig + session counters on the
controller (in __init__, not reset_for_turn, so they persist across
turns). before_call() enforces caps first, independent of
hard_stop_enabled. delegate_task batches count each task.
- hermes_cli/config.py: tool_loop_guardrails.session_caps defaults.
- docs + tests (unit + E2E validated against a real AIAgent).
Port from nearai/ironclaw#6129: their sensitive-marker scrubber matched
markers as bare substrings, so tool results containing 'Secretary of the
Treasury' were scrubbed as 'secret' on replay, evicting legitimate content
and forcing the model into a re-fetch loop. Hermes' lowercase/dotted/YAML
config-key redaction patterns (_CFG_DOTTED_RE, _CFG_ANCHORED_RE,
_YAML_ASSIGN_RE) had the same false-positive class: their key classes allow
arbitrary alphanumeric affixes around the keyword, so ordinary document
text like 'Secretary: J.Smith', 'tokenizer: cl100k_base' (HF model cards),
and BibTeX 'author=Smith' got value-masked on the surfaces that run these
passes (browser snapshots, log lines, kanban summaries, CLI-echoed output).
Fix: post-match word-boundary validation of the keyword occurrence inside
the matched key. Boundaries: key edges, non-letters (_ - . digits),
camelCase transitions (clientSecret, secretKey, APIToken), plural 's'
(secrets:, tokens:). Concatenated real-world compounds keep matching via
explicit alternatives (authtoken, authkey, secretkey, accesstoken). ALL-CAPS
keys keep legacy embedded matching (MYTOKEN=...) — all-caps is almost never
prose, same rationale as _ENV_ASSIGN_RE. Same discipline the file already
applies to exact-match body/query keys (ported from ironclaw#2529) and the
deliberate 'auth' exclusion that keeps 'author:' from matching.
Port from Kilo-Org/kilocode#12162.
Google began rejecting unrestricted legacy 'Standard' Google Cloud API keys
on the Gemini API on June 19, 2026 (all Standard keys stop working in
September 2026). The rejection is a 401 whose message misleadingly tells the
user to supply an OAuth 2 access token. gemini_http_error() now appends
actionable guidance (check key type in AI Studio, mint a new Gemini API key,
temporary restriction bridge) on that narrow shape — matched via
google.rpc.ErrorInfo reason ACCESS_TOKEN_TYPE_UNSUPPORTED or the
'Expected OAuth 2 access token' signature. Plain invalid keys
(API_KEY_INVALID) keep their existing message.
Also fixes a latent sibling gap: _summarize_api_error() preferred re-extracting
the raw response body for errors carrying .response, which stripped adapter-
composed guidance (this one AND the existing free-tier 429 guidance) from the
user-facing summary. GeminiAPIError now surfaces its composed message.
Port from anomalyco/opencode#37848 (+ dev-branch twin #37840): expand
context-overflow patterns and guard against rate-limit messages that
mention tokens.
- 'Throttling error: Too many tokens, please wait before trying again.'
(AWS Bedrock / proxy shape) classified as context_overflow and routed a
healthy session into compression on every throttle. Added 'throttling'
to _RATE_LIMIT_PATTERNS, which the message-only path checks BEFORE the
overflow list.
- 'Input length N exceeds the maximum allowed input length of M tokens.'
(Together/Fireworks shape) fell through to unknown — no compression
recovery. Added 'maximum allowed input length' to overflow patterns.
- 'request_too_large' / 'Request exceeds the maximum size' (Anthropic 413
type re-wrapped without a status code by aggregators/proxies) fell
through to unknown. Added to _PAYLOAD_TOO_LARGE_PATTERNS.
All three shapes proven live on main before the fix; 265 classifier +
bedrock tests and 238 sibling rate-guard/compression tests pass.
