Map Codex Responses status=incomplete with incomplete_details.reason=content_filter to finish_reason=content_filter so the existing refusal/fallback path runs instead of burning incomplete continuation attempts.
Follow-up on @AlexFucuson9's cherry-picked multimodal flattening:
- http(s) image parts render as '[image: <url>]' so the summary keeps a
referenceable handle after compaction (base64 data: URLs still
collapse to '[image]' — no reusable reference, and leaking them is
the bug being fixed)
- unknown part types (document, future shapes) render as '[<type>]'
instead of being silently dropped
- test docstrings corrected: the original crash premise is stale
(redact_sensitive_text grew str() coercion in #52147); the live bug
is repr-noise + base64 leakage into the summarizer input
msg.get('content') can return a list of parts for multimodal messages
(containing text, images, etc.). The old code passed this list directly
to redact_sensitive_text(text: str), which raised AttributeError on
list.replace(), causing context compression to fail entirely for any
session with attached images.
Fix: detect list content and extract text parts before redacting.
Image parts are replaced with '[image]' placeholder.
Follow-up on the salvaged fallback: silent degradation is how #64333 went
unnoticed (jobs dead on arrival, only errors.log knew). Warn once with a
resync hint, and cover both the skewed and healthy paths with tests.
Two related bugs in _AnthropicCompletionsAdapter.create() in
agent/auxiliary_client.py silently discard caller-supplied
reasoning_config and extra_body on the Anthropic-Messages
auxiliary-protocol path:
* Bug A: reasoning_config=None was hardcoded at L1000, so the
reasoning_config parameter on build_anthropic_kwargs was
unreachable for any auxiliary task. The main agent path
(agent/transports/anthropic.py) already reads
reasoning_config from caller params; this PR aligns the
auxiliary adapter with the same pattern.
* Bug B: create(**kwargs) accepts an OpenAI-style kwargs
payload from the caller but only forwards a hand-picked
subset to self._client.messages.create(). Any caller-supplied
extra_body (e.g. thinking control, metadata, service_tier,
vendor-specific fields) was dropped on the floor. The
codex/responses transport in the same file already merges
extra_body; the Anthropic branch is the gap.
This unlocks the caller-supplied extra_body path so auxiliary
callers can set per-vendor request fields (including
thinking: {type: "disabled"} for Anthropic-compatible vendors
that require an explicit disable on the wire), and lets the
reasoning_config kwarg flow into build_anthropic_kwargs like the
main agent does. Both changes are backward-compatible for
callers that don't pass the affected kwargs.
Affected providers (all routed through _AnthropicCompletionsAdapter
via _maybe_wrap_anthropic): anthropic (native), minimax /
minimax-cn, kimi-coding / kimi-coding-cn, z.ai / GLM, and any
custom /anthropic-suffixed endpoint. See PR description for
related issues (#35566, #7209, #16533, #32813, #29248).
With the TTFB watchdog now scaled (not disabled) for large requests on
main, the hard ceiling is a backstop against mid-stream wedges, not the
primary stall detector. The original 600s default clamped BELOW the
intentional 1200s stale floor for >100k-token requests, partially
reverting the floor that keeps healthy gateway-scale payloads alive.
Raise the default to 1500s so the ceiling only catches unbounded
growth, never healthy slow turns.
A large Codex request (estimated context >= 10k tokens) disables the no-byte
TTFB watchdog on purpose, and openai_codex_stale_timeout_floor *raises* the
stale timeout (up to 1200s at >100k tokens) so healthy gateway-scale payloads
aren't aborted mid-prefill. When the backend genuinely stalls — no first byte
AND no events, exactly the #64507 symptom — the request is only reclaimed at
that high stale floor, so the session can hang for 13+ minutes with an idle
slash_worker and no ended_at/end_reason while Desktop still shows it as active.
Add a flat, finite hard ceiling on total openai-codex request time that always
applies (min() of the computed stale timeout and the ceiling) regardless of the
TTFB-disable / stale-floor interaction. A stalled large request is now killed
at the ceiling and the retry loop / visible failure path takes over instead of
hanging indefinitely. Tunable via HERMES_CODEX_HARD_TIMEOUT_SECONDS (default
600s; 0 disables the ceiling to restore pre-fix behavior).
Closes#64507
_summarize_tool_result() built the web_extract summary straight from the
first `urls` entry. When web_search results are forwarded into web_extract
(a common chain), that entry is a dict ({"url"/"href": ...}) rather than a
URL string. With 2+ URLs the `url_desc += " (+N more)"` step then raised
`TypeError: unsupported operand type(s) for +=: 'dict' and 'str'`, aborting
the pre-compression pruning pass; with a single URL the raw dict repr
leaked into the summary text.
Unwrap the URL from dict entries before use — mirroring the existing
web_extract handling in agent/display.py (`_display_url`),
tools/web_tools.py (`_web_extract_url`) and acp_adapter/tools.py
(`build_tool_title`) — so `url_desc` is always a string and the
concatenation is always str + str.
Adds regression tests covering multiple/single dict URLs, the href key,
malformed dicts, and the plain-string path.
Community reports of GPT 'infinitely thinking' are usually a slow or
overloaded provider plus silent retry machinery: the CLI/TUI/Desktop
spinner shows a generic 'cogitating...' verb for the whole wait and the
gateway heartbeat says only 'Working — N min'.
Add AIAgent._emit_wait_notice(): rewrites the live spinner/status line
(thinking_callback → CLI prompt_toolkit widget, thinking.delta → TUI +
Desktop) and updates the activity tracker (included in the gateway's
'⏳ Working — N min' heartbeat). Wired at the four wait points:
- non-streaming wait loop: after 30s with no response, the line becomes
'⏳ waiting on <model> — Ns with no response yet (provider may be slow
or overloaded; auto-reconnect at Ns)'
- streaming wait loop: same explanation after 30s with no chunks,
including the long-thinking case
- TTFB / stale-stream kills: '⚠ no response from provider in Ns —
reconnecting...'
- Codex continuation retries: '↻ model returned reasoning with no final
answer — asking it to continue (n/3)' instead of silence
Notices are fail-open (display errors never break the wait loop) and
gateway sessions without a display callback still get the improved
activity description.
Follow-up to the salvaged #63690: when the interim assistant message is
too empty to append (no content and no reasoning of any kind), the last
message in history is still the prior user/tool turn — appending the
user-role nudge there would create a user→user or tool→user sequence
that strict providers reject. Only append the nudge when the last
message is an assistant turn. Also add IpastorSan to AUTHOR_MAP.
grok-4.x on the xAI /v1/responses surface sometimes ends a turn with only
reasoning items — no message output item, no tool calls — and those
reasoning items carry no encrypted_content. Two compounding problems:
1. The model occasionally emits its final answer INSIDE the reasoning
channel, delimited by grok's internal "<response>" tag. The answer
exists but is classified reasoning-only → finish_reason=incomplete.
2. An interim assistant message holding only plain-text reasoning replays
as nothing in _chat_messages_to_responses_input, so every continuation
request is byte-identical to the one that just failed. The model
deterministically repeats the reasoning-only response until the retry
budget is exhausted and the turn dies with "Codex response remained
incomplete after 3 continuation attempts".
Fixes:
- _normalize_codex_response (xai_responses only): salvage the
<response>-delimited tail from the reasoning text and promote it to
assistant content; the untagged prefix stays as thinking text.
- Codex-incomplete continuation path: when the interim message has
nothing the input converter will replay (no content, no encrypted
reasoning items, no message items), append a user-role nudge so the
retry actually differs and explicitly asks for the final answer /
pending tool call. Mirrors the existing _get_continuation_prompt
pattern used for length truncation.
Observed live with grok-4.20 on xai-oauth (2026-07-13); sibling of the
grok-composer web_search incomplete-loop fix in transports/codex.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to the salvaged #64449: the status-trusting branch flipped
github_responses to 'stop' alongside unknown relays. Copilot fronts the
same OpenAI model family as codex_backend and shows the same
reasoning-only 'still thinking' degeneration, so it stays on the
continuation path. Only unrecognized (other:*) backends trust
response.status='completed' as terminal.
The aggregator is MoA's acting model, but the main loop's reasoning
gates key off the virtual moa://local identity and never fire — so with
no per-slot reasoning_effort the aggregator silently ran at the backend
default, ignoring the user's reasoning config entirely (#64187).
New _aggregator_reasoning_config(): slot value > full acting-model
resolution via the shared chokepoint (agent.reasoning_overrides for the
slot's model > global agent.reasoning_effort; YAML False stays
'disabled'). Applied to both aggregator call sites (acting turn +
one-shot /moa synthesis).
Reference advisors intentionally keep slot-or-default: inheriting a
global xhigh into every advisor fan-out would silently multiply cost.
Fixes#64187.
Consolidation follow-up on top of @liuwei666888's compressor fix (#52431)
and @Frowtek's ACP render guard (#62588) — completes the class fix across
every display-layer consumer of tool-call arguments:
- agent/context_compressor.py: wrap _summarize_tool_result in a
never-raises backstop (same pattern as get_cute_tool_message and the
ACP guard). The per-branch _str_arg coercions keep summaries
informative; the wrapper guarantees compression can never crash-loop
on a summary branch we didn't anticipate. Also reject non-dict parsed
args (a JSON list/scalar would crash every args.get call).
- agent/display.py: build_tool_preview's process branch sliced
session_id/data without coercion — crashed build_tool_preview and
build_tool_label on the live tool-progress callback (probed live:
TypeError 'int' object is not subscriptable). Coerce like the
sibling branches.
- tests: backstop fuzz matrix (16 tools x 6 hostile value shapes),
fallback-shape contracts, display process-preview regressions,
non-dict args guard.
When LLMs return non-string parameter values (e.g. bool, int) in tool call
arguments, _summarize_tool_result() crashes with TypeError because it calls
len(), .count(), or slicing directly on args.get() return values.
This causes an infinite crash loop in the TUI — context compression
triggers on session resume, which crashes, which restarts, which triggers
compression again.
Add _str_arg() helper that coerces any value to str, and use it in all 5
vulnerable call sites:
- terminal: len(cmd)
- write_file: content.count()
- delegate_task: len(goal)
- execute_code: len(code)
- vision_analyze: question[:50]
The chat_completions path raises EmptyStreamError when a stream yields
no chunks and no finish_reason (#64420). _call_anthropic() had no
equivalent, and the eventless failure surfaces differently by client:
- Real Anthropic SDK: no message_start means no final-message snapshot,
so get_final_message() raises a bare AssertionError — not in the
retry loop's transient set, so it burned no retries and surfaced raw.
- OpenAI-compat shims: may fabricate a contentless Message with no
stop_reason (or return None), flowing out as a 'successful' empty turn.
Track whether any stream event arrived and normalize both shapes to
EmptyStreamError, giving the anthropic path the same transient retry
budget in the shared _call() retry loop. A real completed response
always carries a stop_reason, so the guard cannot fire on legitimate
turns (including eventless mocks with stop_reason='end_turn').
Follow-up to #64420.
An exhausted EmptyStreamError previously reported 'Connection to provider
failed' — misleading, since the connection succeeded (stream opened) and
the provider simply sent nothing. Add a dedicated third branch so users
debugging a misconfigured endpoint aren't sent chasing network issues.
Follow-up to #64420.
Follow-up to @michaelHMK's cherry-picked fix for #50892:
- agent/tool_dispatch_helpers.py: guard the segment planner's except-path
debug log against non-string arguments (the planner replaced the old
_should_parallelize_tool_batch body after #64460 and inherited the same
latent arguments[:200] slice).
- tests: the PR's executor-coercion hunks were dropped as redundant —
both executors now route through _parse_tool_arguments, which already
rejects null/non-object args with a structured error result instead of
coercing to {} (the 'we do not repair bad model outputs' contract).
Reworked the salvaged tests to pin the current behavior: None args are
rejected without dispatch, valid siblings still run, the planner treats
them as a barrier without raising, and the mainline run_conversation
path (which normalizes None to '{}' before dispatch) stays crash-free
under verbose logging.
Review nit from verification: the comment claimed newer FastMCP accepts a
JSON schema kwarg — the installed SDK's add_tool has no such parameter; the
synthesized __signature__ is what drives schema generation on both paths.
Fixes#64025
The hermes-tools MCP server was fetching each tool's JSON schema
into params_schema but never passing it to FastMCP's add_tool(),
so all published tools had an empty **kwargs signature. MCP clients
couldn't see parameters and arguments were dropped at dispatch.
This fix:
- Adds _signature_from_schema() to convert JSON schemas to Python
function signatures with type annotations
- Attaches the generated signature/annotations to each handler closure
so FastMCP introspects the real parameter structure
- Filters out None values before dispatch to avoid forwarding unset
optional parameters
Impact: web_search, browser automation, vision, and other Hermes tools
are now properly callable from the codex_app_server runtime.
The upstream migration to inference.nousresearch.com broke routing for
inference-api.nousresearch.com. Restore the correct base_url and drop
the stale alias match in _is_nous_inference_route().
Fixes#60715
The just-merged auxiliary.<task>.reasoning_effort shorthand applied
ensemble-wide to MoA (one value for every advisor) — wrong granularity.
Per-slot preset config supersedes it:
moa:
presets:
deep_review:
reference_models:
- {provider: ..., model: ..., reasoning_effort: low}
- {provider: ..., model: ..., reasoning_effort: xhigh}
aggregator:
{provider: ..., model: ..., reasoning_effort: high}
- Remove reasoning_effort from the moa_reference/moa_aggregator
DEFAULT_CONFIG blocks; _get_task_extra_body now warns-and-ignores the
key on MoA tasks, pointing at the preset config
- Guard tests: MoA aux blocks must not regrow the key; task-level value
is rejected with the pointer warning
- Docs: configuration.md notes the MoA exception and links the MoA page
Every auxiliary task block (vision, web_extract, compression,
title_generation, curator, background_review, moa_reference, ...) now
accepts a reasoning_effort shorthand:
auxiliary:
compression:
reasoning_effort: low
vision:
reasoning_effort: none
_get_task_extra_body() folds it into extra_body.reasoning, which every
auxiliary wire already translates: chat.completions passes it through,
the Codex Responses adapter maps it to top-level reasoning/include, and
the Anthropic auxiliary adapter now forwards it into
build_anthropic_kwargs(reasoning_config=...) (previously hardcoded None).
An explicit extra_body.reasoning on the same task wins over the
shorthand. Invalid levels are ignored with a warning. Empty string
(the shipped default) is a no-op — zero behavior change.
Config: reasoning_effort added to all 16 auxiliary task blocks in
DEFAULT_CONFIG (no version bump — deep-merge handles new keys).
A model response containing several parallel-safe reads plus one unsafe
tool used to lose ALL concurrency: _should_parallelize_tool_batch was
all-or-nothing, so a single barrier call (terminal, clarify, unknown
tool, malformed args) forced the entire batch onto the sequential path.
_plan_tool_batch_segments now splits the batch into ordered segments:
maximal contiguous runs of parallel-safe calls execute on the existing
concurrent path, barrier calls on the sequential path, strictly in the
model's emission order. Invariants preserved:
- one tool result per call, appended in emission order (segments are
contiguous, so no result reordering across a barrier)
- side-effect boundaries: no call starts before an earlier barrier ends
- overlapping file targets split into separate ordered parallel runs
- turn-end budget enforcement + /steer injection run exactly once per
batch (segment executors run with finalize=False; the segmented
dispatcher owns the whole-turn finalize)
- interrupt during segment k drains segments k+1..n with cancelled
results, keeping one result per tool_call_id
Homogeneous batches keep their original single-path dispatch (zero
behavior delta); _should_parallelize_tool_batch remains as a thin view
over the planner for existing callers and tests.
Collapse the six per-surface copies of override-then-global resolution
(CLI startup, gateway, TUI, cron, /model switch, fallback activation)
onto one shared resolve_reasoning_config() in hermes_constants.
Also fixes the gateway resolving reasoning against config model.default
instead of the session's effective model: after a session-only /model
switch, the switched model's override now applies (gateway message paths
pass the resolved session model through _resolve_session_reasoning_config;
/reasoning status reads the session model override).
Cleanup: drop docs/PER_MODEL_REASONING.md (duplicates the website docs
page), drop the change-detector _config_version test (no bump needed —
deep-merge handles new keys), remove a stale plan-reference comment.
Adds chokepoint contract tests (13) and gateway session-effective-model
regression tests (2).
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.
Example:
agent:
reasoning_effort: "medium" # global default
reasoning_overrides:
"openrouter/anthropic/claude-opus-4.5": "xhigh"
"openai/gpt-5": "low"
"claude-sonnet-4.6": "high" # bare model name also works
The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.
Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)
Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
+ _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
+ Re-resolves reasoning_config for the fallback model (best-effort)
Closes#21256 (per-model reasoning_effort defaults).
Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
Pin the Upstage default to the concrete solar-pro3 instead of the
solar-pro rolling alias:
- plugin fallback_models is now ("solar-pro3",); entry [0] is the setup default
- drop the "solar-pro" context-window fallback entry (solar-pro3 covers it)
- update the reasoning default-on docstring and profile tests accordingly
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the `solar-open2-preview` context-window entry; `solar-open2`
covers the Open 2 family at the same 256K window.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds Upstage Solar as a bundled model-provider plugin. Solar exposes an
OpenAI-compatible chat-completions endpoint at https://api.upstage.ai/v1, so
the generic chat_completions transport handles request/response/streaming/tool
calls — the profile is the core integration.
Provider registration (Upstage isn't in models.dev, so each registry that does
not auto-wire from the plugin layer needs an explicit entry — same pattern as
nvidia/gmi):
- plugins/model-providers/upstage/: UpstageProfile + plugin.yaml. Picker default
and offline catalog list only the agentic Solar Pro models, led by `solar-pro`
(rolling alias for the latest Pro). default_aux_model empty so aux tasks use
the main model. `solar` alias. UPSTAGE_BASE_URL overrides the host.
- hermes_cli/providers.py: HERMES_OVERLAYS + label + `solar` alias, so
resolve_provider_full('upstage') resolves (without this, an explicit
`provider: upstage` in config was dropped and fell through to auto-detect).
- hermes_cli/auth.py: PROVIDER_REGISTRY entry + `solar` alias, so `hermes
doctor` / resolve_provider recognise upstage (the static-registry path the
lazy profile-extension doesn't reliably cover at validation time).
- hermes_cli/models.py: CANONICAL_PROVIDERS entry places Upstage Solar in the
curated picker order (above the auto-appended `custom`).
- agent/model_metadata.py: context-window fallbacks (/v1/models omits
context_length); `solar-pro` carries the 128K Pro context as the catch-all.
Reasoning: UpstageProfile.build_api_kwargs_extras wires Solar's top-level
`reasoning_effort` (low|medium|high; xhigh/max→high). Reasoning-capable families
are solar-pro* and solar-open*; solar-mini/syn-pro never receive it. Defaults ON
at medium when unset (matches the /reasoning "medium (default)" label);
`/reasoning none` disables; explicit/saved settings are honored. No
reasoning_content echo handling needed (unlike DeepSeek/Kimi).
Web dashboard:
- web/src/pages/EnvPage.tsx: add an "Upstage Solar" provider group so
UPSTAGE_API_KEY / UPSTAGE_BASE_URL appear under LLM Providers (not "Other").
Docs/tests:
- .env.example: documents UPSTAGE_API_KEY / UPSTAGE_BASE_URL.
- tests: profile wiring, reasoning_effort mapping (pro/open/mini, efforts,
disabled, default-on), provider-resolver regression (resolve_provider_full /
get_provider / solar alias / overlay), `solar-pro` default.
Testing: pytest tests/providers tests/plugins/model_providers
tests/hermes_cli/test_upstage_provider.py tests/run_agent/test_provider_parity.py
tests/hermes_cli/test_api_key_providers.py; ruff clean. Verified end-to-end:
`hermes doctor` shows "Upstage Solar", and live chat works via both
`--provider upstage` and `--provider solar`. Reasoning wire format per
https://console.upstage.ai/api/docs/for-agents/raw. Platforms tested: macOS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the cherry-picked empty-user-turn drop: the placeholder
introduced in 8582f35d9 fired for whitespace-only STRING turns too
(content=' ' flattens to non-stripping text but isn't in the
(None, '', []) exclusion set), fabricating an attachment note for a turn
that carried nothing. Gate the placeholder on isinstance(content, list)
so only genuinely structured (e.g. image-only) turns get it; empty and
whitespace-only string turns now fall through to the drop path.
Edge cases verified: trailing empty user turn still ends the view on the
synthetic advisory marker; an all-empty transcript degenerates to [].
MoA's _reference_messages() unconditionally appended every user-role
message to the advisory view sent to reference models, even when the
message content was an empty string or a non-string/multimodal payload
that the text-extraction step flattens to "".
Strict providers (Kimi/Moonshot, and others that enforce non-empty user
content) reject such a message with:
400 Invalid request: the message at position N with role 'user'
must not be empty
Lenient providers (DeepSeek) accept it, so an identical rendered view
passes on one reference and 400s on another within the same fan-out —
the user sees "kimi doesn't support MoA" when the real cause is an empty
user turn leaking into the advisory transcript.
Skip empty user turns, mirroring the existing behavior for empty
assistant turns (which are already dropped when they carry no parts).
The end-on-user invariant is preserved: the synthetic advisory-request
user turn is still appended when the view would otherwise end on an
assistant turn.
Adds a regression test asserting the advisory view contains no empty
user turn and still ends on a user turn.
The platform hint in PLATFORM_HINTS['telegram'] always encouraged rich
Markdown constructs (tables, task lists, math, collapsible details) even
when rich_messages: false (the default). This caused the agent to produce
formatting that MarkdownV2 cannot render, especially broken on Telegram Web.
Split the hint into a base hint (MarkdownV2-compatible) and a
TELEGRAM_RICH_MESSAGES_HINT extension. The extension is conditionally
appended in system_prompt.py only when
platforms.telegram.extra.rich_messages is true.
Fixes#57122
Review follow-up to the reasoning_config cache-parity fix:
- Only inherit the parent's reasoning_config when the fork runs on the
parent's model (not routed). On the routed aux path
(auxiliary.background_review.{provider,model}) the cache is cold
regardless, so parity buys nothing, and the parent's effort vocabulary
can be invalid for the routed model/provider: OpenRouter
extra_body.reasoning.effort is forwarded unclamped
(chat_completions.py) and codex_responses only maps max/ultra for
gpt-5.6 — an exotic parent effort routed to a strict provider could
400 the review. Mirrors the existing 'not _routed' gate on
_cached_system_prompt / session_start three lines below.
- Add a routed-path regression test asserting reasoning_config is
omitted from the fork kwargs when _resolve_review_runtime returns
routed=True.
- Extract the four copy-pasted recorder stubs in
test_background_review_cache_parity.py into a single
_make_recorder_class() factory so a new fork attribute needs one stub
edit, not four.
PR #17276 painstakingly pinned `_cached_system_prompt`, `session_start`,
`session_id`, and the toolset config on the background-review fork so its
outbound request body would byte-match the parent's and hit Anthropic's
exact-prefix cache. The contributor measured a ~26% end-to-end cost
reduction on Sonnet 4.5.
That optimization is currently being silently undone by a missing
`reasoning_config` kwarg. The fork's `AIAgent(...)` call omits it, so the
fork's `reasoning_config` defaults to `None`. `anthropic_adapter.build_anthropic_kwargs`
(line ~2165) then short-circuits the `thinking` / `output_config` block,
and the fork's request body lands in a DIFFERENT Anthropic cache namespace
from the parent's.
Result on the wire: 0 `cache_read_input_tokens`, full `cache_creation_input_tokens`
of the entire parent prefix — every single background review.
7 days of midagent.db traffic from one host running stock Hermes against
Anthropic Sonnet:
```
Background-review FIRST calls (the moment a review fork is born):
count = 68
cache_write tokens = 7,004,297
cache_read tokens = 1,016,335
Cost on Sonnet ($3.75/M write vs $0.30/M read):
Spent on these writes: $26.27
Cost if they had hit parent cache instead: $2.10
WASTED: $24.16 / week / user
```
That is from one user. Multiply by Hermes's installed base for the full
impact.
Tested against api.anthropic.com directly (see refs/api-tests/ in the
attached investigation repo if needed):
| pair | cache_r | cache_w |
|---------------------------------------------|---------|---------|
| parent fresh | 0 | 24,047 |
| parent same again | 24,047 | 0 |
| fork: appends 2 new tail msgs, thinking ON | 24,047 | 22 |
| fork: appends 2 new tail msgs, thinking OFF | 0 | 24,047 |
Same fork-shape request, only difference is `thinking`. With the fix,
the fork hits the parent's full prefix and only writes the delta
(the `Review the conversation above…` prompt block, ~3-5K tokens).
One line in `agent/background_review.py`: pass
`reasoning_config=getattr(agent, "reasoning_config", None)` to the
`AIAgent(...)` constructor of the review fork. A short comment block
above it explains why so the next person who reads this code doesn't
re-introduce the regression.
`tests/run_agent/test_background_review_cache_parity.py` already covers
the system-prompt / session-id / toolset-config parity contracts that
PR #17276 introduced. I added:
* a `reasoning_config` attribute to `_make_agent_stub` so the stub has
a non-None parent value the test can verify is propagated.
* `test_review_fork_inherits_parent_reasoning_config()` — asserts the
fork's `AIAgent(...)` kwargs carry the parent's `reasoning_config`.
Pre-fix this test fails with `None vs expected {'enabled': True, 'effort': 'medium'}`;
post-fix all 4 tests in the file pass.
```
$ python -m pytest tests/run_agent/test_background_review_cache_parity.py -v
test_review_fork_inherits_parent_cached_system_prompt PASSED
test_review_fork_pins_session_start_and_session_id PASSED
test_review_fork_inherits_parent_toolset_config PASSED
test_review_fork_inherits_parent_reasoning_config PASSED ← new
```
Also runs against the broader background-review test suite:
`test_background_review.py` (4), `test_background_review_summary.py` (8),
`test_background_review_toolset_restriction.py` (3) — 19/19 pass.
`agent/curator.py:1691` has the same omission for the umbrella-curation
fork, but curator's prompt is "curate all skills" — it shares no prefix
with any user conversation, so cache-parity is a non-issue there. Worth
auditing if the curator ever takes a parent conversation as input, but
not part of this PR.
The `agent/auxiliary_client.py:1006` `reasoning_config=None` hardcode is
intentional (title/summary one-shots on short prompts — per-call cost
of namespace flip is negligible) and is also out of scope.
Return actionable errors when HERMES_WRITE_SAFE_ROOT blocks a path instead of
labeling every denial as a protected credential file. Wire the helper through
write_file, patch, delete/move, and the Copilot ACP shim; sync docs examples.
Provider auto-detection (URL-based inference for Anthropic, OpenAI Codex,
and xAI endpoints) runs before credential-pool validation in AIAgent init,
but #63048 placed the pool validation before auto-detection. When the agent
is constructed with provider=None and a recognized endpoint URL, the pool
is validated against an empty provider identity and discarded, even though
auto-detection correctly resolves the provider moments later.
Fix: move the credential-pool validation block to after the URL-based
auto-detection chain. The pool is stored on the agent before
auto-detection; validation now checks the resolved provider and only
nullifies agent._credential_pool when the pool's scoped provider genuinely
doesn't match.
Regression test covers all three auto-detection paths:
- Anthropic (api.anthropic.com)
- OpenAI Codex (chatgpt.com/backend-api/codex)
- xAI (api.x.ai)
Fixes#63425.
- Nudge text now warns that repeated protocol violations will block the
task and require manual intervention, so the model understands the
consequence of ignoring the nudge.
- Kanban docs restructured to clearly separate the two defense layers:
agent-side prevention (nudge, from #64350) and dispatcher-side
recovery (bounded retry, from this PR).
- Two new integration tests verifying the nudge mentions blocking and
that the agent-side and dispatcher-side budgets are independent.
scan_skill_commands() had two collision bugs in the same loop body:
1. Core-command collision: a skill whose normalized slug matches a core
Hermes command name or alias (e.g. "skills", "learn", "bg") would
get an auto-generated /command that shadows the core command in the
gateway dispatch path (skill map is consulted before built-in
handlers). The skill command silently overrode the core command.
2. Inter-skill slug collision: the seen_names set deduped on the raw
frontmatter name, but the command map was keyed by the normalized
slug. Two distinct names collapsing to the same slug (e.g.
"git_helper" vs "git-helper") both passed the dedup, and the second
silently clobbered the first.
Fix: add two guards in scan_skill_commands() after slug normalization:
- resolve_command(cmd_name) check skips skills colliding with any core
CommandDef (name or alias), logging a warning. Uses the existing
resolve_command() API so aliases and case variants are covered
without a separate cache. The skill remains loadable via /skill.
- cmd_key in _skill_commands check dedups on the resolved slug,
first-wins (preserving local-before-external precedence), logging
a warning naming the shadowed skill.
Combines and supersedes #31204 (@cyrkstudios), #53450 (@Gridzilla),
#50304 (@petrichor-op), and #63305 (@Vissirexa).
Co-authored-by: cyrkstudios <cyrkstudios@users.noreply.github.com>
Co-authored-by: Gridzilla <Gridzilla@users.noreply.github.com>
Co-authored-by: petrichor-op <petrichor-op@users.noreply.github.com>
Co-authored-by: Vissirexa <Vissirexa@users.noreply.github.com>
Add a bounded turn-end stop guard for kanban workers. When a worker
tries to exit with finish_reason=stop without having called
kanban_complete or kanban_block, inject up to two synthetic nudges
so the conversation loop continues instead of exiting cleanly (which
the dispatcher records as protocol_violation).
Mirrors the existing verify-on-stop pattern: same ephemeral scaffolding
flag (_kanban_stop_synthetic), same role-alternation contract, same
_pending_verification_response fallback for budget exhaustion.
Disabled by default (gated on HERMES_KANBAN_TASK env var set by the
dispatcher); kill switch via HERMES_KANBAN_STOP_NUDGE=0.
Salvaged from #62262 by @mdc2122. The original branch was 272 commits
behind main with ~538 files of stale-base reversions; this salvage
applies only the 4 substantive files (agent/kanban_stop.py,
conversation_loop.py insertion, run_agent.py _EPHEMERAL_SCAFFOLDING_FLAGS,
tests/agent/test_kanban_stop.py).
Salvage of #63888. The original fix clears stale _last_content_with_tools
on substantive tool-only turns but doesn't clear _mute_post_response, which
a prior housekeeping turn may have set. This suppresses tool progress
output via _vprint until the no-tool-call branch resets it at line ~4834
— after all tools have finished executing.
Fix: also reset _mute_post_response = False when clearing stale fallback.
Added test: verify pure housekeeping turns (content + only housekeeping
tools) still set the fallback correctly — the original use case the
fallback was designed for.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
A cached _last_content_with_tools response from a housekeeping-only turn
could survive a later substantive tool-only turn. When the model returned
an empty response, Hermes incorrectly finalized the older housekeeping
narration instead of invoking the post-tool empty-response nudge.
Production impact: scheduled cron jobs could return early without completing
their actual work (e.g., daily report job returning a housekeeping message
instead of producing the report artifact).
Root cause: The fallback state was only updated when a turn had both
content AND tool_calls. A turn with tool_calls but empty visible content
would skip state updates entirely, leaving stale fallback state intact.
Fix: Classify tools in every tool-call turn (regardless of visible content).
When any tool is substantive (non-housekeeping), clear the older fallback state
before processing later empty responses. This prevents two-turn-old housekeeping
narration from being treated as if it belonged to the immediately preceding
substantive tool turn.
Regression test added: tests/run_agent/test_conversation_fallback_state.py
Fixes#63860
OpenAI lets ChatGPT-plan Codex users bank rate-limit reset credits, but
until now they could only be redeemed from the Codex CLI/app or the
website. This wires the same backend API into Hermes:
- /usage on the openai-codex provider now shows "You have N resets
banked - use /usage reset to activate" (parsed from the
rate_limit_reset_credits field the /usage endpoint already returns).
- New /usage reset subcommand (CLI + gateway) redeems one banked
credit via POST .../rate-limit-reset-credits/consume with a UUID
idempotency key, mirroring codex-rs backend-client semantics
(PathStyle /wham vs /api/codex, ChatGPT-Account-Id header,
reset/nothing_to_reset/no_credit/already_redeemed outcomes).
- Guard: redemption is refused while no rate-limit window is fully
exhausted, since a banked reset restores the FULL 5h + weekly
allowance and spending it early wastes it. /usage reset --force
overrides. Zero banked credits and non-codex providers are refused
with clear messages; nothing_to_reset reports the credit was NOT
spent.
- i18n: new gateway.usage.unknown_subcommand / reset_wrong_provider
keys across all 16 locales; docs updated (cli.md, messaging index).
Tested with unit tests plus a real-socket E2E against a local fake
Codex backend exercising redeem/guard/force and the /usage hint.
