api_server already caps every read via client_max_size (chunked
included), but when the limit tripped mid-read the handler's broad JSON
except turned it into 400 'Invalid JSON'. Catch
HTTPRequestEntityTooLarge in body_limit_middleware and return the
OpenAI-style 413.
Status-code polish extracted from PR #3949 by @Gutslabs — the PR's core
client_max_size change already exists on main.
The webhook adapter enforced max_body_bytes only via the Content-Length
header; a Transfer-Encoding: chunked request (content_length=None) or a
spoofed small Content-Length bypassed the cap entirely and read the full
body (bounded only by aiohttp's implicit 1 MiB default, above any
operator-configured smaller limit).
- web.Application(client_max_size=max_body_bytes): aiohttp enforces the
cap on every read path, chunked included
- catch HTTPRequestEntityTooLarge -> 413 (was swallowed into generic 400)
- post-read length re-check as defense in depth
- chunked-upload regression test
Manual port of PR #3955 by @Gutslabs onto current main (handler had
been restructured since); authorship preserved.
Add unit tests for resolve_placeholder_terminal_cwd and extend the config
bridge simulation for docker mount-on vs mount-off vs local fallback.
Co-authored-by: Cursor <cursoragent@cursor.com>
Split placeholder TERMINAL_CWD resolution into three cases: local falls
back to MESSAGING_CWD/home; docker without workspace mount leaves cwd unset;
docker with mount enabled preserves an explicit host MESSAGING_CWD path for
terminal_tool's /workspace mapping. Stops leaking host Path.home() into
containers without breaking the mount contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
The conflict-retry ladder schedules a background recovery task via
loop.create_task(self._handle_polling_conflict(...)) on each failed
start_polling. test_polling_conflict_becomes_fatal_after_retries never
cancelled the last one, so under a loaded scheduler a leaked task could
get a turn, re-drive the counter into the fatal branch, and fire
_notify_fatal_error a second time — breaking assert_awaited_once()
non-deterministically. The bounded updater.stop() guard added in this
salvage introduced an extra await/scheduling yield that surfaced the
latent leak in CI slice 3. Cancel the leaked task before the fatal
assertions so the test is deterministic regardless of scheduler timing.
The salvaged fix (#58272) guarded the primary network-error reconnect path.
Issue #58270's Scope section names three more unguarded await updater.stop()
sites that can hang identically on a CLOSE-WAIT socket:
- conflict handler (before the retry back-off sleep)
- conflict-retries-exhausted teardown (before the fatal notify)
- disconnect() teardown (would hang gateway shutdown/restart)
Each is now wrapped in asyncio.wait_for(..., _UPDATER_STOP_TIMEOUT) with a
warning on timeout, matching the primary path, so no reconnect/teardown ladder
can wedge on a dead socket.
Also hoist the shared 15.0s bound to a single module constant
_UPDATER_STOP_TIMEOUT (self-documenting + DRY across all 4 sites), and update
the CLOSE-WAIT regression test to patch that constant instead of monkeypatching
asyncio.wait_for process-wide.
Fatal-notify idempotency: bounding the conflict-exhausted teardown stop() adds
an await AFTER _set_fatal_error, which yields the loop and lets a concurrent
retry task (scheduled by an earlier conflict, already suspended past the entry
guard) reach the fatal branch too — double-firing the fatal handler (surfaced
as a Python 3.11 CI failure in test_polling_conflict_becomes_fatal_after_retries).
Snapshot the pre-transition fatal state and only notify on the first transition.
When the TCP connection enters CLOSE-WAIT the PTB polling task is blocked
on epoll on a dead socket and never wakes. updater.stop() awaits that task
and therefore hangs indefinitely.
Consequence: _polling_error_task stays alive-but-blocked forever; every
subsequent heartbeat probe sees it as "in-flight" and skips triggering a
new reconnect; the gateway silently drops messages for hours until a manual
restart. Field incident: 11-hour outage on 2026-07-04 UTC despite the
heartbeat loop firing a reconnect at 01:11 — stop() blocked the entire
ladder.
Fix: wrap the updater.stop() call inside asyncio.wait_for(timeout=15).
On TimeoutError log a warning and continue to _drain_polling_connections()
+ start_polling() — same recovery path, just unblocked.
The heartbeat loop (PR #48496) correctly detects the dead socket and fires
_handle_polling_network_error. This commit is the missing second half:
ensuring the reconnect itself always completes.
Test: test_handle_polling_network_error_updater_stop_timeout() simulates
a hang by making stop() sleep forever and verifies that drain + start_polling
are still reached after the timeout.
Fixes#58270
Strict providers (DeepSeek) reject a payload where the same tool_call_id
appears more than once with HTTP 400 'Duplicate value for tool_call_id'.
The issue was filed as an 'orphaned tool message' compression bug, but the
pasted error is a DUPLICATE tool_call_id — orphans are already handled on
main; duplicates were not. Reproduced live on main: both shapes leaked
through repair_message_sequence and sanitize_api_messages.
Two chokepoints, two shapes:
- repair_message_sequence: consume the id from known_tool_ids on first
match so a SECOND tool result reusing it falls into the drop branch
(duplicate tool-result shape). This is @Robinlovelace's kernel from
#55436 (applied manually — that PR was ~800 commits stale and bundled
an unrelated duplicate-DB-write change for #860, which is dropped here).
- sanitize_api_messages (final pre-API pass): add a dedup pass covering
BOTH (a) duplicate tool_calls sharing an id WITHIN one assistant message
(the message[6] shape) and (b) later tool result messages reusing an
already-seen id. #55436 covered neither of these at this chokepoint.
Tests: duplicate-tool-result dedup at both functions, duplicate-assistant-
tool_call-id collapse, and a negative control proving distinct ids are
never dropped (no over-dedup).
Credit: @Robinlovelace (#55436) for the repair_message_sequence dedup kernel.
Closes#58327.
Contributor of the salvaged PR #58276 fix commit. Required so the
contributor attribution CI check passes on the rebase-merge that
preserves their authorship.
Replace the inline dict-copy + _db_persisted pop in
_ensure_compressed_has_user_turn with the canonical
_fresh_compaction_message_copy helper (the same primitive the compressor's
own protected-head/tail assembly uses), so the persistence-marker strip
stays consistent across all compaction copy sites (#57491). Expand the
docstring to record the alternation-safety and end-placement rationale.
22c5048d9 restored Anthropic-style cache_control for two of MoA's three
call paths: the acting aggregator (MoAChatCompletions.create, the
persistent `provider: moa` model) and the advisor fan-out (_run_reference).
aggregate_moa_context() -- the /moa <prompt> one-shot command's synthesis
call -- is the third, independent call path and was never covered: its
call_llm(task="moa_aggregator", ...) sent a single undecorated user message
containing the full joined reference output, re-billing the entire input on
every invocation even when the resolved aggregator slot is a cache-honoring
route (Claude on OpenRouter/native Anthropic, MiniMax, Qwen/DashScope).
- Generalize _maybe_apply_advisor_cache_control to
_maybe_apply_moa_cache_control (it never had advisor-specific logic --
same policy function, same breakpoint layout as the main loop, judged
purely on the passed-in runtime) and reuse it in aggregate_moa_context
the same way _run_reference already does.
- Compute _slot_runtime(aggregator) once and reuse it for both the
decoration call and the call_llm kwargs, instead of calling it twice.
Mutation-verified: reverting the moa_loop.py change makes the new
regression test fail by asserting a plain string aggregator-message
content where the cache-honoring case expects native cache_control
content blocks.
Follow-up to @msh01's wall-deadline init-timeout fix.
- Resource leak: on timeout the initialize() task is abandoned without
awaiting its (shielded, possibly-never-completing) cancellation, so the
half-built PTB app's httpx client / connection pool was never closed —
up to 8x across the retry ladder. Add an optional on_abandon cleanup to
_await_with_thread_deadline that best-effort app.shutdown()s the abandoned
app, run detached + exception-swallowed so it can never re-block or re-hang
the ladder (mirrors _close_client_on_timeout in agent/auxiliary_client.py).
- Cover the helper itself: the salvaged test monkeypatched out the real
_await_with_thread_deadline, so its abandonment/cleanup path was untested.
Add direct tests for happy-path return, prompt-timeout-with-cleanup, and
cleanup-error-swallowed; the wedged coroutines swallow cancellation for a
bounded window (proving the helper returns before cancellation completes,
the #58236 shielded-scope behavior) without leaving an immortal task that
would wedge pytest teardown. Widen the salvaged stub to accept on_abandon.
- Attribution: add yingwaizhiying@gmail.com -> msh01 to AUTHOR_MAP (bare
gmail does not auto-resolve the check-attribution gate).
Known follow-up (not addressed here): the retry ladder reuses the same
self._app across all 8 attempts; a fresh app per attempt would fully close
the coherence risk if an abandoned initialize() completes in the background.
That is a larger restructure of the ~130-line builder+handler setup, left
for a separate change.
Follow-up to @srojk34's basename-denylist widening. Two gaps the
basename-only guard left, both covered by the two canonical guards it
mirrors:
- Directory-tree stores mcp-tokens/ (live MCP OAuth tokens) and pairing/
are denied as whole trees by gateway.platforms.base._ROOT_CREDENTIAL_DIRS
and agent.file_safety, but the dashboard files API descends into subdirs,
so mcp-tokens/<server>.json (non-canonical basename) stayed
listable/readable/downloadable. Add _is_sensitive_path(), a path-aware
check that blocks any path with a credential-directory component, and
route all three call sites (list/read/download) through it.
- Add .git-credentials to the basename set (agent.file_safety blocks it too).
- Correct the docstring: it now says it mirrors the credential-FILE basenames
of the canonical guards, with the directory trees handled by the new
path-aware helper (the prior wording overstated parity).
Scope stays on the read/list/download exfil surface (#57505); the write
endpoints (upload/mkdir/delete) are a separate threat and out of scope.
Tests: dir-tree descent blocked (mcp-tokens/pairing per-server files),
.git-credentials blocked, plus a positive control that a benign subdir file
stays browsable. Mutation-checked (neuter _is_sensitive_path -> new tests
fail). 39 web_server_files + fs tests pass, ruff clean.
_is_sensitive_filename() only blocked .env / .env.<suffix>, but the
dashboard Files tab's managed root is operator-configurable and, per the
docker-mount scenario #57505 was filed against, can point directly at
HERMES_HOME — where the canonical credential stores enforced elsewhere
in the codebase (gateway.platforms.base._ROOT_CREDENTIAL_FILES,
agent.file_safety.get_read_block_error) all live: auth.json, OAuth
token stores, webhook HMAC secrets, the Bitwarden disk cache. None of
those basenames were blocked, so the Files tab could still list, read,
and download them. .envrc (direnv) also slipped past the old check
since it doesn't equal ".env" or start with ".env.".
Widen the basename set to mirror both existing guards so the dashboard
doesn't lag behind them.
Both wired against features the iron-proxy author (@mslipper) confirmed on
PR #30179 — and both verified present in the pinned v0.39.0 source.
Header-auth providers (match_headers):
- New _HEADER_AUTH_PROVIDERS: Anthropic native (x-api-key), Azure OpenAI
(api-key on *.openai.azure.com / *.cognitiveservices / *.services.ai),
Gemini (x-goog-api-key + ?key= query param via match_query).
- TokenMapping grows match_headers + alias_env_names; per-provider header
sets flow into the secrets rules; mappings.json roundtrips them
(legacy files load with the Authorization default).
- GEMINI_API_KEY / GOOGLE_API_KEY collapse into ONE mapping (two
require-rules on the same host would reject each other); the sandbox
gets the token under both names, and the proxy child env mirrors the
alias into the canonical name when only the alias is set.
- Docker backend injects alias env names alongside canonical ones.
- The fail-closed tier is now empty, so fail_on_uncovered_providers and
discover_blocked_providers are deleted (dead toggle otherwise);
_NON_BEARER_PROVIDERS shrinks to genuinely-unswappable signature auth
(AWS SigV4, GCP service-account OAuth) — warn-only, as before.
Management API (hot reload):
- Generated proxy.yaml enables the v0.39 management listener: loopback
only at tunnel_port+2, bearer key from HERMES_IRON_PROXY_MGMT_KEY.
- Key minted at setup (management.token, 0600); start_proxy injects it
(v0.39 refuses to start when api_key_env is empty).
- hermes egress reload -> POST /v1/reload: re-reads proxy.yaml and
atomically swaps the pipeline; 422 leaves the running ruleset
untouched; actionable errors for not-running / pre-management config /
key mismatch. Secrets changes still require restart (daemon env is
read at spawn) — the CLI says so.
Validation: 218/218 unit+CLI+docker tests; 3/3 gated live E2E against the
real v0.39.0 binary (Authorization swap, x-api-key swap, live reload with
token rotation on the same pid). Docs updated.
Two review findings on the #57922 salvage:
1. Stale inline comment at the login-exchange site still claimed the token
endpoint uses the claude-code/ UA prefix and 404s claude-cli/ — now
contradicts the axios/ fix. Repointed it at _OAUTH_TOKEN_USER_AGENT.
2. The inherited Path.home test isolation on the three TestRefreshOauthToken
tests only stubbed the ~/.claude *file* source, not the macOS Keychain.
_refresh_oauth_token re-reads read_claude_code_credentials() (keychain
first) in its adopt-already-refreshed branch, so on any macOS dev/CI runner
with real Claude Code creds the branch short-circuits and the 3 tests fail.
Stub read_claude_code_credentials -> None so the tests are hermetic.
(The remaining TestResolveAnthropicToken/TestResolveWithRefresh/TestRunOauthSetupToken
failures on macOS are the same pre-existing keychain-leak class on origin/main,
unrelated to this OAuth-UA fix, and pass in CI — left out of scope.)
hermes auth add anthropic fails 100% at token exchange with HTTP 429 while
Claude Code /login succeeds through the same client_id/redirect/scope. The
discriminator is the User-Agent on the /v1/oauth/token request.
Verified live against platform.claude.com (throwaway code, nothing burned):
claude-code/2.1.200 (external, cli) -> 429 rate_limit (Hermes, blocked)
Mozilla/5.0 -> 429 rate_limit
axios/1.7.9 -> 400 invalid_grant (reached validation)
node / empty / SDK-style UAs -> 400 invalid_grant
Anthropic now rate-limits token-endpoint requests whose UA starts with
claude-code/ (the anti-abuse net for Max-sub-as-API-key). This is the same
prefix-block shape that #48534 first hit on claude-cli/, then #56263 dodged by
switching to claude-code/ — which held ~2 weeks and is now blocked too. Bumping
_CLAUDE_CODE_VERSION_FALLBACK cannot help; the gate is prefix-based.
Fix: shared _OAUTH_TOKEN_USER_AGENT (axios/) on the token endpoint only — the
two refresh POSTs (refresh_anthropic_oauth_pure) and the login exchange POST
(run_hermes_oauth_login_pure). The real Claude Code CLI exchanges the auth code
with a bare axios client, NOT its claude-code/ inference UA.
The INFERENCE client (build_anthropic_kwargs, /v1/messages) is deliberately left
on claude-code/ + x-app: cli — that fingerprint is required there and is NOT
throttled on the messages API. Two endpoints, opposite UA requirements.
Also isolate two _refresh_oauth_token tests from live ~/.claude creds and update
the UA regression tests to assert the split (token endpoint uses a
non-claude-code UA while inference keeps claude-code/).
Verified E2E: Hermes' own login path now returns 400 (past the 429 wall)
instead of 429, using the real _OAUTH_TOKEN_USER_AGENT constant against the live
platform.claude.com token endpoint.
Salvaged from #57922 (authorize-host + scope changes dropped as non-load-bearing;
they only add a redirect hop back to claude.ai and the UA fix alone clears 429).
repair_message_sequence Pass 1 registered only tc.get("id") when building
the set of known assistant tool_call ids, then matched tool results against
it by tool_call_id. In the Codex Responses format an assistant tool_call
carries both id (fc_...) and a distinct call_id (call_...); a tool result's
tool_call_id may be keyed on either depending on which builder produced it.
Registering only id made a valid tool result whose tool_call_id matched
call_id look orphaned, so the pass dropped it and left the assistant
tool_call unanswered -- producing HTTP 400 on strict providers (DeepSeek,
Kimi): 'Messages with role tool must be a response to a preceding message
with tool_calls'. Long-running sessions that persisted such a sequence were
permanently broken, re-sending the orphan every turn.
Register both id and call_id for each assistant tool_call so a result
matching either key is recognized, consistent with
AIAgent._get_tool_call_id_static and the compressor's _sanitize_tool_pairs.
Apply the same call_id||id precedence to the corrupted-args sanitizer's
existing-result scan / stub insertion, which had the identical mismatch.
Adds 3 regression tests covering the codex id!=call_id case (match on
call_id, match on only call_id, match on id when both present).
On X11 a window's PID comes from the optional _NET_WM_PID property, so
cua-driver's list_windows legitimately returns pid: null for windows that
don't set it (desktop root, panels, override-redirect popups). capture()
and focus_app() coerced every entry via int(w["pid"]) inside a list
comprehension, so a single null-pid window raised TypeError and aborted
the whole enumeration before any screenshot — capture was impossible on
any X11 desktop with even one such window.
Route both ingestion sites through a new _ingest_windows() helper that
skips entries lacking a usable pid/window_id (uncapturable anyway) and
coerces the rest.
Adds tests/tools/test_computer_use_null_pid_windows.py covering the
helper's filtering/coercion and an end-to-end capture() regression that
reproduced the crash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The #22773 heuristic classified any telegram:<positive_chat_id>:<numeric_thread_id>
cron target as a Bot API channel Direct-Messages topic and routed it via
direct_messages_topic_id, which nulls message_thread_id. A normal forum-style
topic inside a private chat has the identical shape, so every such cron
delivery landed in General instead of the target thread (#52060). It also
means the only way to address a genuine channel DM topic was this same
ambiguous guess.
Disambiguate with the real runtime signal instead: probe the live adapter's
get_chat_info once and route via direct_messages_topic_id only when the chat
is actually a channel; everything else (private forum topic, forum supergroup,
group) and any probe failure fall back to message_thread_id — parity with
0.16.0 and with live reply routing. This fixes forum-topic delivery AND makes
genuine channel DM-topic cron delivery work correctly.
check-attribution requires every contributor author email to be in
AUTHOR_MAP; the salvaged commit is authored by
huanshan5195 <huanshan5195@users.noreply.github.com>.
PR #57601's original branch added a top-level reasoning_effort emit to the
LEGACY build_kwargs path (agent/transports/chat_completions.py), but
provider=custom resolves to CustomProfile (plugins/model-providers/custom/),
so chat_completion_helpers takes the profile path and returns early — the
added branch was unreachable dead code for every custom endpoint.
Move the fix to its real site, CustomProfile.build_api_kwargs_extras(), and
follow the DeepSeek/Zai profile precedent:
- disabled -> extra_body.think = False (unchanged)
- enabled + effort -> TOP-LEVEL reasoning_effort (the OpenAI-compatible
format GLM-5.2/ARK expect), passed through verbatim
incl. max/xhigh
- enabled + no effort -> omit, so the endpoint's server default applies
(avoids silently forcing 'medium' as the original
branch did)
Deliberately does NOT force think=True on enable — that flag is Ollama-only
and risks a 400 on GLM/vLLM endpoints that don't recognize it; thinking is
already server-default-on for these backends.
Verified end-to-end through the real profile dispatch (temp HERMES_HOME):
custom+high -> reasoning_effort=high; custom+max -> reasoning_effort=max;
custom+none -> think=False; custom+unset -> nothing; num_ctx composes.
Adds tests/plugins/model_providers/test_custom_profile.py (13 cases).
Addresses the custom-provider half of #55276.
Co-authored-by: huanshan5195 <huanshan5195@users.noreply.github.com>
Follow-up to salvaged #57601. Adding "max" to VALID_REASONING_EFFORTS
made parse_reasoning_effort("max") valid, so:
- test_unknown_levels_return_none no longer lists "max" (it is now valid;
auto-covered by test_each_valid_level which iterates the tuple).
- test_known_supported_levels_are_documented and the parse_reasoning_effort
docstring now include "max" so the doc-sync guard actually protects it.
- Add 'max' to VALID_REASONING_EFFORTS (GLM-5.2 native parameter)
- Emit top-level reasoning_effort string for custom providers
- Stop hardcoding 'medium' in legacy extra_body.reasoning, use actual effort
Custom providers (e.g. GLM-5.2 on Volcengine ARK) silently dropped
reasoning_effort — the value never reached the upstream API. Kimi,
TokenHub, and LM Studio all had dedicated branches for this, but
custom providers had none.
Follow-up to the salvaged SSH-tilde-cwd fix. The predicate
"backend == ssh and (cwd == ~ or cwd.startswith(~/))" was inlined at
each expanduser guard site, which is how the test simulator drifted from
production (it grew an SSH guard on a top-level-alias branch that has no
production counterpart).
- Add tools/terminal_tool._is_ssh_remote_tilde_cwd(backend, cwd) as the
single source of truth (case/whitespace-tolerant).
- Use it in _get_env_config and the gateway config bridge.
- Test simulator imports the real helper instead of re-implementing the
predicate; revert the phantom SSH guard on the top-level-alias branch
(production maps top-level cwd: to a plain env var, not TERMINAL_CWD via
an SSH-guarded path — that branch tested nothing real).
Self-review (hermes-pr-review Phase 2) flagged the mid-turn pre-API
compaction for reuse/duplication (W1/W2); fixing that surfaced a
regression against a deliberate existing feature, now also fixed.
The block now mirrors the turn-prologue preflight's guard chain exactly
(agent/turn_context.py) instead of a hand-rolled pressure limit:
1. should_defer_preflight_to_real_usage(rough) — defer when the rough
estimate is known-noisy vs a recent real provider prompt that fit
under threshold (schema overhead / post-compaction over-count, #36718).
2. get_active_compression_failure_cooldown() — skip during a same-session
compression-failure cooldown.
3. should_compress(rough) — reuses the canonical threshold_tokens (output
room already reserved by _compute_threshold_tokens) plus its summary-LLM
cooldown + anti-thrash guards (#11529).
Dropped the seven inline _reserve/_output_pressure locals (W2: they
re-derived _compute_threshold_tokens and omitted its 85% degenerate-window
fallback). compression_attempts stays as the hard per-turn backstop.
Without guard (1) the block fired a compaction the preflight deliberately
defers, breaking test_413_compression::test_preflight_defers_when_recent_
real_usage_fit (ValueError from the mocked _compress_context). Verified:
test_413_compression 26/26, codex 82/82, and anti-thrash engaged
(_ineffective_compression_count=2) still suppresses the block (0 calls).
Follow-up to the salvaged #58000 fix.
- Extract _raise_if_non_interactive(lead) so the shared 'hermes mcp login'
next-step wording lives in one place across both OAuth boundaries
(_redirect_handler, _wait_for_callback), rather than two copy-pasted
inline raises. Boundary-specific lead sentences preserved verbatim, so
existing message-match tests stay green.
- Add a positive-control test asserting the guard does NOT over-fire on the
interactive path (valid/refreshable tokens keep working), satisfying an
explicit regression-coverage line from #57836.
Add TestNonInteractiveFailFastAtCallbackBoundary: the callback boundary must
reject before binding a listener and without entering the poll loop, the guard
must hold even when a (stale) token file exists on disk, the redirect handler
must not print a URL or open a browser, and both boundaries must point users at
`hermes mcp login`.
Mark the existing timeout test and the SSH-hint redirect tests interactive so
they exercise their intended paths rather than short-circuiting on the new
non-interactive guard.
