Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
Closes#63529
Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.
Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.
Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
The base adapter's signature was updated to include
, which the reconnect watcher passes as
during reconnection. All other platform adapters were
updated, but QQAdapter was missed, causing:
TypeError: QQAdapter.connect() got an unexpected keyword argument 'is_reconnect'
This leads to an infinite retry loop since every reconnect attempt fails
immediately with the same TypeError.
Fix: add to QQAdapter.connect()'s signature.
QQBot has no server-side update queue, so the flag is accepted only for
interface conformance.
Test: new test_connect_accepts_is_reconnect_param verifies both
adapter.connect() and adapter.connect(is_reconnect=True) succeed without
raising.
- run_route_script shells out with subprocess.run (up to 30s timeout); wrap
the call in asyncio.to_thread so a slow script can't stall every other
webhook and gateway task on the loop.
- scripts/release.py: map grace@weeb.onl -> evelynburger for the salvaged
contributor commit.
Follow-up to the salvaged #54944: before this, aiohttp's implicit 1 MiB
default client_max_size tripped BEFORE the intended 3 MB Meta cap could
apply on read() paths — the explicit value makes the documented limit
real while the bounded reader keeps chunked bodies from buffering past
3 MB (#58536/#58902/#59180 pattern).
Sibling sweep from the #58902 raft review found aiohttp servers still
running on the implicit 1 MiB default with no explicit body cap:
- bluebubbles webhook (127.0.0.1): 1 MiB explicit cap — events are small
JSON/form payloads; attachments arrive via the REST API
- teams Bot Framework listener (0.0.0.0 bind — most exposed): 1 MiB cap;
activities are JSON well under that
- hermes proxy server: 10 MB cap mirroring api_server's MAX_REQUEST_BYTES
(chat-completion payloads can be large, but must stay bounded)
client_max_size bounds every read path including chunked transfer-encoding
requests that carry no Content-Length (#58536/#58902 pattern).
Deliberately excluded: feishu, whatsapp_cloud, sms, line, wecom, msgraph —
open contributor PRs (#54938, #54944, #54620, #54931, #54934, #25296)
already cover those; reviewing them separately preserves their credit.
3 regression tests pin the wiring.
ruff check --fix --select F541 . on current main. Pure prefix removals;
adjacent-string concatenations keep the f only on interpolating fragments.
No string content or live placeholder altered.
The unreadable-config-overwrite bug (an existing config.yaml that reads as
{} on a permission/IO error gets replaced with only defaults or the edited
section) is not limited to save_config / config set / auth. The same
read-then-atomic_yaml_write pattern lives at ~7 other independent write
sites that don't route through those functions:
- gateway/slash_commands.py: _save_config_key, memory/skills write_approval
toggles, tool_progress toggle, runtime_footer toggle, personality set
- hermes_cli/doctor.py --fix (stale root-key migration)
- gateway/platforms/yuanbao.py auto-sethome
- plugins/platforms/telegram/adapter.py topic thread_id persistence
- tui_gateway/server.py _save_cfg
- agent/onboarding.py mark_seen
Rather than sprinkle require_readable_config_before_write() at each site,
add a single fail-closed chokepoint, atomic_config_write(), that runs the
guard then delegates to atomic_yaml_write, and route every config.yaml
write through it. Root cause remains that read_raw_config() can't tell an
absent file from an unreadable one (returns {} for both) — read-only
callers correctly stay fail-open, but any full-file replacement now fails
closed in one enforced place instead of relying on each caller to remember
the guard.
save_config / set_config_value / auth keep the contributor's original
guard calls (their commit); this commit widens the fix to the sibling
call paths and adds a regression test on the chokepoint (fails closed on
unreadable existing file + still creates a genuinely absent file).
- warn once per route instead of on every request (busy senders would
spam the log)
- document X-Webhook-Signature-V2 / X-Webhook-Timestamp in the webhooks
user guide
Follow-ups for salvaged #58461.
The Cloud setup wizard and docs tell operators to set
WHATSAPP_CLOUD_ALLOWED_USERS (and WHATSAPP_CLOUD_ALLOW_ALL_USERS), but the
adapter DM intake gate only read WHATSAPP_CLOUD_ALLOW_FROM + WHATSAPP_CLOUD_DM_POLICY
(default open, opted-in only via GATEWAY_/WHATSAPP_ALLOW_ALL_USERS). So an
allowlist set via the documented var silently dropped every inbound
(_should_process_message -> None -> HTTP 200, no dispatch, no log line).
- _allow_from also reads WHATSAPP_CLOUD_ALLOWED_USERS
- dm_policy defaults to allowlist when an allowlist is present (else open)
- _open_dm_opted_in() also honors WHATSAPP_CLOUD_ALLOW_ALL_USERS
Explicit DM_POLICY / ALLOW_FROM still win -> backward compatible.
_do_reconnect() succeeded but never called
YuanbaoAdapter.set_active(adapter), leaving get_active()
permanently returning None after any WS disconnect/reconnect
cycle. This caused cron delivery to silently fail because
_send_yuanbao() checks get_active_adapter() and gives up
immediately when it returns None.
Fix: call set_active(adapter) after successful reconnect,
matching the pattern in connect().
Fixes#58363
api_server already caps every read via client_max_size (chunked
included), but when the limit tripped mid-read the handler's broad JSON
except turned it into 400 'Invalid JSON'. Catch
HTTPRequestEntityTooLarge in body_limit_middleware and return the
OpenAI-style 413.
Status-code polish extracted from PR #3949 by @Gutslabs — the PR's core
client_max_size change already exists on main.
The webhook adapter enforced max_body_bytes only via the Content-Length
header; a Transfer-Encoding: chunked request (content_length=None) or a
spoofed small Content-Length bypassed the cap entirely and read the full
body (bounded only by aiohttp's implicit 1 MiB default, above any
operator-configured smaller limit).
- web.Application(client_max_size=max_body_bytes): aiohttp enforces the
cap on every read path, chunked included
- catch HTTPRequestEntityTooLarge -> 413 (was swallowed into generic 400)
- post-read length re-check as defense in depth
- chunked-upload regression test
Manual port of PR #3955 by @Gutslabs onto current main (handler had
been restructured since); authorship preserved.
_resolve_media_to_data_urls's ad-hoc _MEDIA_TAG_RE matched any bare
token after MEDIA: (no absolute-path anchor) and read the resolved
path directly with no denylist. A relative/traversal path like
MEDIA:../../../../etc/passwd.png slipped through, and any image-
suffixed file the process could read (including under ~/.ssh, ~/.aws,
etc.) was base64-inlined into the API response if its path merely
appeared in the model's own final reply text.
Every other platform adapter's MEDIA: handling already goes through
two shared primitives in gateway/platforms/base.py:
- MEDIA_TAG_CLEANUP_RE, which anchors the path to ~/, /, or a
Windows drive letter plus a known deliverable extension.
- validate_media_delivery_path, which resolves symlinks and rejects
paths under the credential/system-path denylist.
Reuse both here instead of the local unanchored pattern and naive
Path().expanduser() resolution.
The merged webhook session-close fix (#57370, salvaging #57322) wrapped
handle_message in a try/finally — but BasePlatformAdapter.handle_message
is fire-and-forget: it spawns _process_message_background and returns
before the agent run starts. The finally-close therefore ran BEFORE
get_or_create_session created the session row, found no session_id, and
silently no-op'd — the ghost-session leak persisted on the real path.
(The shipped test masked this by stubbing handle_message with a fake
that created the row synchronously.)
Move the close to an on_processing_complete override — the lifecycle
hook the base class fires at the TRUE end of the run, on the success,
failure, and cancellation paths alike. Empirically verified through the
real fire-and-forget pipeline: before, ended_at stayed NULL; after,
ended_at is set with end_reason=webhook_complete and the row is
prunable.
Tests now stub only the runner-side _message_handler (the seam the live
gateway injects) so handle_message / _process_message_background /
on_processing_complete all run for real; adds an AsyncSessionDB-facade
coverage test for the coroutine-await branch.
Replace the webhook delivery-close path's direct reach into private
SessionStore._entries (which also bypassed the store lock) with a public,
lock-held peek_session_id(session_key) accessor. Mirrors the existing
lookup_by_session_id inverse helper. Keeps a getattr fallback for older
stores / test doubles. Adds a unit test for the accessor.
Webhook deliveries created a unique one-shot session (delivery_id baked into
the session key at gateway/platforms/webhook.py:668) but the adapter fired
handle_message via asyncio.create_task WITHOUT ever ending the session
(webhook.py:713, pre-fix). Nothing else closes it: the gateway caches/expires
the agent per session_key but never calls end_session for the webhook path,
and _end_session_on_close teardown doesn't run for these fire-and-forget tasks.
SessionDB.prune_sessions (hermes_state.py:4965) only deletes rows WHERE
ended_at IS NOT NULL. So every webhook session stayed with ended_at NULL ->
unprunable -> unbounded state.db growth. This was the primary driver of the
SQLite lock-contention gateway outage.
Fix: wrap the delivery in _run_delivery_and_close, which awaits
handle_message and then (in finally, so failures still reap) calls
_end_webhook_session -> SessionDB.end_session(session_id, 'webhook_complete').
This mirrors how cron closes its session with 'cron_complete'
(cron/scheduler.py:3065). end_session is first-reason-wins and no-ops on an
already-ended row, so it never clobbers a compression/agent_close reason.
Adds tests/gateway/test_webhook_session_close.py asserting the invariant
(a completed webhook session has ended_at set + is prunable), including the
error-path case, against a real SessionStore + SessionDB.
Adds a no-code routing layer to the OpenAI-compatible API server so one
Hermes deployment can map different API clients to different
model/provider backends. Clients pick a backend by sending a configured
alias as the OpenAI 'model' field; unmatched values fall back to the
global model. Configured aliases are listed by GET /v1/models.
Precedence (highest first): session /model override > model_routes
route > global config. Route provider credentials resolve through
_resolve_runtime_agent_kwargs_for_provider (same seam as
channel_overrides); per-route api_key/base_url are upstream provider
credential overrides — never caller auth, never logged.
Salvaged and rebased from PR #3176 by @Mibayy onto current main.
Salvage of the surviving piece of #2696 by @tarunravi. The PR's other two
changes (tool progress streaming, SSE None-sentinel fix) were independently
superseded on main by the structured hermes.tool.progress SSE events and the
rewritten queue-drain loop.
Remote OpenAI-compatible frontends can't read server-local file paths, so
MEDIA:<path> tags (browser screenshots, generated images) were dead text.
_resolve_media_to_data_urls() now inlines small (<=5MB) local images as
markdown data URLs across all four response surfaces: chat completions
(non-streaming), session chat, session chat stream final event, and the
Responses API. Non-image, missing, or oversized paths pass through
untouched.
/health/detailed leaked runtime state (gateway state, connected
platforms, active-agent counts, PID, exit reason) with no auth. Gate it
behind the same Bearer auth as other API routes; plain /health stays
open for liveness probes.
Also refuse to start on a placeholder/too-short (<16 char) API_SERVER_KEY
regardless of bind address — a guessable key on a terminal-capable
endpoint is RCE-adjacent even on loopback, since any local process can
reach it. The required-key check was already unconditional; this extends
the strength floor to loopback binds too. Startup guards are hoisted
above app/background-task creation so a rejected start leaves no partial
state.
Salvaged from #44073 (external-surface hardening), split into a focused
PR per maintainer request.
Co-authored-by: Hermes Agent <agent@nousresearch.com>
Aligns runtime behaviour with SECURITY.md 2.6: externally reachable
messaging adapters must fail closed unless access is explicitly
configured. Closes the confirmed multiplex authorization bypass a
secondary profile's open dm/group policy no longer inherits the default
profile's allowlist trust.
- Own-policy adapters (WhatsApp, WeCom, Weixin, QQBot, Yuanbao) default
dm_policy/group_policy to pairing/allowlist instead of open; open now
requires an explicit GATEWAY_ALLOW_ALL_USERS or per-platform allow-all.
- Startup guard (_own_policy_open_startup_violation) refuses to boot when
an enabled adapter is open without the allow-all opt-in; the guard now
runs for every secondary profile in multiplex mode too.
- Profile-aware own-policy authorization: _authorization_adapter /
_adapter_for_source resolve the live adapter via SessionSource.profile,
so _is_user_authorized and the ingress/pairing/busy/queue paths read the
originating profile's adapter policy, not the default profile's.
- Fail-closed intake for Email, Feishu P2P, and Discord (blank-principal
denial, empty-allowlist deny, missing-interaction.user deny).
Salvaged from #44073 (external-surface hardening), split into a focused
gateway-authz PR per maintainer request. Follow-up fix by Hermes Agent:
the Discord slash-auth channel bypass now matches DISCORD_ALLOWED_CHANNELS
by the same name-inclusive keys (id + name + #name + parent) the on_message
scope gate uses, so a name-form channel allowlist authorizes slash
interactions consistently (was id-only, breaking #name matching).
Co-authored-by: Hermes Agent <agent@nousresearch.com>
Add a per-platform `cron_continuable_surface` extra key
(`thread` default | `in_channel`) so a continuable cron job can deliver
FLAT into a Slack channel — no dedicated thread — and still be
replied-to. In `in_channel` mode the scheduler skips the thread-open
branch (leaves `thread_id=None`); the shipped origin-mirror then seeds
the `(slack, chat_id, None)` shared-channel session — the same bucket
`reply_in_thread: false` routes inbound channel replies to — so a plain
channel reply continues the job in context.
Design: specs/cron-inchannel-continuable (D1–D7, F5). Model B
(shared-channel session), NOT anchoring to the delivery `ts` — on Slack
replying to a specific message IS threading, so a `ts` anchor would only
relocate the thread, never deliver true threadless continuable.
- gateway/platforms/base.py: `supports_inchannel_continuable` capability
flag (default False → unsupported platforms fail SAFE to `thread`).
- plugins/platforms/slack/adapter.py: flag=True; `_cron_continuable_surface()`
resolver (coerces to the two-value enum); `_warn_if_inchannel_without_flat_reply`
connect-time warning (D5: warn, not hard-require — the misconfig fails safe).
- gateway/config.py: shared-key bridge line (top-level OR nested config).
- cron/scheduler.py: read the key generically from platform config, gate
the `in_channel` branch on the adapter capability flag, skip thread-open.
No new seed function (reuses the existing mirror — G6).
Pairing (docs): `in_channel` + `reply_in_thread: false` +
`require_mention: false` (or a free-response channel). Missing
`reply_in_thread: false` fails safe to a threaded continuation.
Gateway-side config flag — `/restart` to apply; NO Slack app reinstall.
Tests (from inside the worktree, PYTHONPATH=$PWD):
- +6 cron scheduler tests (in_channel skips thread-open; seeds flat
channel session with thread_id=None; thread-mode regression;
fail-safe on unsupported platform; value coercion). Prove-fail:
removing the `and not in_channel_surface` guard turns the two
load-bearing tests RED; restore → GREEN.
- +10 slack resolver/capability/warning tests; +2 config-bridge tests.
- tests/manual/cron_inchannel_e2e.py: offline E2E driving BOTH real
legs (delivery seed + inbound reply keying) → both converge on
(slack, C, None).
- No regressions: test_slack.py 216 passed alone; broader sweep green
(4 pre-existing cross-file-ordering failures reproduce identically on
pristine origin/main).
Docs: cron.md + slack.md + zh-Hans mirrors of both.
Follow-up to the #55780 dead-target not_found blast-radius fix (merged in
#56225). classify_send_error and is_chat_level_not_found each built their own
lowercased error blob, but divergently: classify_send_error appended the
exception CLASS NAME while is_chat_level_not_found did not. A caller passing
exc= to both could get inconsistent answers on the same failure.
- Extract _error_blob(exc, error_text) as the single source of truth both
classifiers use (str(exc) when non-empty + class name; no stray leading
space).
- Align is_chat_level_not_found's signature to (exc, error_text), matching
classify_send_error, removing the swapped-positional footgun; update the
sole caller and the three tests to keyword form.
- Add a regression guard asserting _error_blob keeps the class name.
Surfaced by the hermes-pr-review Phase 2c structured review of #56225.
#55115 added the dead-target registry so confirmed-dead delivery targets are
short-circuited. Its documented scope (gateway/dead_targets.py) is deliberately
narrow: only *whole-chat* deaths -- the `forbidden` and chat-level `not_found`
(`chat not found`) kinds -- should be recorded; "Thread/topic-level not_found is
NOT recorded here ... a deleted topic does not mean the parent chat is dead."
But the implementation doesn't honor that scope. classify_send_error collapses
chat-level "chat not found" AND thread/message-level not_found ("thread not
found", "topic_deleted", "message_id_invalid", "message to edit/reply not
found") into one "not_found" kind, _DEAD_ERROR_KINDS contains "not_found"
wholesale, and deliver()'s except marks the PARENT chat_id dead. So a single
deleted Telegram topic or edited-away message permanently marks the entire chat
(and every future scheduled / cron / agent delivery to it) dead -- silently. The
adapter self-heal the docstring relies on only covers the non-private-group
thread retry; named-DM-topic and message-level failures propagate to deliver()'s
except and wrongly kill the whole chat.
Add is_chat_level_not_found() (factoring the not_found substrings into chat-level
vs sub-chat-level constants) and gate the delivery dead-path: a "not_found" only
marks the target dead when it is chat-level. classify_send_error's public
contract is unchanged (still returns "not_found" for every shape); only the
mark_dead decision is refined, restoring the registry's documented scope.
Cross-platform: telegram/slack/discord delivery all flow through
classify_send_error -> mark_dead. Adds regression tests through the real
deliver() path plus helper/classifier units.
When two features register a post-delivery callback for the same session
(e.g. background-review release + /goal continuation), the second
registration is composed with the first via a `_chained` wrapper. That
wrapper was `def _chained()` — a sync function calling each callback
via `_prev()` / `_new()` and discarding the return value.
For sync callbacks that's fine. For async callbacks (such as the
`_deliver()` coroutine the /goal feature registers to inject the
continuation prompt) the returned coroutine was silently dropped:
RuntimeWarning: coroutine '_deliver' was never awaited.
Outer invoker in `_handle_message` already checks
`inspect.isawaitable(_post_result)` and awaits — but only sees the
wrapper's return value, which was `None`.
Fix: make `_chained` async, iterate over chained callbacks, await any
that return an awaitable. Outer invoker already handles awaitable
wrappers, so no other change is needed.
Tested:
* Added two regression tests in test_post_delivery_callback_chaining.py
covering an async callback chained behind sync (and vice versa).
* Updated existing chaining tests + test_run_cleanup_progress.py to
await the popped callback when it's awaitable.
* 62 tests pass across the touched suites.
Live-validated on Discord: /goal continuations now arrive after the
first turn's response is delivered (previously silent).
Refs: NousResearch/hermes-agent#31922
When the primary provider's auth fails (expired token / 429 quota cap),
_resolve_runtime_agent_kwargs() falls through to the fallback provider
chain, whose runtime dict carries its own 'model' key. api_server's
_create_agent then did AIAgent(model=model, **runtime_kwargs), colliding
on 'model' and 500ing every /v1/chat/completions request while a fallback
was active. Pop the runtime model and let it override the config model,
mirroring the native gateway path (_resolve_session_agent_runtime).
Salvaged from #35716 by @ryo-solo (earliest submitter); the PR's second
half (Mistral reasoning_content strip) is already handled on main and
dropped.
Co-authored-by: Hermes Agent <noreply@nousresearch.com>
mcp-tokens/ holds live MCP OAuth access tokens (<server>.json) and
dynamically-registered OAuth client credentials (<server>.client.json),
layout per tools/mcp_oauth.py. This is the same credential class as
auth.json/credentials/, which _media_delivery_denied_paths() already
blocks. The write side already denies this dir (file_tools
_check_sensitive_path), but the media-delivery (read/exfil) side did
not, leaving an unpaired half-door.
Without it, a prompt-injection MEDIA: tag emitting
~/.hermes/mcp-tokens/<server>.json would, in default (non-strict)
mode, pass the denylist and exfiltrate a live OAuth bearer token to
the same untrusted channel. Sibling follow-up to commit 4ec0adebe
(config.yaml media-delivery denylist).
mcp-tokens is a directory and _path_under_denied_prefix already does
containment matching, so the whole subtree (.json/.client.json/
.meta.json) is denied, mirroring credentials/.
Inside httpx AsyncClient response event hooks, response.next_request is
often None even for a genuine redirect, so guards keyed on
`if response.is_redirect and response.next_request` silently never fire.
A public URL that 302s to http://169.254.169.254/ was followed anyway,
defeating the pre-flight is_safe_url() check.
Resolve the redirect target from the Location header (via urljoin, so
relative Locations work too), falling back to next_request only when no
Location is present. Extracted as tools.url_safety.redirect_target_from_response
and wired into every SSRF redirect guard:
- gateway/platforms/base.py (shared image + audio download for all platforms)
- tools/vision_tools.py (two download hooks)
- plugins/platforms/slack/adapter.py
Original fix by @zapabob (PR #35940), which targeted the since-refactored
gateway/platforms/slack.py; reconstructed onto the current shared sites and
widened to the whole bug class.
