test_auto_reset_does_not_recover_session_being_ended asserted the old
end_session('session_reset') call; auto-reset now writes through
promote_to_session_reset with the specific auditable reason
('suspended' for a suspended entry). Missed in the local targeted run
because the file wasn't in the touched-suite list; caught by CI shard 6.
Unifies the two gateway subsystems that were fighting each other: the
'never lose a session' recovery machinery (#54878 stale-route self-heal,
find_latest_gateway_session_for_peer reopening agent_close/ws_orphan_reap
rows) and the session reset/expiry machinery (expiry watcher, /new,
/resume, resume_pending freshness gate).
The unified contract:
- INTENTIONAL boundaries (expiry finalization, auto-reset, /new,
/resume switch) are recorded durably via promote_to_session_reset(),
which upgrades accidental recoverable end_reasons (agent_close,
ws_orphan_reap) to the explicit boundary while preserving other
explicit reasons (compression, etc.). Recovery then correctly refuses
to resurrect them.
- ACCIDENTAL ends (crash, cleanup bug, mistaken reaper) stay
recoverable — genuine crash recovery is untouched.
On top of the cherry-picked contributor commits:
- promote_to_session_reset widened to ws_orphan_reap + parameterized
reason so auto-reset paths stay auditable (idle/daily/suspended/
resume_pending_expired) (#61220, #61993, #63539)
- get_or_create_session auto-reset, reset_session (/new), and
switch_session (/resume) all write through the promote path — the
first-reason-wins end_session no-op could previously leave a reset
session resurrectable behind a stale agent_close row (#61993)
- resume_pending freshness gate now honors session_reset.mode=none:
explicit opt-out of automatic resets also opts out of the zombie
gate (#61052)
- resume recovery note extracted to build_resume_recovery_note() and
made adapter-aware via a new interactive_resume capability flag:
webhook/api_server auto-resume turns now CONTINUE the interrupted
task instead of emitting an unanswerable 'session restored'
acknowledgement that abandoned the work (#57056)
- tests updated to call the real note builder instead of mirroring it
E2E-validated against a real SessionDB + SessionStore in a temp
HERMES_HOME: expiry->agent_close->no-resurrection, /new promote,
crash recovery preserved, mode=none opt-out, routing-table flag sync.
When a gateway session with resume_pending=True is not recovered within the
auto-continue freshness window (e.g. because repeated API calls timed out on a
large context), get_or_create_session correctly creates a new session. However
two gaps existed:
1. The user received no notification — resume_pending_expired fell through the
generic "inactive for Xh" else-branch in run.py, which produces wrong wording
and (for session_reset.mode: none users) is gated on policy.notify that
evaluates to False.
2. The old session was ended in state.db with the hardcoded generic reason
"session_reset", making it impossible to distinguish from a normal idle/daily
reset in post-mortem analysis.
Fix:
- gateway/run.py: add an explicit resume_pending_expired case for the agent
context note ("gateway restart recovery timed out") and the user-facing
notification. Always notify for this reason — like suspended — because the
user had an active session that was silently replaced.
- gateway/session.py: pass auto_reset_reason as the DB end_reason instead of
the hardcoded "session_reset", so all auto-reset paths are auditable.
- tests: extend TestResumePendingExpiredAutoReset in test_session_reset_notify.py
with five new cases that cover the reason, activity flag, DB end_reason,
non-regression of the idle path, and freshness-disabled bypass.
Closes#58933
Co-authored-by: Cursor <cursoragent@cursor.com>
When the #54878 self-healing path drops a stale sessions.json entry, the
fix at gateway/session.py:1765 now checks _should_reset() before falling
through to DB recovery. This test covers the case where the stale entry's
session is overdue under an idle/daily reset policy — it must create a
fresh session, set auto-reset metadata, and NOT call reopen_session().
Address review feedback on #63068:
1. Replace unconditional reopen_session() + end_session() with a
conditional promote_to_session_reset() method in SessionDB.
The new method only promotes live rows or rows ended with
agent_close — explicit boundaries (compression, session_reset,
new_command) are preserved via first-writer-wins semantics.
2. Rewrite tests to use real SessionDB instead of MagicMock:
- 7 unit tests for promote_to_session_reset edge cases
- 3 integration tests verifying the actual recovery contract
in find_latest_gateway_session_for_peer after promotion
Bare /reasoning and /fast now render a native one-tap picker on
picker-capable platforms, with automatic fallback to the text status
card everywhere else — parity with the /model picker UX.
- gateway/slash_commands.py: generic send_choice_picker capability gate
(detected on the adapter type, like send_model_picker); selection and
typed arguments flow through one shared application path so they can
never diverge; choices built from VALID_REASONING_EFFORTS so future
levels appear automatically
- telegram: flat inline-keyboard picker (cp:<idx> callbacks), authorized
users only (same gate as approval buttons)
- discord: ChoicePickerView select menu, auth mirrors ExecApprovalView,
2-minute timeout with expiry edit
- matrix: reaction-based picker; reaction set extended to 12 slots to
fit the full effort ladder + subcommands
- locales: picker_title + choice labels in all 16 languages
- docs: ADDING_A_PLATFORM.md capability table
Closes#61110.
When no agent is active, '/queue <prompt>' previously fell through
dispatch with its raw text intact instead of being treated as a
normal prompt. Rewrite event.text to the bare payload (mirroring the
/steer no-active-agent path just below) and return a usage hint when
the payload is empty.
Salvaged from PR #29290 (queue half only — the /footer mid-run
dispatch half already landed on main via #65521).
Replaces the salvaged tests' dict-config path (which required a
GatewayRunner.__init__ dict-coercion hack — dropped from this salvage;
the second commit on the PR branch existed only to support it) with the
established bare-runner test pattern. Also asserts full argument
passthrough to the inner task.
When multiplex_profiles is true, background tasks spawned by /background
command failed with UnscopedSecretError because _resolve_session_agent_runtime()
was called without a profile secret scope. This fix wraps the task in
_profile_runtime_scope, mirroring the pattern used by _run_agent.
Fixes#60726
Same bug class as the salvaged #65305/#65307: hmac.compare_digest (and
secrets.compare_digest) raise TypeError when given a str containing
non-ASCII characters, and these call sites feed it raw request input.
Compare as UTF-8 bytes everywhere:
- gateway/platforms/msgraph_webhook.py: clientState from request body
- gateway/platforms/whatsapp_cloud.py: hub.verify_token query param +
X-Hub-Signature-256 header (comment claimed 'works on str' — it
doesn't for non-ASCII)
- plugins/platforms/feishu: verification token + x-lark-signature
- plugins/platforms/raft: bridge token header
- plugins/platforms/line: X-Line-Signature
- plugins/platforms/sms: X-Twilio-Signature
- tools/code_execution_tool.py: sandbox RPC token (both loops)
Regression tests for the two gateway-core sites (msgraph, whatsapp).
The fix routes the Svix v1 comparison through _hmac_str_equal too, but the
existing non-ASCII tests only exercised the GitHub/GitLab/generic V1/V2
branches. Add a Svix case (valid svix-id + fresh svix-timestamp so it
reaches the v1,<sig> compare) with a non-ASCII signature, which raised
TypeError before the fix and now rejects cleanly.
_validate_signature backs the public webhook receiver. It compared each
attacker-supplied signature/token header (GitHub X-Hub-Signature-256,
GitLab X-Gitlab-Token, generic X-Webhook-Signature / -V2, and the Svix v1
header) against a computed hex/base64 digest with hmac.compare_digest on
two str values. compare_digest raises TypeError on a str containing
non-ASCII characters, and the header is raw client input on an
unauthenticated endpoint — so any internet client could POST a single
non-ASCII byte in the signature header and raise out of the handler,
returning a 500 instead of a clean 401. Fail-closed, but an on-demand
crash of the request path.
Route all five comparisons through a small _hmac_str_equal() helper that
encodes both sides to UTF-8 bytes before the constant-time compare
(compare_digest has no ASCII restriction on bytes). Semantics are
unchanged for valid signatures; a hostile non-ASCII header now fails
closed with a rejection instead of raising.
Adds regression tests: non-ASCII GitHub/GitLab/generic/V2 signature
headers return False (no raise), and a non-ASCII configured secret still
matches its exact token value.
Also maps drexux0@gmail.com in scripts/release.py AUTHOR_MAP.
_check_auth gates every OpenAI-compatible API server endpoint. It compared
the client's raw bearer token against the configured key with
hmac.compare_digest on two str values. compare_digest raises TypeError on
a str containing non-ASCII characters, and the token comes straight from
the Authorization header — so a request with a single non-ASCII byte in
the key (a stray unicode char, a smart quote, a pasted BOM) crashed the
handler with an unhandled TypeError. Every endpoint calls _check_auth
without a try/except, so the framework turned that into a 500 Internal
Server Error instead of the intended 401 Invalid API key.
Compare as bytes, matching web_server.py's dashboard-token check
(hmac.compare_digest(auth.encode(), expected.encode())). Encoding both
sides keeps the timing-safe comparison and its semantics identical for
valid keys while making a non-ASCII token fail closed with a clean 401.
Adds regression tests: a non-ASCII bearer token returns 401 (no raise),
and a non-ASCII configured key still authenticates against its exact
value.
Follow-up to the truncate_message split-loop floor. Two review points:
- CapabilityDescriptor.from_json trusted the wire max_message_length
verbatim, so a connector advertising 0 ('no limit') — or a buggy one
sending 0/negative — produced a descriptor whose bound flowed straight
into the adapter's MAX_MESSAGE_LENGTH and truncate_message. Normalize
it to the documented 4096 default (mirrors from_platform_entry's
'or 4096' and docs/relay-connector-contract.md), fixing the degenerate
budget at its source rather than only surviving it downstream.
- Document the truncate_message length contract for a budget too small
for one codepoint (max_length=1 with a 2-unit surrogate pair under
utf16_len): the chunk intentionally exceeds max_length by that one
indivisible codepoint, because emitting it whole preserves content
where the alternatives are data loss or an infinite loop.
Tests: from_json normalizes 0 and negative bounds to 4096 and passes a
real positive bound through unchanged; the sub-codepoint budget emits
whole codepoints with no data loss (all emojis preserved) and a chunk
that necessarily exceeds the 1-unit budget.
BasePlatformAdapter.truncate_message() splits an over-length reply into
chunks. When max_length is 0 or 1 (and the content is longer), the split
loop makes no progress and spins forever, appending empty chunks — an
unbounded hang that pins a CPU and grows the chunk list until OOM:
- headroom = max_length - INDICATOR_RESERVE - ... goes negative, and the
< 1 fallback (max_length // 2) is also 0;
- so _cp_limit is 0, the region is empty, no split point is found, and
split_at falls back to _cp_limit (0);
- chunk_body is remaining[:0] = "", remaining never shrinks, loop repeats.
The same stall is reachable under utf16_len (Telegram) whenever the next
char is a surrogate-pair emoji wider than the whole budget, so _cp_limit
maps to 0 codepoints even for max_length >= 2.
A pathological max_length is not hypothetical: the relay capability
descriptor's max_message_length is taken verbatim from the connector
(gateway/relay/descriptor.py from_json) and assigned straight to the
adapter's MAX_MESSAGE_LENGTH (gateway/relay/adapter.py), and 0 is a
documented "no limit" value there.
Guarantee forward progress: floor headroom at 1, and floor the
final split_at at max(1, _cp_limit) so at least one codepoint is always
consumed per iteration. Normal splitting is unaffected (both floors only
bite when the budget is already degenerate).
Adds regression tests that run truncate_message on a worker thread and
fail if it doesn't return: max_length 0/1/2 terminate and preserve every
character, and the utf16 emoji case terminates too.
Rebuilt from PR #61283 onto the /p/<profile>/ routing world (7aa21e336):
_profile_scope(None) now enters the DEFAULT profile's runtime scope when
multiplexing is active instead of returning nullcontext(). api_server is
a port-binding platform living on the default profile, so plain requests
(no /p/ prefix) are the primary path — with fail-closed get_secret they
crashed with UnscopedSecretError on the first credential read (#61276).
All three wrapped call sites (chat-completions executor, /v1/runs agent
construction and _run_sync) inherit the fix through the one seam.
Single-profile gateways keep the no-op. Regression tests ported from
the original PR to the _profile_scope seam.
Fixes#61276
Subset of PR #61985: _make_adapter_auth_check gains a profile_name
parameter and secondary-profile adapters (started in
_start_one_profile_adapters) bind it, so the auth callback's
SessionSource resolves the routed profile's adapter and pairing store
instead of silently falling back to the default profile. This is the
gap left open by the #65629 merge — adapter-internal auth checks (e.g.
Slack thread-context fetch) fire outside the wrapped message handler.
The PR's authz_mixin.py hunks are dropped: main's _auth_env (merged via
PR #65629) already covers the scoped allowlist reads they targeted.
With the mode-conditional check centralized, default (websocket) Feishu
no longer counts as port-binding in the secondary batch report — pin the
fixture to connection_mode=webhook so the test still exercises the
multi-platform report path.
Feishu was unconditionally listed in _PORT_BINDING_PLATFORM_VALUES,
causing the multiplexer to reject ALL Feishu secondary profiles. But
Feishu in websocket mode (the default) uses an outbound WebSocket
connection and does NOT bind an HTTP port — only webhook/callback mode
needs a listener.
Add _platform_binds_port() helper that checks connection-dependent
platforms (currently only Feishu) against their actual config before
raising MultiplexConfigError. Feishu websocket profiles are now allowed;
Feishu webhook profiles still raise as before.
Fixes#52563
Review follow-up: honor source.profile and enter _profile_runtime_scope
only when gateway.multiplex_profiles is on, mirroring the gating in
_run_agent, _reset_notice_session_info, and _resolve_profile_for_key.
When multiplexing is off (the default) a stamped source is ignored and
/profile reports the active profile and default home, byte-identical to
before this PR.
The stamped-source test now enables multiplexing (it previously
exercised the ungated path under the default config), and a new
regression asserts the stamp is ignored when multiplexing is off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a multiplexed gateway the process-level active profile is always the
multiplexer's own (usually "default"), so /profile answered "default" in
every chat regardless of which profile actually served it — making
per-chat persona routing look broken when it was working.
Report source.profile (stamped by the /p/<profile>/ URL prefix, a
per-credential adapter, or a room->profile map) and resolve the
displayed home under that profile's runtime scope, mirroring the scoped
/reset banner (#59003). Unstamped sources fall back to the active
profile and default home, so single-profile gateways are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A bare False from connect() on EADDRINUSE made the gateway reconnect
watcher treat a port conflict as transient and retry forever at the
backoff cap — 1568+ retries over 5 days in a multi-profile production
setup, filling errors.log and leaking 2 ResponseStore fds per retry.
Set a non-retryable fatal error (api_server_port_in_use) in the bind
OSError branch so the platform drops from the reconnect queue;
recover via /platform resume api_server after changing the port.
Re-implementation of #52132 by @msalles1 against the direct-bind path
from #65621 (the pre-probe block their patch targeted no longer
exists). Their production diagnosis and test scenario preserved.
Follow-ups on top of the cherry-picked cluster commits:
- slack: scope-authoritative app-token read — get_secret() with a
narrow UnscopedSecretError fallback to os.getenv. Keeps @kohoj's
correct semantics (scoped profile can never silently inherit the
default profile's Socket Mode app) while fixing the regression where
the default-profile startup loop and background reconnect rebuild,
which call connect() unscoped under multiplex, would raise and
fail-loop. Supersedes the 'or os.getenv' variant from #64461 which
reintroduced the cross-profile fallback leak.
- test: unscoped-multiplex fallback regression test for connect().
- run.py: convert the last legacy self.adapters.get(source.platform)
site (_rename_discord_auto_thread) to _adapter_for_source(source)
so profile-routed Discord sources rename threads on the right
adapter (from #57417's sweep).
- AUTHOR_MAP entry for @aguung.
Partial cherry-pick of a7ffbbff7 from PR #63256: secondary-profile
adapter creation errors no longer abort the whole secondary startup
(try/except around _create_adapter + loud warning on None return), and
Home Assistant's check_ha_requirements() becomes dep-only with the
credential moved to a new validate_ha_config() so secondary profiles
whose HASS_TOKEN lives in the profile secret scope are not silently
dropped by the registry gate.
Telegram diagnostic hunks and profile-label stamping dropped: the
regression they targeted does not exist on current main and they
conflict with the connect() teardown fence.
The single-family pre-probe (_port_is_available) raced the real bind and
reported a lingering TIME_WAIT socket as 'in use', failing gateway
restarts for up to ~60s (#10297). Port the webhook adapter's bind
mechanics (#63711/#65482): delete the probe, bind directly with clean
OSError handling and runner teardown, and scope reuse_address=False to
macOS only so Linux restarts rebind past TIME_WAIT instantly.
Credit to @lrawnsley (#10297) for identifying the TIME_WAIT restart
failure.
Add a telegram.free_response_topics config list of '<chat_id>:<thread_id>'
entries (plus the TELEGRAM_FREE_RESPONSE_TOPICS env bridge) so a single
forum topic can be free-response — the bot replies without a mention —
without opening the whole chat via free_response_chats. A missing
message_thread_id is normalized to the General topic ('1') via
_effective_message_thread_id.
Re-ported from PR #36049 (by @wesleion): the original patched
gateway/platforms/telegram.py, which has since moved to
plugins/platforms/telegram/adapter.py with a second gating site
(_should_observe_unmentioned_group_message) and plugin-hook config
bridging (_apply_yaml_config). Both gating sites now honor
free_response_topics.
Salvaged-from: #36049
load_transcript is now a live-replay restore site that heals alternation
violations on load (#64934), so the old all-user 120-row fixture was
merged into a single message and the precondition len==120 failed. The
fixture was never a valid conversation shape; alternate user/assistant
so it exercises the same bloat scenario without tripping the repair.
Telegram only auto-derives a voice/audio clip's duration from container
metadata for short recordings; clips longer than ~4:50 are delivered with
duration 0 and render as 0:00 in the player. Probe the length locally
(stdlib wave -> mutagen -> ffprobe) and pass duration explicitly to
sendVoice/sendAudio. Best-effort: when nothing can read the file we omit
duration and fall back to Telegram's prior behavior.
Extracts and hardens the Telegram-only part of the stale, Piper-bundled
PR #7815 (ffprobe-only, predates the send_voice retry/anchor refactor);
relates to #8508.
When an agent is running, the gateway runner's running-agent block routes
"session-level toggles that are safe to run mid-agent" through a membership
set before the catch-all that rejects everything else with
"Agent is running — /<cmd> can't run mid-turn".
A dedicated /footer dispatch branch already sat inside that guard, but the
set listed only {"yolo", "verbose"}, so the footer branch was unreachable:
/footer fell through to the catch-all and was rejected, forcing users to
/stop a running agent just to toggle the runtime-metadata footer. /footer
is a pure display toggle like its sibling /verbose — it only writes
display.runtime_footer.enabled and returns a status string — so it belongs
in the same set.
Add "footer" to the set so the existing dispatch branch becomes reachable.
Regression test (tests/gateway/test_footer_command_mid_run.py): asserts
/footer and /footer <arg> dispatch to _handle_footer_command while an agent
is running, with a /verbose parity guard. Verified failing before the fix
(handler awaited 0 times) and passing after.
When gateway.multiplex_profiles is on, the default-profile GatewayRunner
used to call load_gateway_config() unscoped. Platform tokens that lived
only in a profile .env (often a secondary profile) never reached the
primary Telegram adapter, producing "No bot token configured" and an
infinite reconnect watcher loop (#64674).
- Load primary config under the default profile secret scope when multiplex
is enabled (same path secondary adapters already use).
- Skip starting token platforms on the default profile when no credential
is present under multiplex; secondary profiles still connect with their
scoped tokens.
- Drop empty-token configs from the reconnect queue so they cannot spin
forever.
Regression coverage in tests/gateway/test_64674_multiplex_primary_token_scope.py.
Extends @AlexFucuson9's 3-site fix (#58594) across the full adapter:
every logger call and SendResult.error that interpolates a raw PTB
exception now routes through _redact_telegram_error_text(). Covers
polling conflict/retry/network ladders, overflow-split edits, draft
sends, prompt/approval/clarify/picker sends, media send fallbacks,
media cache failures, reactions, and chat-info lookups (48 additional
sites). Telegram Bot API exceptions embed the token in the request URL
(/bot<TOKEN>/<method>), so any raw str(exc) is a leak surface.
Adds regression tests for SendResult.error redaction (update prompt,
clarify) and delete_message debug-log redaction.
MEDIA: tags whose path had an unknown extension (.py, .log, .toml,
.weirdext, ...) fell between both extraction passes: the anchored
extension allowlist (MEDIA_TAG_CLEANUP_RE) did not match them, and the
extension-less pass explicitly skipped any path that HAD a suffix. The
file was never delivered even though the intended design (universal
ingress/egress) says any non-credential file should ship.
Widen _path_lacks_deliverable_extension() so the validated delivery
pass (MEDIA_EXTENSIONLESS_TAG_RE + validate_media_delivery_path)
covers every path the extension allowlist does not — unknown
extensions and extension-less files alike. Security posture is
unchanged: unknown-extension paths only deliver after full validation
(exists, symlinks resolved, credential/system denylist, strict-mode
allowlist+recency), and unvalidated tags stay visible in the text
instead of being silently dropped. Known extensions keep their
unconditional pre-existing behavior.
Because extract_media, _strip_media_tag_directives (non-streaming
dispatch), and strip_media_directives_for_display (streaming) all
share the same two regexes + predicate, all delivery paths pick up the
widened behavior with no per-site changes. Dispatch partition in
gateway/run.py already routes non-image/video extensions through
send_document.
Closes the gap reported in PR #36060; supersedes the allowlist-append
approach there (an extension allowlist can never enumerate every file
type a user asks the agent to produce).
Co-authored-by: Randimt <randimt@users.noreply.github.com>
Disable address reuse so an existing family-specific listener cannot silently split traffic with the webhook server. Normalize wildcard bind hosts for local CLI URLs and align setup documentation with the dual-stack default.
The webhook adapter defaulted to host='0.0.0.0' — IPv4 only. On Fly.io
hosted agents the edge router (hermes-agent-router) reverse-proxies public
webhook traffic to <app>.internal:8644 over 6PN, Fly's private network,
which is IPv6-only (.internal resolves to an fdaa:… address). An IPv4-only
listener is unreachable there, so public webhook POSTs to
https://<agent>.agents.nousresearch.com/webhooks/<route> never landed on
the adapter — the router's dial was refused.
Fix: DEFAULT_HOST = None, which makes aiohttp/asyncio create_server bind
BOTH address families. '::' is NOT a valid substitute: on hosts where the
kernel sets bindv6only=1 (verified on Fly machines) it yields an IPv6-only
socket, breaking the IPv4 loopback /health check and the AF_INET
port-conflict probe in connect(). None binds per-family regardless of the
sysctl. An explicit empty-string/null host in config now also normalises to
None (dual-stack) rather than an invalid host=''. Users can still pin a
specific host via platforms.webhook.extra.host.
Validated live on a Fly staging agent: with this default and no config
override, the adapter binds both v4 and v6 (127.0.0.1:8644 and [::1]:8644
both answer), and a public signed webhook POST through the router returns
202 (valid sig) / 401 (bad sig) instead of the router's 502.
Tests: new TestDualStackBind asserts the None default, config resolution
(missing/empty→None, pinned preserved), and a real dual-stack bind opens
both AF_INET and AF_INET6 listeners. Red-proof: these fail on the old
'0.0.0.0' default.
