The served-profiles block in _start_secondary_profile_adapters references
PairingStore, but the class's only import in gateway/run.py is method-local
inside __init__ — so the reference raised NameError at runtime, silently
swallowed by the enclosing try/except ('could not record served_profiles').
Result: multiplexing gateways never created per-profile pairing stores, and
authz pairing checks for secondary profiles fell through to the global
whitelist. Also masked the served_profiles runtime-status write.
One-line fix (local import alongside write_runtime_status) + regression
tests that drive the real method and assert the stores materialize,
verified red without the import and green with it.
Surfaced during the profile-routing sweep by @CocaKova's PR #61689, which
included the same fix as part of a larger feature.
The #54878 self-heal (SessionStore.get_or_create_session) drops a routing
key pointing at a session already ended in state.db and recovers/recreates
a fresh session_id under the same session_key. The #54947 fix (agent-cache
cache-hit guard in gateway/run.py) treats a cached agent whose snapshot
session_id differs from the current session_id, under the same
session_key, as an intentional /resume-/branch-style switch between two
live sibling conversations, and reuses it unchanged to protect the prompt
cache.
These two fixes compose incorrectly: when the #54878 self-heal just fired,
the cached agent's session_id is not a live sibling — it's the dead session
just routed away from. #54947's "different session_id -> reuse freely" rule
reuses it anyway. The stale agent runs the turn, and the post-run "session
split" sync (agent.session_id != session_id) then writes the routing key
straight back onto the dead session_id, undoing the self-heal. This repeats
on every subsequent message until an interrupt (e.g. /stop) happens to race
in before that post-run sync, silently discarding conversation context.
Reproduced live on the engineering gateway (2026-07-12, routing key
agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines
over ~40 minutes, each followed by the dead session_id being reused and
re-synced back, until an interrupted /stop finally let a fresh session
stick — at which point all prior context was gone.
No open upstream issue tracks this specific interaction as of 2026-07-12
(checked #54878, #54947, #59580, #59597, #61220 — all cover adjacent but
distinct edges of the self-heal / agent-cache system).
Fix: before applying #54947's reuse-on-mismatch rule, check (outside the
cache lock, via SessionStore._is_session_ended_in_db) whether the cached
snapshot's session_id is itself ended in state.db. If so, treat it as a
stale self-heal artifact and evict/rebuild fresh -- same as a genuine
cross-process write -- instead of reusing it. Re-validates the peeked
verdict against the tuple actually held under the lock so a race can't
apply a stale verdict to a different (possibly live) cache entry.
Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new
cases: dead-session eviction, live-sibling reuse preserved [#54947 intact],
cross-process invalidation preserved [#45966 intact], same-session_id dead
edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed,
9040 passed, 11 skipped -- all 14 failures verified pre-existing on
unpatched main (confirmed via git stash + re-run), unrelated to this
change.
GatewayRunner._prepare_inbound_message_text() interpolated
source.user_name — the platform-supplied, user-settable display name —
directly into the message text of every turn in a shared multi-user
session: f"[{source.user_name}] {message_text}". Shared sessions are the
default for any threaded conversation (thread_sessions_per_user defaults
to False) and apply to any group with group_sessions_per_user=False, so
no special configuration is needed to reach this path.
An unescaped display name containing embedded newlines could therefore
masquerade as a new markdown section (a fake "## Override" heading)
inside the live conversation the model reads on every turn — the same
indirect-prompt-injection vector gateway/session.py's
build_session_context_prompt() already guards against for the identical
user_name field via _format_untrusted_prompt_value(), which was never
applied to this sibling call site.
Add neutralize_untrusted_inline_text() alongside the existing helper in
gateway/session.py: it collapses embedded newlines/control characters to
a single inert line without JSON-quoting, so inline "[Name] message"
formatting is preserved byte-for-byte for the common case (unlike
reusing _format_untrusted_prompt_value directly, which would add visible
quote marks to every sender prefix). Wire it into the sender-prefix
construction in gateway/run.py.
Follow-ups on the salvaged #20096 profile-routing feature:
- _profile_name_for_source now returns None unless gateway.multiplex_profiles
is on. Routing stamps source.profile, which namespaces session/batch keys,
but the profile-scoped agent run only activates under multiplexing — without
the gate, configured routes with multiplexing off split batch/session keys
into agent:<profile> while the agent still ran from agent:main.
- Widen the profile-aware _text_batch_key fix from Discord to every adapter
that builds batch keys via build_session_key (telegram, whatsapp, matrix,
feishu, wecom, weixin) — routing is platform-generic, so the batch-key
namespace fix must be too.
- Downgrade the no-route-matched log from INFO to DEBUG (fired on every
unrouted inbound message).
- GatewayConfig.to_dict(): serialize profile_routes as plain dicts
(ProfileRoute dataclasses are not JSON-safe).
- Docs: correct the 'independent of multiplexing' claim in
docs/profile-routing.md (routing requires multiplexing), fix the
platform-only specificity row (0, not 1), and document profile_routes in
website/docs/user-guide/multi-profile-gateways.md.
- Tests: pin the multiplex gate (routes ignored when off, active when on,
build_source end-to-end stays in agent:main when off).
Completes the review's ask for "adapter-to-session-key integration coverage
for Discord and a non-Discord platform" on #20096.
Drives a concrete adapter's real BasePlatformAdapter.build_source with an
injected gateway_runner, asserts the matched route's profile is stamped on
the source, and that build_session_key scopes the key under agent:<profile>:
(versus the shared agent:main: namespace). Covers Discord and Telegram — the
Telegram case is the bug-#2 path that previously fell through to default.
Adds a regression anchor: without gateway_runner, profile stays None and the
key lands in agent:main (the silent fallback the fix removes for non-Discord).
Co-Authored-By: Claude <noreply@anthropic.com>
Addresses hermes-sweeper review on #20096.
Problem 1 (profile_routing.py): route matching returned True on a chat_id
hit before the guild_id constraint was consulted, so a route declaring both
guild_id and chat_id matched on chat_id alone. Restored conjunctive (AND)
semantics — every declared discriminator must hold; hierarchical parent_chat_id
matching is preserved. Added a regression test for the guild+chat case.
Problem 2 (base.py / run.py): gateway_runner was injected only when an adapter
pre-declared the attribute, and only Discord did — so build_source never called
_profile_name_for_source for Telegram/Feishu/Slack/etc., despite the
platform-generic claim. Declared gateway_runner on BasePlatformAdapter and made
the plugin-registry injection unconditional, so profile routing now reaches
every platform. Added non-Discord (Telegram) resolution coverage and an
injection-inheritance test.
Also adds docs/profile-routing.md documenting gateway.profile_routes
(matching rules, specificity, profile isolation) — requested in review.
Co-Authored-By: Claude <noreply@anthropic.com>
load_gateway_config only forwarded the top-level multiplex_profiles
key, ignoring the nested gateway.multiplex_profiles form. The latter
is what `hermes config set gateway.multiplex_profiles true` writes,
so users who ran that command got multiplex_profiles=False silently
— no warning, no fallback, profile_routes just stopped matching.
Loader now checks the top-level key first, falls back to the nested
gateway section, and only then defaults to False. Same precedence is
applied to other nested-form keys (profile_routes already did this).
Tests cover: top-level honored, nested honored (regression test for
the silent-fallback bug), default False, top-level overrides nested.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_adapter_credential_fingerprint only looked at adapter.token directly,
but Discord (and similar) adapters store the bot token on their config
sub-object, not on self. Every Discord adapter in a multiplexed
gateway therefore returned None, the same-token conflict check was
silently skipped, and N adapters all polled the same bot token —
producing a per-message race where whichever adapter won the GIL
answered the user.
Adds a config-token fallback (token, then bot_token) so the check
actually fires for config-backed adapters. Direct adapter.token
still takes precedence when both exist.
Tests cover: config-backed token produces a fingerprint, distinct
tokens produce distinct fingerprints, direct token wins over config,
config without token attributes returns None.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three follow-ups to the initial routing PR after code review:
1. Remove dead forum-post hierarchy cache. The `_forum_post_cache`,
`register_forum_post()`, and `resolve_forum_channel()` were never
wired up — no caller in the codebase. Discord's adapter already
sets `parent_chat_id` to the immediate parent (forum channel for a
forum post), so the existing `self.chat_id == parent_chat_id`
branch in `matches()` handles forum posts correctly without a
cache. The hierarchical-resolution branch in `matches()` and the
bounded-LRU infrastructure are removed.
2. Fix docstring specificity numbers (8 → 14, 4 → 6) and rewrite the
"Hierarchical matching" section to describe the actual one-level
parent_chat_id behavior. Removed unused `OrderedDict` and `Set`
imports.
3. Warn loudly when a routed profile doesn't exist on disk. Previously,
a typo in `profile_routes` (e.g. `crypto-tradr`) silently fell back
to the global HERMES_HOME, causing the message to read the default
profile's memory/credentials with no signal to the operator. Now
emits a `logger.warning` with the profile name, source identifier,
and the fallback reason. Bare-exception path also gets `exc_info`.
Tests:
- test_profile_routing.py: +2 tests verifying forum post matching via
direct parent_chat_id (covers the case the removed cache was meant
for). 31 total, all pass.
- test_profile_resolution.py: NEW, 12 tests covering resolution order
(source.profile > routing > active > default), missing-profile
warning, exception handling, and routing consultation. All pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds gateway.profile_routes config that routes specific Discord
guilds/channels/threads (and other platforms) to different profiles.
The routing engine uses hierarchical specificity matching
(thread > channel > guild) with bounded LRU caching for forum post
resolution.
Routing result is stamped on source.profile by BasePlatformAdapter
.build_source() at inbound time. When gateway.multiplex_profiles is on,
the existing _profile_runtime_scope machinery picks up source.profile
and runs the whole turn inside the profile's HERMES_HOME — so memory,
skills, config, and secrets all resolve to that profile automatically.
No new isolation code is added; this PR only adds the routing decision
layer on top of the existing multiplexing infrastructure.
Configuration:
gateway:
multiplex_profiles: true
profile_routes:
- name: server-default
platform: discord
guild_id: "GUILD_ID"
profile: server-profile
- name: special-channel
platform: discord
guild_id: "GUILD_ID"
chat_id: "CHANNEL_ID"
profile: channel-profile
When multiplex_profiles is off, profile_routes is ignored (no behavior
change for single-profile gateways).
Tests: 29 unit tests covering specificity scoring, hierarchical
matching, path-traversal validation, and config parsing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Follow-up to the salvaged #51657: the conversation loop returns the
retry-exhaustion sentinel as BOTH final_response and error, so the
original detector (which required final_response to be falsy) never
fired on real exhaustion turns — the sentinel text was delivered
verbatim into the channel, exactly the #51628 poisoning vector. Detect
the sentinel echo, blank it before empty-response normalization, and
never suppress a turn whose final_response is genuine model text.
Also: dedupe-guard mock fix in the test fixture (has_platform_message_id
must return False, not a truthy MagicMock) and two guard tests
(real answer never suppressed; interrupted/failed never classified).
Salvaged from #32340 by @LeonSGP43, adapted to the workspace-scoped
status tracking that landed in #63709:
- /stop with no running agent now best-effort clears the platform
status indicator, so a phantom 'is thinking...' left by a gateway
restart or a turn that died without a final send can always be
dismissed (#32295).
- SlackAdapter.stop_typing clears an untracked thread when the caller
names it explicitly in metadata — clearing an unset status is a
harmless no-op on Slack's side. The fallback is skipped when multiple
Slack Connect workspaces track the same channel+thread and no team_id
is given, preserving #63709's cross-workspace safety guarantee.
Phase 3 of relay-channel-context (gateway/agent side, single PR). The
connector (gateway-gateway #122/#123/#124) now attaches read-only
surrounding channel/group context to an addressed relay turn; this wires
the gateway to consume it.
- descriptor.py: additive optional supports_context (default False) on
CapabilityDescriptor. from_json already filters unknown keys, so this is
back-compat both directions within contract_version 1.
- ws_transport.py: _event_from_wire maps the connector's read-only
context[] array into the EXISTING MessageEvent.channel_context field via
a new _render_relay_context() helper — reusing the same read-only
injection path history-backfill uses (run.py prepends channel_context
ahead of the trigger message). Never raises; absent/empty/malformed ->
channel_context unset (byte-identical to today).
- docs/relay-connector-contract.md: document supports_context in the §2
descriptor table (fixes the contract-doc conformance test) + the
context/context_error inbound fields in §3.
- tests: descriptor default/round-trip/forward-compat; _render_relay_context
rendering + malformed-safe; _event_from_wire context->channel_context
mapping + the read-only invariant (trigger text untouched).
RELAY-ONLY: only gateway/relay/* + the shared MessageEvent consumption via
its existing channel_context field. No native adapter touched.
The docs state feedback_buttons requires rich_blocks: true, but
_maybe_blocks rendered full Block Kit whenever feedback_buttons alone
was enabled — implicitly turning on rich-block rendering the user never
opted into. Align the code with the documented contract and add a
regression test.
Two test mocks stubbed the old zero-arg signature; the chokepoint refactor
added an optional model param that call sites now pass. Swept the full test
tree for other stale stubs of the changed functions — the rest use
MagicMock/patch(return_value=...), which tolerate the new arg.
Collapse the six per-surface copies of override-then-global resolution
(CLI startup, gateway, TUI, cron, /model switch, fallback activation)
onto one shared resolve_reasoning_config() in hermes_constants.
Also fixes the gateway resolving reasoning against config model.default
instead of the session's effective model: after a session-only /model
switch, the switched model's override now applies (gateway message paths
pass the resolved session model through _resolve_session_reasoning_config;
/reasoning status reads the session model override).
Cleanup: drop docs/PER_MODEL_REASONING.md (duplicates the website docs
page), drop the change-detector _config_version test (no bump needed —
deep-merge handles new keys), remove a stale plan-reference comment.
Adds chokepoint contract tests (13) and gateway session-effective-model
regression tests (2).
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.
Example:
agent:
reasoning_effort: "medium" # global default
reasoning_overrides:
"openrouter/anthropic/claude-opus-4.5": "xhigh"
"openai/gpt-5": "low"
"claude-sonnet-4.6": "high" # bare model name also works
The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.
Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)
Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
+ _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
+ Re-resolves reasoning_config for the fallback model (best-effort)
Closes#21256 (per-model reasoning_effort defaults).
Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
Remove the pipe-table bypass from _rich_delivery_enabled() so that
rich_messages: false is fully honoured. Previously, pipe tables were
auto-routed to sendRichMessage regardless of the config flag, breaking
delivery on clients without Bot API 10.1 support (AyuGram, Telegram
Web, some desktop clients).
Fixes#53824
prune_old_entries' active-process guard failed open: when
has_active_processes_fn raised, the except block logged at debug and
fell through to the age check, so sessions with live background
processes attached could still be pruned — violating the documented
invariant that such sessions are never dropped. Add a continue so an
exception in the safety check fails safe (the entry is kept).
Commit 6b408e131 fixed the session_key/session_id mismatch in this
same guard but left the exception path failing open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #63309 hang class — gateway stuck at 'Connecting to Telegram
(attempt 1/8)' with no retry, no timeout, for minutes — can only occur
when the event loop thread itself is blocked in a synchronous call:
_await_with_thread_deadline's timer fires off-loop, but its expiry
hand-off (call_soon_threadsafe) still needs the loop to run, and the
gateway's outer wait_for is a pure loop timer. When the loop is pinned,
every layer goes silent simultaneously and the process wedges with no
evidence of where.
Two changes:
1. Loop-blocked watchdog in _await_with_thread_deadline: a second
daemon timer fires one grace period (5s) after the deadline; if the
loop still hasn't processed the expiry, it logs a WARNING from the
timer thread and faulthandler-dumps all thread stacks to stderr —
converting the silent hang into a trace that names the exact
blocking frame. A threading.Event set by the expiry callback (and on
normal exit) keeps completed awaits from ever being misreported.
2. discover_fallback_ips: the system-resolver leg runs
socket.getaddrinfo in a worker thread with no timeout, and
asyncio.gather waited on it unboundedly — a wedged OS resolver
stalled discovery for minutes between the two startup log lines. Its
result only feeds a log message, so it no longer gates discovery:
DoH legs (already client-bounded) are gathered alone and the system
leg is awaited with a _DOH_TIMEOUT cap, best-effort.
Refs #63309
Tests: 3 watchdog regressions (blocked-loop dump fires; responsive-loop
timeout does not; completed await does not) + 2 hung-resolver
regressions (DoH results returned promptly; worst-case seed fallback
stays bounded).
Regression tests for find_latest_gateway_session_for_peer and
SessionStore stale-routing self-heal when end_reason is ws_orphan_reap.
Pin manual approval mode in blocking E2E tests so smart aux-LLM
resolution does not flake CI.
OpenAI lets ChatGPT-plan Codex users bank rate-limit reset credits, but
until now they could only be redeemed from the Codex CLI/app or the
website. This wires the same backend API into Hermes:
- /usage on the openai-codex provider now shows "You have N resets
banked - use /usage reset to activate" (parsed from the
rate_limit_reset_credits field the /usage endpoint already returns).
- New /usage reset subcommand (CLI + gateway) redeems one banked
credit via POST .../rate-limit-reset-credits/consume with a UUID
idempotency key, mirroring codex-rs backend-client semantics
(PathStyle /wham vs /api/codex, ChatGPT-Account-Id header,
reset/nothing_to_reset/no_credit/already_redeemed outcomes).
- Guard: redemption is refused while no rate-limit window is fully
exhausted, since a banked reset restores the FULL 5h + weekly
allowance and spending it early wastes it. /usage reset --force
overrides. Zero banked credits and non-codex providers are refused
with clear messages; nothing_to_reset reports the credit was NOT
spent.
- i18n: new gateway.usage.unknown_subcommand / reset_wrong_provider
keys across all 16 locales; docs updated (cli.md, messaging index).
Tested with unit tests plus a real-socket E2E against a local fake
Codex backend exercising redeem/guard/force and the /usage hint.
Commit 62a76bd3d (feat: make smart approvals the default, #62661)
changed approvals.mode default from "manual" to "smart". The
TestBlockingApprovalE2E tests did not patch the approval mode, so
check_all_command_guards routed through _smart_approve() first —
calling call_llm() which tried all auxiliary providers, failed (no
API keys in test env), and returned "escalate" before falling through
to the gateway blocking path. The LLM failure cascade took longer
than the test's 2.5s wait window (50 × 0.05s), so the notify
callback had not fired yet when assert len(notified) == 1 ran.
Force {"mode": "manual"} via patch in setup_method/teardown_method,
matching the pattern already used by test_blocking_approval_uses_
canonical_timeout in the same class.
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
Closes#63529
Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.
Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.
Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
