Follow-up to the salvaged #51657: the conversation loop returns the
retry-exhaustion sentinel as BOTH final_response and error, so the
original detector (which required final_response to be falsy) never
fired on real exhaustion turns — the sentinel text was delivered
verbatim into the channel, exactly the #51628 poisoning vector. Detect
the sentinel echo, blank it before empty-response normalization, and
never suppress a turn whose final_response is genuine model text.
Also: dedupe-guard mock fix in the test fixture (has_platform_message_id
must return False, not a truthy MagicMock) and two guard tests
(real answer never suppressed; interrupted/failed never classified).
Salvaged from #32340 by @LeonSGP43, adapted to the workspace-scoped
status tracking that landed in #63709:
- /stop with no running agent now best-effort clears the platform
status indicator, so a phantom 'is thinking...' left by a gateway
restart or a turn that died without a final send can always be
dismissed (#32295).
- SlackAdapter.stop_typing clears an untracked thread when the caller
names it explicitly in metadata — clearing an unset status is a
harmless no-op on Slack's side. The fallback is skipped when multiple
Slack Connect workspaces track the same channel+thread and no team_id
is given, preserving #63709's cross-workspace safety guarantee.
Phase 3 of relay-channel-context (gateway/agent side, single PR). The
connector (gateway-gateway #122/#123/#124) now attaches read-only
surrounding channel/group context to an addressed relay turn; this wires
the gateway to consume it.
- descriptor.py: additive optional supports_context (default False) on
CapabilityDescriptor. from_json already filters unknown keys, so this is
back-compat both directions within contract_version 1.
- ws_transport.py: _event_from_wire maps the connector's read-only
context[] array into the EXISTING MessageEvent.channel_context field via
a new _render_relay_context() helper — reusing the same read-only
injection path history-backfill uses (run.py prepends channel_context
ahead of the trigger message). Never raises; absent/empty/malformed ->
channel_context unset (byte-identical to today).
- docs/relay-connector-contract.md: document supports_context in the §2
descriptor table (fixes the contract-doc conformance test) + the
context/context_error inbound fields in §3.
- tests: descriptor default/round-trip/forward-compat; _render_relay_context
rendering + malformed-safe; _event_from_wire context->channel_context
mapping + the read-only invariant (trigger text untouched).
RELAY-ONLY: only gateway/relay/* + the shared MessageEvent consumption via
its existing channel_context field. No native adapter touched.
The docs state feedback_buttons requires rich_blocks: true, but
_maybe_blocks rendered full Block Kit whenever feedback_buttons alone
was enabled — implicitly turning on rich-block rendering the user never
opted into. Align the code with the documented contract and add a
regression test.
Two test mocks stubbed the old zero-arg signature; the chokepoint refactor
added an optional model param that call sites now pass. Swept the full test
tree for other stale stubs of the changed functions — the rest use
MagicMock/patch(return_value=...), which tolerate the new arg.
Collapse the six per-surface copies of override-then-global resolution
(CLI startup, gateway, TUI, cron, /model switch, fallback activation)
onto one shared resolve_reasoning_config() in hermes_constants.
Also fixes the gateway resolving reasoning against config model.default
instead of the session's effective model: after a session-only /model
switch, the switched model's override now applies (gateway message paths
pass the resolved session model through _resolve_session_reasoning_config;
/reasoning status reads the session model override).
Cleanup: drop docs/PER_MODEL_REASONING.md (duplicates the website docs
page), drop the change-detector _config_version test (no bump needed —
deep-merge handles new keys), remove a stale plan-reference comment.
Adds chokepoint contract tests (13) and gateway session-effective-model
regression tests (2).
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.
Example:
agent:
reasoning_effort: "medium" # global default
reasoning_overrides:
"openrouter/anthropic/claude-opus-4.5": "xhigh"
"openai/gpt-5": "low"
"claude-sonnet-4.6": "high" # bare model name also works
The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.
Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)
Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
+ _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
+ Re-resolves reasoning_config for the fallback model (best-effort)
Closes#21256 (per-model reasoning_effort defaults).
Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
Remove the pipe-table bypass from _rich_delivery_enabled() so that
rich_messages: false is fully honoured. Previously, pipe tables were
auto-routed to sendRichMessage regardless of the config flag, breaking
delivery on clients without Bot API 10.1 support (AyuGram, Telegram
Web, some desktop clients).
Fixes#53824
prune_old_entries' active-process guard failed open: when
has_active_processes_fn raised, the except block logged at debug and
fell through to the age check, so sessions with live background
processes attached could still be pruned — violating the documented
invariant that such sessions are never dropped. Add a continue so an
exception in the safety check fails safe (the entry is kept).
Commit 6b408e131 fixed the session_key/session_id mismatch in this
same guard but left the exception path failing open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #63309 hang class — gateway stuck at 'Connecting to Telegram
(attempt 1/8)' with no retry, no timeout, for minutes — can only occur
when the event loop thread itself is blocked in a synchronous call:
_await_with_thread_deadline's timer fires off-loop, but its expiry
hand-off (call_soon_threadsafe) still needs the loop to run, and the
gateway's outer wait_for is a pure loop timer. When the loop is pinned,
every layer goes silent simultaneously and the process wedges with no
evidence of where.
Two changes:
1. Loop-blocked watchdog in _await_with_thread_deadline: a second
daemon timer fires one grace period (5s) after the deadline; if the
loop still hasn't processed the expiry, it logs a WARNING from the
timer thread and faulthandler-dumps all thread stacks to stderr —
converting the silent hang into a trace that names the exact
blocking frame. A threading.Event set by the expiry callback (and on
normal exit) keeps completed awaits from ever being misreported.
2. discover_fallback_ips: the system-resolver leg runs
socket.getaddrinfo in a worker thread with no timeout, and
asyncio.gather waited on it unboundedly — a wedged OS resolver
stalled discovery for minutes between the two startup log lines. Its
result only feeds a log message, so it no longer gates discovery:
DoH legs (already client-bounded) are gathered alone and the system
leg is awaited with a _DOH_TIMEOUT cap, best-effort.
Refs #63309
Tests: 3 watchdog regressions (blocked-loop dump fires; responsive-loop
timeout does not; completed await does not) + 2 hung-resolver
regressions (DoH results returned promptly; worst-case seed fallback
stays bounded).
Regression tests for find_latest_gateway_session_for_peer and
SessionStore stale-routing self-heal when end_reason is ws_orphan_reap.
Pin manual approval mode in blocking E2E tests so smart aux-LLM
resolution does not flake CI.
OpenAI lets ChatGPT-plan Codex users bank rate-limit reset credits, but
until now they could only be redeemed from the Codex CLI/app or the
website. This wires the same backend API into Hermes:
- /usage on the openai-codex provider now shows "You have N resets
banked - use /usage reset to activate" (parsed from the
rate_limit_reset_credits field the /usage endpoint already returns).
- New /usage reset subcommand (CLI + gateway) redeems one banked
credit via POST .../rate-limit-reset-credits/consume with a UUID
idempotency key, mirroring codex-rs backend-client semantics
(PathStyle /wham vs /api/codex, ChatGPT-Account-Id header,
reset/nothing_to_reset/no_credit/already_redeemed outcomes).
- Guard: redemption is refused while no rate-limit window is fully
exhausted, since a banked reset restores the FULL 5h + weekly
allowance and spending it early wastes it. /usage reset --force
overrides. Zero banked credits and non-codex providers are refused
with clear messages; nothing_to_reset reports the credit was NOT
spent.
- i18n: new gateway.usage.unknown_subcommand / reset_wrong_provider
keys across all 16 locales; docs updated (cli.md, messaging index).
Tested with unit tests plus a real-socket E2E against a local fake
Codex backend exercising redeem/guard/force and the /usage hint.
Commit 62a76bd3d (feat: make smart approvals the default, #62661)
changed approvals.mode default from "manual" to "smart". The
TestBlockingApprovalE2E tests did not patch the approval mode, so
check_all_command_guards routed through _smart_approve() first —
calling call_llm() which tried all auxiliary providers, failed (no
API keys in test env), and returned "escalate" before falling through
to the gateway blocking path. The LLM failure cascade took longer
than the test's 2.5s wait window (50 × 0.05s), so the notify
callback had not fired yet when assert len(notified) == 1 ran.
Force {"mode": "manual"} via patch in setup_method/teardown_method,
matching the pattern already used by test_blocking_approval_uses_
canonical_timeout in the same class.
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
Closes#63529
Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.
Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.
Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
GatewayRunner._prepare_inbound_message_text's "@" context-reference
block read self._model / self._base_url to resolve the model for
get_model_context_length_async. GatewayRunner never sets either
attribute (copy-pasted from HermesCLI in da44c196b, which does carry
self.model/self.base_url). Every message containing "@" raised
AttributeError inside the try block, silently swallowed by the
surrounding except Exception at debug level, so
preprocess_context_references_async never ran and @file:/@folder:/@diff/
etc. references passed through to the model unexpanded.
Fix: resolve model/provider/base_url via
self._resolve_session_agent_runtime(source=, session_key=,
user_config=), the same session-aware resolution the hygiene
compression block already uses a few hundred lines later in this file.
Also raise the swallow log from debug to warning (with exc_info at
debug) so a future regression here is visible instead of silent.
Keep empty-tail recovery scoped to the current stream segment and bound fallback flood retries. Preserve Telegram's server retry hint without blocking final delivery through a long cooldown.
