prune_old_entries' active-process guard failed open: when
has_active_processes_fn raised, the except block logged at debug and
fell through to the age check, so sessions with live background
processes attached could still be pruned — violating the documented
invariant that such sessions are never dropped. Add a continue so an
exception in the safety check fails safe (the entry is kept).
Commit 6b408e131 fixed the session_key/session_id mismatch in this
same guard but left the exception path failing open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #63309 hang class — gateway stuck at 'Connecting to Telegram
(attempt 1/8)' with no retry, no timeout, for minutes — can only occur
when the event loop thread itself is blocked in a synchronous call:
_await_with_thread_deadline's timer fires off-loop, but its expiry
hand-off (call_soon_threadsafe) still needs the loop to run, and the
gateway's outer wait_for is a pure loop timer. When the loop is pinned,
every layer goes silent simultaneously and the process wedges with no
evidence of where.
Two changes:
1. Loop-blocked watchdog in _await_with_thread_deadline: a second
daemon timer fires one grace period (5s) after the deadline; if the
loop still hasn't processed the expiry, it logs a WARNING from the
timer thread and faulthandler-dumps all thread stacks to stderr —
converting the silent hang into a trace that names the exact
blocking frame. A threading.Event set by the expiry callback (and on
normal exit) keeps completed awaits from ever being misreported.
2. discover_fallback_ips: the system-resolver leg runs
socket.getaddrinfo in a worker thread with no timeout, and
asyncio.gather waited on it unboundedly — a wedged OS resolver
stalled discovery for minutes between the two startup log lines. Its
result only feeds a log message, so it no longer gates discovery:
DoH legs (already client-bounded) are gathered alone and the system
leg is awaited with a _DOH_TIMEOUT cap, best-effort.
Refs #63309
Tests: 3 watchdog regressions (blocked-loop dump fires; responsive-loop
timeout does not; completed await does not) + 2 hung-resolver
regressions (DoH results returned promptly; worst-case seed fallback
stays bounded).
Regression tests for find_latest_gateway_session_for_peer and
SessionStore stale-routing self-heal when end_reason is ws_orphan_reap.
Pin manual approval mode in blocking E2E tests so smart aux-LLM
resolution does not flake CI.
OpenAI lets ChatGPT-plan Codex users bank rate-limit reset credits, but
until now they could only be redeemed from the Codex CLI/app or the
website. This wires the same backend API into Hermes:
- /usage on the openai-codex provider now shows "You have N resets
banked - use /usage reset to activate" (parsed from the
rate_limit_reset_credits field the /usage endpoint already returns).
- New /usage reset subcommand (CLI + gateway) redeems one banked
credit via POST .../rate-limit-reset-credits/consume with a UUID
idempotency key, mirroring codex-rs backend-client semantics
(PathStyle /wham vs /api/codex, ChatGPT-Account-Id header,
reset/nothing_to_reset/no_credit/already_redeemed outcomes).
- Guard: redemption is refused while no rate-limit window is fully
exhausted, since a banked reset restores the FULL 5h + weekly
allowance and spending it early wastes it. /usage reset --force
overrides. Zero banked credits and non-codex providers are refused
with clear messages; nothing_to_reset reports the credit was NOT
spent.
- i18n: new gateway.usage.unknown_subcommand / reset_wrong_provider
keys across all 16 locales; docs updated (cli.md, messaging index).
Tested with unit tests plus a real-socket E2E against a local fake
Codex backend exercising redeem/guard/force and the /usage hint.
Commit 62a76bd3d (feat: make smart approvals the default, #62661)
changed approvals.mode default from "manual" to "smart". The
TestBlockingApprovalE2E tests did not patch the approval mode, so
check_all_command_guards routed through _smart_approve() first —
calling call_llm() which tried all auxiliary providers, failed (no
API keys in test env), and returned "escalate" before falling through
to the gateway blocking path. The LLM failure cascade took longer
than the test's 2.5s wait window (50 × 0.05s), so the notify
callback had not fired yet when assert len(notified) == 1 ran.
Force {"mode": "manual"} via patch in setup_method/teardown_method,
matching the pattern already used by test_blocking_approval_uses_
canonical_timeout in the same class.
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
Closes#63529
Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.
Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.
Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
GatewayRunner._prepare_inbound_message_text's "@" context-reference
block read self._model / self._base_url to resolve the model for
get_model_context_length_async. GatewayRunner never sets either
attribute (copy-pasted from HermesCLI in da44c196b, which does carry
self.model/self.base_url). Every message containing "@" raised
AttributeError inside the try block, silently swallowed by the
surrounding except Exception at debug level, so
preprocess_context_references_async never ran and @file:/@folder:/@diff/
etc. references passed through to the model unexpanded.
Fix: resolve model/provider/base_url via
self._resolve_session_agent_runtime(source=, session_key=,
user_config=), the same session-aware resolution the hygiene
compression block already uses a few hundred lines later in this file.
Also raise the swallow log from debug to warning (with exc_info at
debug) so a future regression here is visible instead of silent.
Keep empty-tail recovery scoped to the current stream segment and bound fallback flood retries. Preserve Telegram's server retry hint without blocking final delivery through a long cooldown.
The second lock block in get_or_create_session held self._lock during six
blocking operations on every inbound message: _is_session_ended_in_db
(SQLite SELECT), _should_reset (callback), _save (SQLite write + JSON write
+ os.fsync), and _recover_session_from_db (SQLite SELECT + UPDATE).
A code comment at line 1607 claimed 'SQLite calls are made outside the
lock' -- true only for _compression_tip_for_session_id, which was moved
out in a prior fix. The remaining I/O was never addressed.
Restructure into a four-phase lock/no-lock split that mirrors the pattern
already established at the bottom of the function:
Phase 1 (lock) -- read entry + session_id
Phase 1b (no lock) -- stale check + reset policy
Phase 2 (lock) -- apply decisions to _entries, capture snapshot + flags
Phase 3 (no lock) -- recovery DB query, _save from snapshot, end/create
_save_entries(snapshot) replaces _save() to avoid dict-mutation races when
called outside the lock. _query_recoverable_session splits the DB I/O out
of _recover_session_from_db so only the _entries assignment needs the lock.
Three early returns inside the lock block are eliminated in favour of a
unified save + return path.
The sync _session_has_compression_in_flight sat on the message hot path
and blocked the event loop twice: under session_store._lock during
_ensure_loaded_locked (JSON read) and via db.get_compression_lock_holder
(SQLite SELECT). Async-ify the method and offload both sources via
asyncio.to_thread; await the call site in _handle_active_session_busy_message.
Without this UA tag the Feishu server does not push group @mention events
over the WebSocket transport. The "channel" tag tells the server to use
the Channel protocol which enables group-message routing in addition to P2P
direct messages.
Root cause: FeishuWSClient was created without any UA signaling tag, so the
server defaulted to the basic DM-only push mode. Group @mention events were
silently dropped before reaching Hermes.
Fixes https://github.com/NousResearch/hermes-agent/issues/50656
Also adds a regression test verifying the UA tag is present in the
FeishuWSClient constructor call.
The gateway reconnect watcher forwards is_reconnect=True to every
adapter.connect() call on every retry. Adapters whose signature omits
the kwarg raise TypeError at every reconnect attempt and stay silently
disconnected — the exact bug that shipped for QQAdapter and only
surfaced after messages stopped flowing on the QQ channel for hours.
This test statically parses every adapter.py under gateway/platforms/
and plugins/platforms/ (via AST, so third-party SDKs like slack_sdk,
matrix-nio, aiohttp, telegram, etc. are NOT required in the test env)
and asserts every *Adapter class with an async connect() accepts
is_reconnect — either as a keyword-only argument or absorbed by
**kwargs.
Also fixes plugins/platforms/wecom/callback_adapter.py:WecomCallbackAdapter,
which the new test caught as a second offender. Same class of bug: bare
'async def connect(self)' signature would die on the first reconnect.
Companion to #59429 (which fixed the original QQAdapter offender).
The base adapter's signature was updated to include
, which the reconnect watcher passes as
during reconnection. All other platform adapters were
updated, but QQAdapter was missed, causing:
TypeError: QQAdapter.connect() got an unexpected keyword argument 'is_reconnect'
This leads to an infinite retry loop since every reconnect attempt fails
immediately with the same TypeError.
Fix: add to QQAdapter.connect()'s signature.
QQBot has no server-side update queue, so the flag is accepted only for
interface conformance.
Test: new test_connect_accepts_is_reconnect_param verifies both
adapter.connect() and adapter.connect(is_reconnect=True) succeed without
raising.
The streaming fallback path read yaml_cfg.get("gateway", {}).get("streaming") when top-level streaming was absent or malformed. If a user accidentally set gateway to a scalar value, config loading crashed with AttributeError instead of ignoring the malformed block and using defaults.
Read the gateway block once, verify it is a mapping before accessing nested streaming, and keep the existing gateway.platforms fallback using the same checked value.
Adds a regression test for config.yaml containing gateway: disabled.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
GatewayConfig.from_dict(), PlatformConfig.from_dict(), SessionResetPolicy.from_dict(), and StreamingConfig.from_dict() assumed their input sections were mappings. A malformed scalar from legacy gateway.json or an internal caller could crash config loading with AttributeError before env overrides/defaults had a chance to recover.
Coerce non-mapping sections to empty dicts, skip malformed platform entries, and keep valid sibling platform configs loading normally.
Tests cover scalar platform blocks, scalar nested reset/streaming sections, and malformed PlatformConfig home_channel/extra values.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Flip the two tests that pinned the old buggy behavior (rewrite_transcript
called after in-place compaction) to assert the corrected invariant from
#61145: archive_and_compact() already persisted, so the handler must NOT
call rewrite_transcript — its replace_messages(active_only=False) would
DELETE the just-archived rows.
E2E-verified against a real SessionDB: 6 soft-archived rows are wiped by
replace_messages' default path, confirming the data-loss premise.
Follow-ups on the #60987 salvage (review pass):
- _refresh_fallback_model: keep last known-good chain on transient
config.yaml read/parse failure (user mid-edit, torn write) — only a
successful read that lacks the key clears the chain. Previously a
refresh error wiped a cached agent's working fallback for the turn.
- Move the cached-agent refresh+apply OUTSIDE the agent-cache lock:
config.yaml read is disk I/O and the idle-sweep watcher contends on
that lock (same reasoning as #52197). Per-session turn serialization
keeps the post-lock apply safe.
- _apply_fallback_chain_to_agent: clear _unavailable_fallback_keys when
chain content actually changes, so an entry re-configured mid-uptime
(e.g. credentials added) is retried instead of staying suppressed for
the cached agent's lifetime; no-op refreshes keep the memo.
- Tests: cwd-independent source pin (Path(__file__) anchor), pin the
reuse-path apply call, + regression tests for last-known-good, memo
clear-on-change, memo keep-on-unchanged (mutation-verified).
