A dedicated /context (alias /ctx) gateway slash command that gives a full
context-window view with:
- Usage gauge: visual bar + fraction + percentage + headroom
- Auto-compression threshold and how far away it is
- Compression count and how much the last one freed
- Cumulative session throughput (explicitly labelled as throughput,
NOT context size — each call re-sends the window)
- Cascading fallback: running agent → cached agent → SessionStore metadata
→ rough transcript estimate
Not included (per current-main design):
- Cache reporting removed: commit 446b8e239 intentionally removed cache
reporting from user-facing surfaces because providers that omit cached-token
details produce misleading values
- Sync DB calls replaced with async_session_store (current main requires
AsyncSessionStore with await)
Also rewords the /status tokens line from 'Cumulative API tokens (re-sent
each call)' to 'Lifetime tokens billed: ... (not your current context size;
use /context)' to reduce the recurring confusion that the cumulative figure
is the current context window.
Fixes salvation of PR #52184 (salvage commit replaces a 12K-commit-behind
fork branch with a fresh implementation against current main, incorporating
reviewer feedback from @whoislikemiha and the hermes-sweeper).
Cross-surface coverage #23768 missed (per review feedback):
- tools/clarify_gateway.py: _ClarifyEntry carries a multi_select flag
(register() accepts it; signature() exposes it to adapters).
_coerce_text_response now parses multi-select replies — comma- or
space-separated numbers ('1,3' / '1 3'), exact labels, dedup — into a
JSON array string that _parse_multi_select_response decodes into a
list. Out-of-range/unknown tokens reject the reply (native button UI)
or fall back to custom text (awaiting_text/'Other' mode).
- gateway/run.py: _clarify_callback_sync accepts multi_select and
registers it on the pending entry.
- gateway/platforms/base.py: default numbered-list text fallback tells
the user multiple selections are allowed and how to reply.
- tui_gateway/server.py: clarify_callback passes multi_select through
the clarify.request payload as a hint; renderers without checkbox
support ignore the field and remain single-select-compatible.
- tests: 13 new gateway tests (flag storage, comma/space/single-number
parsing, label matching, out-of-range rejection, dedup, end-to-end
resolve, single-select regressions).
faulthandler.enable() writes to sys.stderr by default, and raises
RuntimeError('sys.stderr is None') when the gateway is launched
without an attached console — e.g. via the Windows Startup VBS shim,
pythonw.exe, a detached service, or any parent that redirects stderr
to DEVNULL. Because this happens on the very first line of
GatewayRunner.start(), the whole gateway used to die at startup and
every configured platform adapter (Discord bot, Telegram, Slack, …)
would silently show offline until the user manually re-ran
'hermes gateway run --replace' from a real terminal.
Wrap the call and fall back to a log-file file descriptor
(logs/gateway_faulthandler.log) when stderr is unavailable, so
fatal-error stack dumps still land somewhere useful. If even the
fallback fails we log-and-continue rather than kill the gateway.
Repro traceback (from a real user's gateway-exit-diag.log, launched
via the Startup VBS with stdin_is_tty=false):
File "gateway/run.py", line 7821, in start
faulthandler.enable()
RuntimeError: sys.stderr is None
Completes #51690 on top of the salvaged #60378 timeout metadata:
- async_delegation: terminal 'stalled' events now carry structured
stall context (stalled_after_quiet_seconds, stall_threshold_seconds,
stall_phase idle|in_tool, stall_grace_seconds) on both single and
batch paths, persisted in the durable row so restart-restored events
keep it. Mirrors the sync path's timeout_seconds/timed_out_after_
seconds/timeout_phase from #60378.
- list_async_delegations(): exposes seconds_since_progress and live
children_activity (per-child api_calls, current_tool,
seconds_since_activity) sampled from the dispatch's progress_fn
outside the records lock; private monitor bookkeeping and callables
never leak.
- /agents (CLI + gateway): background delegations render per-child
activity rows, quiet-time hints, and the stalling state; gateway
section is new (previously async delegations were invisible there).
New locale key gateway.agents.background_delegations in all 17
catalogs.
Tests: stall-metadata event shape, live-listing projection, gateway
/agents rendering (real registry dispatch, sabotage-verified), sync
timeout metadata fields, non-timeout None contract.
Follow-up to the salvaged #71756: instead of webhook importing cron's
private _is_cron_silence_response, the loose autonomous-lane matcher now
lives in gateway/response_filters.py as is_autonomous_silence_response,
sharing LIVE_GATEWAY_SILENT_MARKERS with the interactive exact-marker
rule so the marker sets can never drift. Cron and webhook both delegate
to it. Interactive gateway behavior unchanged.
A webhook route that answered `[SILENT]` still delivered, whenever the model
added a sentence saying why it was staying quiet:
[SILENT]
The new inbound was the same email quoted back a second time, on a ticket
we already answered. Nothing new to reply to, so I closed it.
Webhook subscription prompts tell the agent to answer `[SILENT]` on a tick that
produced no story — a duplicate inbound, a stand-down because a sibling lane
already replied, a routine close. Nobody is waiting on the other end of a
webhook, so a "nothing happened" message has no reader.
Delivery went through the live gateway's `is_intentional_silence_response`,
which requires the response to be EXACTLY a marker. That rule is right for an
interactive chat: swallowing a real answer because it opens with a marker is
much worse than showing a stray marker. It is the wrong trade for an autonomous
lane, where a leaked non-story is a pointless notification on every tick and
models reliably append the explanation that flips the check back to "deliver".
Cron already resolved this the other way — `cron/scheduler.py` treats a marker
on its own first or last line as silence — so the two autonomous lanes
disagreed while the interactive path was fine.
Suppress in `WebhookAdapter.send`, before the deliver-type switch, so every
route (log, github_comment, cross-platform) behaves the same. Reuses cron's
`_is_cron_silence_response` rather than restating the rule, so the two lanes
cannot drift; prose that merely mentions a marker mid-sentence still delivers.
The interactive gateway path is untouched.
Tests: six cases in tests/gateway/test_webhook_adapter.py — bare marker,
marker + trailing prose (the reported shape), marker on the last line, a real
report, a report quoting a marker mid-sentence, and a `log` route. Verified
red-first: with the suppression removed the three silence cases fail
("Expected send to not have been awaited") while the three delivery cases still
pass, so the tests assert the fix rather than the framework.
_collect()'s active_platforms pre-filter was derived solely from
self.adapters (the default profile), so a subscription owned by a
secondary profile on a platform the default profile never connected
(e.g. beta owns discord, default has no discord adapter at all) was
skipped before claim_unseen_events_for_sub ever ran. Unlike the
disconnected-adapter path, an unclaimed event is never rewound, so this
was a permanent, silent notification/wake loss — directly contradicting
the point of routing notifications via the owning profile
(c69643026/b225b30d0). Same cross-profile-adapter-lookup bug class the
delivery-side _authorization_adapter chokepoint already guards against,
one gate earlier. The precise per-profile check still runs unchanged at
delivery time, with its existing rewind-on-None safety net.
Salvaged from PR #63001 (reduced scope): probe each board with the new
read-only kanban_db.count_notify_subs() before the writable connect(),
so boards with zero subscriptions are never opened writable on the 5s
notifier tick (no schema migration, no WAL/-shm sidecar churn, no
checkpoints).
The PR's machine-global .notifier.lock singleton gate was deliberately
NOT salvaged: a lock-winning default-profile gateway cannot deliver a
secondary profile's subscriptions in standalone-profile deployments
(profile routing fails closed in _authorization_adapter), so the lock
could suppress delivery entirely. The probe captures the per-tick cost
win without that regression.
- honor SendResult(success=False) instead of discarding it, so an adapter
that REPORTS (not raises) a soft send failure — e.g. the Telegram adapter's
"Not connected" mid-reconnect — no longer advances the cursor past an
undelivered event and silently loses the notification. Addresses the
notifier half of #31901.
- add block_loop_detected to the notifier's TERMINAL_KINDS so a task routed to
triage for a human decision (re-blocked past the recurrence limit) actually
pings its subscribers instead of stalling silently.
- raise MAX_SEND_FAILURES 3 -> 12 (~60s at the 5s tick) so a transient
Telegram/API outage does not permanently unsubscribe a live channel now that
reported soft-failures also reach this counter.
- route active-profile-stamped subscriptions via the primary adapter on a
single-profile gateway (self.adapters[platform] when the stamped
notifier_profile equals the active profile). Related to #56802.
Adds test_kanban_notifier_rewinds_claim_on_reported_send_failure asserting a
reported send failure leaves the event unseen (rewound) rather than consumed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The kanban notifier _collect() loop iterates subscriptions without
per-subscription error handling. When claim_unseen_events_for_sub raises
for one subscription (e.g. DB corruption, lock contention), the entire
tick aborts — silently blocking delivery for ALL other subscriptions.
Wrap the per-subscription logic in try/except so one bad subscription
logs a warning and continues to the next, instead of jamming the
entire notifier.
Closes#59269
Both the wake chat-scope salvage (#72191, merged) and the DM-topic
metadata salvage added HERMES_SESSION_CHAT_TYPE plumbing; the rebase
auto-merge kept both copies. Dedupe the ContextVar declaration, _VAR_MAP
entry, set_session_vars parameter/token, and the run.py call-site kwarg,
and prefer the persisted chat_type column with delivery_metadata as the
legacy fallback in the notifier wake path.
Follow-up wave to #72170 resolving the remaining open MEDIA-delivery gaps:
- #24032: add .kmz/.kml/.geojson/.gpx to MEDIA_DELIVERY_EXTS, and recover
unknown-extension paths containing spaces via _match_extensionless_path —
validation-gated forward extension across single spaces (bounded at 8
tokens, stops at newline / next MEDIA: keyword). The regex itself stays
non-greedy and whitespace-bounded so the #68773 absorption bug class
cannot return; the on-disk file check is the oracle.
- #16434 (streaming half): _strip_media_tag_directives now uses the same
mask-as-locator pattern as extract_media, so MEDIA tags inside fenced
code blocks, inline-code examples, and JSON string values survive in
streamed display text instead of being mangled. Display and delivery
now agree on every protected-span rule.
- Updated three stream_consumer display expectations that pinned the old
inconsistent behavior (backtick/double-quote tags stripped from display
while delivery never attempted them).
The model sometimes echoes a previous MEDIA:path tag or bare file path in a later response. Previously both streaming delivery and non-streaming delivery would re-send those files, causing duplicate documents/images on later turns.
Changes:
- _collect_history_media_paths now also scans assistant messages for MEDIA: tags, not just tool results.
- _deliver_media_from_response accepts pre-computed history_media_paths and filters extracted media/local files.
- BasePlatformAdapter gains _history_media_paths_for_session for non-streaming dedup.
The default max_iterations/agent.max_turns budget was set when long
agentic runs were rare; complex tasks now routinely exceed 90 tool
calls. Raise the default to 500 across every surface that hardcodes
the fallback: AIAgent constructor, DEFAULT_CONFIG, CLI resolution
chain, gateway env bridge, cron scheduler, and TUI gateway. Explicit
user config values are unaffected (deep-merge preserves them; no
_config_version bump needed).
Docs (en + zh-Hans), CLI help text, tips, and pinned tests updated
to match.
Two remaining formatting variants that silently killed file delivery:
- A trailing sentence period (MEDIA:/x/data.csv.) failed the boundary
lookahead, so the tag neither extracted nor stripped. The period is now
accepted as a boundary only when followed by whitespace/EOL, keeping
multi-part extensions (.tar.gz) intact.
- A whole tag wrapped in inline code (`MEDIA:/x/data.csv`) was masked as
a prose example (#35695). Models routinely format paths as inline code,
eating real deliveries. Inline-code tags now deliver when the path
validates on disk; non-existent example paths stay masked and fenced
code blocks remain fully masked.
Adds a regression matrix covering both plus the salvaged contributor
fixes (emphasis wrap, glued tags, glued [[as_document]], dedupe,
unknown-extension and extensionless delivery).
MEDIA_DELIVERY_EXTS in gateway/platforms/base.py omitted .3gp, causing
MEDIA: tags with .3gp files to leak as plain text instead of being
extracted for native video delivery. _VIDEO_EXTS in
tools/send_message_tool.py and _MIGRATION_VIDEO_EXTS in the Feishu
adapter omitted .webm, causing .webm files to be classified as
documents instead of video on Telegram and other platforms.
Both extensions are already present in every gateway-side _VIDEO_EXTS
definition (run.py, kanban_watchers.py, weixin.py, base.py local).
Closes#71621, Closes#71603
Surface a user-visible delivery notice when non-streaming media dispatch
gets success=False after MEDIA tags were stripped, and validate forum
starter-message attachments the same way as direct channel sends (#66797).
Outbound MEDIA video/document tags on the Discord non-streaming path
were extracted (stripped from the visible text) but never delivered:
no attachment, no error, and no dispatch log line (#66797). The
open-handle discord.File plus singular file= form could race Discord's
multipart encoder after an earlier image batch on the same channel and
return a successful message carrying zero attachments.
Fix _send_file_attachment (used by send_video/send_document/
send_image_file) to:
- use a path-based discord.File via the plural files=[...] kwarg, the
same pattern as the working send_multiple_images batch path;
- pre-flight os.path.isfile and return a File not found result instead
of raising deep inside discord.File;
- fail loud when Discord accepts the message but attaches nothing, so
the dispatch loop surfaces a warning rather than a silent drop;
- add INFO dispatch logs (base non-image MEDIA fan-out, video send, and
file attachment) so a MEDIA:.mp4 cannot vanish without a trace.
Tests (tests/gateway/test_discord_send.py):
- path-based files=[...] kwarg used, singular file= unset;
- fail-loud when the returned message has no attachments;
- missing file fails fast without resolving the channel;
- forum-parent delivery routes through create_thread with files=[...];
- end-to-end image+video response routes the mp4 to send_video while
images still batch.
Fixes#66797
When the same file is referenced multiple times in one message (common
when the agent emits MEDIA tags both inline and in a summary footer),
the platform adapter uploads/sends the same file twice — visible as
duplicate attachments in Telegram, duplicate posts in Slack, etc.
Set-based dedup on the expanded path, preserving first-occurrence order.
Two MEDIA: path tags emitted back-to-back without a separator merged
into a single invalid path and were silently dropped. The same happened
for extension-less tags.
Root cause: both regexes in gateway/platforms/base.py used greedy
quantifiers in their path class, causing adjacent tags to be absorbed.
Fix: make both regexes non-greedy (add ? to quantifiers) and add
MEDIA: to the trailing lookahead boundary set so the next MEDIA:
keyword stops the current match cleanly.
Models routinely present a file to the user with the delivery tag wrapped in
Markdown emphasis — `**MEDIA:/path.pptx**`, `*MEDIA:/path*`, `_MEDIA:/path_`.
MEDIA_TAG_CLEANUP_RE only tolerated a single leading/trailing quote or backtick
(`[`"']?`), and its closing lookahead set excluded `*` and `_`, so an
emphasis-wrapped tag never matched. The file was then silently never delivered
and the literal `MEDIA:/path` text leaked into the chat instead — the user sees
a path, not the attachment.
Allow a short run of emphasis/quote markers (`[`"'*_]{0,3}`) on both sides of
the tag and add `*`/`_` to the closing lookahead. Code-block, inline-code and
blockquote contexts are still neutralised earlier by `_mask_protected_spans`
(#35695), so documentation/example tags remain non-deliverable; the
absolute-path anchor still rejects relative paths; `_` inside a filename is
unaffected.
Adds regression coverage in TestExtractMedia for bold/italic/underscore
wrapping, mid-prose bold, emphasis-wrapped .html, underscore-in-filename, and
emphasis-wrapped relative-path rejection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MEDIA_TAG_CLEANUP_RE failed to match when a directive like [[as_document]]
was concatenated directly to the file extension without whitespace
(e.g., MEDIA:/home/user/report.xlsx[[as_document]]). This caused the
file to be silently not delivered while the gateway reported success.
Root cause: the lookahead character class [\s`",;:)\}\]|$) did not
include [, so [[as_document]] immediately after the extension broke
the lookahead assertion.
Fix: add \[ to the lookahead class so directives can follow the path
without whitespace. This is safe because [ is already stripped elsewhere
in the same file via .replace("[[as_document]]", "").
Added regression tests covering:
- Directive glued to extension (issue case)
- Directive with whitespace (baseline)
- Tag at end of string ($ anchor)
The sidebar strip and the Channels page could contradict each other on
the same page load — "Gateway running" next to "The gateway is not
running." /api/status and /api/messaging/platforms each open-coded their
own liveness ladder: status probed GATEWAY_HEALTH_URL and scoped its
PID/state reads to the requested profile, messaging did neither and used
the uncached raw PID probe.
Three deployments hit the split: a cross-container gateway (no local PID,
only the health probe can see it), a profile-scoped dashboard (messaging
borrowed a DIFFERENT profile's runtime state, reporting a false
"connected" that hides a real outage — #71211), and a launch-service
managed gateway with no PID file.
Adds resolve_gateway_liveness() in gateway/status.py as the single ladder
(cached PID -> HTTP health probe -> runtime-status PID with
expected_home) and routes both endpoints, /api/messaging/platforms/{id}/test,
and the kanban dispatcher-presence probe through it. Probe callables are
injectable so the existing monkeypatch seams keep working, and
GatewayLiveness.probe_error distinguishes "down" from "couldn't tell" so
the kanban warning keeps failing OPEN instead of crying wolf.
Closes#71211.
- RelayAdapter.create_handoff_thread → one op-gated thread_create op
(Discord channel thread / Telegram forum topic / Slack named seed root);
None fallback contract preserved for the handoff watcher.
- RelayAdapter.rename_thread → thread_rename with only_if_current_name
no-clobber guard on the wire (connector enforces; Telegram guarded
renames fail safe). The native semantic-rename lane
(_is_discord_auto_thread_lane) lights over the relay via the
connector-stamped auto_thread_created/auto_thread_initial_name markers
parsed onto SessionSource in _event_from_wire.
- reply_to {text, author, is_own} wire parse onto the SAME MessageEvent
reply-context fields native adapters populate.
- gateway/relay/command_manifest.py: the gateway-declared slash-command
manifest (native Discord tree mirror) sent on the DISCORD hello; the
connector reconciles Discord's global registration (additive field,
older connectors ignore).
- Contract doc §OutboundAction ops + Phase 4 semantics sections.
- 12 new tests (tests/gateway/relay/test_relay_threads.py); relay suite
266 passed.
- RelayAdapter.send_exec_approval / send_slash_confirm / send_clarify:
override the base text fallbacks with ONE platform-abstract `prompt` op
(connector renders Discord components / Telegram inline keyboards /
Slack Block Kit / WhatsApp buttons+lists). Option sets mirror the native
adapters exactly (once/session/always/deny with the same
allow_session/allow_permanent/smart_denied gating; once/always/cancel;
choices + Other). Clarify option ids are positional (c0..cN/other) —
choice text is arbitrary UTF-8, callback budgets are 64 bytes.
- Pending-prompt registry: gateway-minted 8-hex prompt ids →
{kind, session_key, extras}; one answer wins, lazy expiry, unanswered
prompts swept opportunistically. Wire timeout_s stays advisory.
- _consume_prompt_response (wired into _on_inbound AND the Discord
passthrough lane): routes answers to the SAME primitives the native
button handlers call — tools.approval.resolve_gateway_approval,
tools.slash_confirm.resolve, tools.clarify_gateway
resolve/mark_awaiting_text — then acks in-channel. Unknown/expired ids
fall through as command-shaped text (typed-reply degradation, the
relay's analog of the native 'approval expired' edit).
- Discord type-3 stub replaced: an hp1:<prompt>:<option> custom_id decodes
to a structured prompt_response (codec mirrored from the connector's
promptCodec); foreign custom_ids keep the legacy best-effort text shape.
- MessageEvent.prompt_response field + ws_transport wire parsing (additive).
- react ack lifecycle: on_processing_start/complete → `react` ops
(👀 → ✅/❌, remove-then-add), op-gated on supported_ops, best-effort by
contract (a react failure never touches the turn).
- Op gating throughout: a connector not advertising `prompt` gets
success=False from send_exec_approval/send_slash_confirm (run.py's text
fallback takes over — same contract as a failed native button send) and
the base numbered-text clarify; `react` silently no-ops.
- docs/relay-connector-contract.md §4: prompt / prompt_response / react
semantics (callback token, budgets, authorization-parity, foreign-id
behavior, per-platform react mappings).
- tests: tests/gateway/relay/test_relay_interactive.py (19) — option-set
rendering + gating matrices, registry consume-once/expiry, resolver
routing for all three kinds (monkeypatched primitives), fall-through
cases, Discord hp1 decode + foreign-id shape, react lifecycle
(success/failure/cancelled), op-gated/best-effort react.
Cross-repo pair: gateway-gateway 'Phase 3 interactive' PR (prompt/react
senders on all four lanes + interaction ingest).
The gateway's pre-agent session-hygiene compression killed the summary
call at a fixed 30s wall-clock deadline (compression.hygiene_timeout_seconds),
regardless of whether the summary model was hung or merely slow. A reasoning
model happily streaming a large summary was cut off mid-generation, the user
got '⚠️ Context compression timed out after 30.0s', and a 300s failure
cooldown left the session oversized — a doom loop for slow-but-healthy
auxiliary models.
Timeouts are now liveness-based instead of wall-clock-based:
- agent/auxiliary_client.py: new thread-local aux_progress_hook. When
installed (only by context compression today), the primary call_llm
attempt streams (stream=True) and aggregates chunks back into a complete
response, ticking the hook per chunk. The configured timeout then acts
per stream read (idle) instead of as a total budget. Providers that
reject streaming fall back to the plain non-streaming call; auth/payment/
rate-limit/transport errors propagate unchanged into the existing
recovery chains. Codex Responses (per SSE event) and Anthropic Messages
(per stream event, via the new create_anthropic_message on_stream_event
callback) tick the same hook from inside their wire adapters.
- agent/conversation_compression.py: CompressionCommitFence gains
touch_progress()/seconds_since_progress(); compress_context() installs
fence.touch_progress as the progress hook around the compress call.
- gateway/run.py: the hygiene wait loop treats hygiene_timeout_seconds as
an INACTIVITY budget — while the fence reports fresh progress the wait
extends, bounded by the new compression.hygiene_total_ceiling_seconds
(default 600s, clamped >= the idle budget) so a degenerate trickle
stream still dies. The timeout warning now says the summary model
produced no output, which is the only case that still triggers it.
- config/docs: hygiene_total_ceiling_seconds added to DEFAULT_CONFIG and
configuration.md; hygiene_timeout_seconds documented as inactivity-based.
Tests: tests/agent/test_aux_progress_streaming.py (hook plumbing, stream
aggregation incl. tool-call deltas and reasoning deltas, rejection
fallback, ceiling kill, fence progress surface); two new gateway tests
prove a slow-but-streaming worker survives past the fixed timeout
(sabotage-verified: fails with the old fixed deadline) and a
forever-trickling worker is still cut off at the ceiling.
- gateway/relay/media.py: RelayMediaClient for the connector's /relay/media
plane (upload local files → re-host reference for send_media; download
re-hosted inbound attachments → local temp paths). Same connector base URL
the WS dials, same per-gateway signed bearer as the upgrade (auth.py) —
no new configuration. stdlib urllib in a thread executor (no new deps);
25MB cap mirroring the connector's MEDIA_MAX_BYTES.
- RelayAdapter: send_image / send_image_file / send_voice / send_video /
send_document overrides route through ONE send_media op (media by
reference: local paths upload first, public URLs pass through). Gated on
supported_ops advertising send_media — legacy connectors keep today's
text fallbacks; connector declines/failed uploads degrade the same way.
Scope/user egress discriminators ride metadata exactly like send.
- Inbound: _localize_inbound_media downloads each media_urls entry to a
local temp path (native-adapter parity — vision/file tools consume
paths); dead re-host refs are dropped, public URLs survive a missing
client. Best-effort, never blocks handle_message.
- docs/relay-connector-contract.md §4: send_media op row + media
ingress/egress semantics (replaces the 'deferred to a later revision'
note). Additive within contract_version 1.
- tests: tests/gateway/relay/test_relay_media.py (15) — kind mapping,
upload-first path handling, op gating (explicit + legacy-empty),
decline/upload-failure fallbacks, scope metadata, inbound localization
matrix, client URL derivation/credential gating. Stub connector grew a
canned send_media result.
Cross-repo pair: gateway-gateway 'Phase 2 media parity' PR (re-host plane +
four ingress lanes + four platform send_media senders).
A real APPLICATION_COMMAND interaction forwarded over the relay arrived
slash-less: _discord_interaction_to_event set text = data['name'] ("new",
not "/new"), MessageType.TEXT, and dropped options entirely — so a
registered /new dispatched as plain chat instead of a command
(MessageEvent.is_command() is text.startswith("/")).
Port the connector's Slack slash-command precedent (normalizeSlackCommand
builds `${command} ${args}`.trim() with a leading slash and explicit
command type): for type-2 interactions build "/" + name, append rendered
options space-separated (scalar options contribute their value, matching
the native adapter's f"/model {name}" shape; SUB_COMMAND/
SUB_COMMAND_GROUP contribute their name then recurse into nested
options), and set MessageType.COMMAND. Type-3 (custom_id) and other
interaction types are unchanged. This implements the interaction->command
sub-design previously flagged as deferred in the _on_passthrough
docstring.
Companion connector fix in gateway-gateway: fix(relay): strip own-mention
prefix so addressed slash commands dispatch.
Give Telegram a 180s default outer connect budget so cold polling can prove getUpdates readiness. Preserve the 30s default for other platforms and all explicit config/env overrides.\n\nRefs #67498
Teardown-path tests build bare runners via object.__new__ without
the liveness-guard machinery; the unguarded call raised
AttributeError in 8 tests. Same guard pattern as the start path.
Follow-up to the salvaged #69164 commits: policy forbids introducing new
HERMES_* environment variables, so the four watchdog env knobs
(HERMES_GATEWAY_LOOP_WATCHDOG / _INTERVAL / _TIMEOUT / _STRIKES) are
replaced with a single config.yaml boolean:
gateway:
loop_watchdog: true # default; false disables both guards
- gateway/config.py: new GatewayConfig.loop_watchdog field (default True),
parsed from top-level or nested gateway: form, round-trips via
to_dict/from_dict.
- gateway/run.py: _start_loop_liveness_guards() checks config.loop_watchdog
before arming the floor timer + watchdog (getattr-guarded for bare
object.__new__ runners).
- gateway/shutdown_watchdog.py: start_loop_liveness_watchdog() no longer
reads the environment; probe interval/timeout/strikes are module
constants (30s/10s/3 — ~90s to restart, matching the systemd watchdog
layer's posture).
- hermes_cli/config.py: documented gateway.loop_watchdog default so
'hermes config set gateway.loop_watchdog false' validates.
- tests: env-knob tests replaced with config-gate + round-trip tests;
the final-strike boundary test injects its probe via max_strikes
directly instead of patching the removed env helper.
- A stop() landing while the final diagnostics (critical log,
traceback dump) are executing could still reach os._exit(75) after
the pre-diagnostic check. Add a third stop_event recheck immediately
before the hard exit: diagnostics may complete, but a disarmed
watchdog never exits.
- Deterministic regressions for both windows (stop triggered from
inside logger.critical and from inside faulthandler.dump_traceback);
mutation-verified (removing the check turns both red). Frozen-loop
semantics unchanged.
Addresses the second round of the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Re-check stop_event after a missed probe (before the strike
increment) and again on entering the final-strike branch (before the
critical log, dump, and hard exit), so a normal stop() landing
between the last timeout check and the exit path can no longer be
misclassified as a freeze and trigger a supervisor restart.
- Deterministic boundary tests pin both re-checks independently
(mutation-verified: removing either check turns its own test red);
frozen-loop semantics are unchanged.
Addresses the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A self-rescheduling 5s call_later floor timer, armed before any
adapter connects, guarantees the selector always has a finite
timeout, so the existing async defenses (polling heartbeat, timeout
guards) regain a chance to run after a zero-pending-timer stall.
- A resident daemon-thread liveness watchdog probes the loop via
call_soon_threadsafe every 30s; after 3 consecutive 10s-timeout
misses (~120s of total unresponsiveness) it dumps all thread
tracebacks and exits with the established
GATEWAY_SERVICE_RESTART_EXIT_CODE (75) so a supervisor restarts the
gateway - async-level recovery cannot run on a frozen loop.
- stop() disarms both guards before any teardown await so a busy
shutdown is never misjudged as a freeze.
HERMES_GATEWAY_LOOP_WATCHDOG=0 disables; _INTERVAL/_TIMEOUT/_STRIKES
tune the thresholds.
Fixes#69089
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
signal.SIGUSR2 and faulthandler.register() don't exist on Windows;
the bare reference raised AttributeError at import time per the
windows-footgun checker. faulthandler.enable() still covers
fatal-error dumps on all platforms.
When connected_count == 0 and at least one platform failed with a
non-retryable error, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE
(78) even if OTHER platforms failed for merely transient reasons.
Real-world shape (NS-609, hosted instance): WhatsApp enabled but never
paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during
polling startup (retryable) => exit 78 => the gateway either goes
permanently down (supervisors honoring the exit-78 contract via
RestartPreventExitStatus / the s6 finish->125 translation from #51228) or
crash-loops (anything else). Either way Telegram never gets its retry and
the dashboard drops with every exit, so a single unpaired platform plus
one network blip disconnected every channel on the instance.
Now exit 78 is reserved for the case where ALL startup failures are
non-retryable (true config error, nothing to wait for). With mixed
failures the gateway stays alive in degraded state: the reconnect watcher
recovers the retryable platforms and the misconfigured ones stay
fatal-parked and visible in runtime status.
Three-part fix for the gateway going silently deaf after a retryable
fatal adapter error (e.g. httpx.ConnectError on Telegram):
1. **Detach-on-timeout in _connect_adapter_with_timeout** — Replaced
plain asyncio.wait_for with the task-detach pattern used by
_await_adapter_cleanup_with_timeout. asyncio.wait_for cancels the
overdue task but then waits for it to exit, so a connect() that
catches CancelledError can block recovery forever. The detach
pattern releases the runner at the deadline via
consume_detached_task_result.
2. **Ensure reconnect watcher always runs after escalation** — Added
_ensure_reconnect_watcher_running(), called after queueing a
retryable fatal error. If the reconnect watcher task has died
(exhausted restart budget, terminal exception), it is respawned
so queued platforms are never permanently stranded.
3. **Faulthandler at gateway startup** — Enabled faulthandler +
SIGUSR2 dump to a rotating file under HERMES_HOME/logs/ for
post-mortem diagnosis of future event-loop freezes.
Tests added for _ensure_reconnect_watcher_running (alive, dead,
not-started, not-running), fatal-error integration (retryable calls
ensure, non-retryable does not), and _connect_adapter_with_timeout
(timeout raises, success returns).
Builds on this PR's diagnosis by @Frowtek: a missing workdir is
ambiguous (deleted project vs. an unmounted external volume / network
share / VPN not yet up), so it's not safe evidence for a destructive
GC sweep — especially one that runs unattended at startup.
- cli.py / gateway/run.py: the startup auto-maintenance sweep now
always passes delete_orphans=False to maybe_auto_prune_checkpoints().
It still prunes by retention_days, size cap, and legacy archives —
none of which require guessing whether a project was deleted or is
just temporarily unreachable.
- hermes_cli/config.py: drop the now-unused delete_orphans default.
- hermes_cli/checkpoints.py: `hermes checkpoints prune` (the explicit,
human-invoked path) now previews the orphan project list and asks
for confirmation before deleting, unless -f/--force is passed.
- Docs updated (EN + zh-Hans) to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sibling of the #69678/#69567 ledger leak class found while widening the
sweep: _probe_state_db used 'with sqlite3.connect(...)', whose context
manager only commits/rolls back and never closes, leaking one connection
(db fd) per health poll in the long-running gateway. Wrap the connection
in contextlib.closing so every probe closes deterministically.
Three durable ledgers used `with _connect() as conn:` where the sqlite3
connection context manager commits/rolls back but never closes, leaking the
db/-wal/-shm file descriptors on every call. On a long-running gateway this
exhausts RLIMIT_NOFILE and fails unrelated components with
`[Errno 24] Too many open files`. Same bug class as the cron execution ledger
(#69567 / PR #69594), which the connection helpers here are modeled on.
Fix: route every ledger operation through a `_transaction()` context manager
that guarantees `conn.close()` on exit. `_connect()` keeps its
schema-on-connect contract (several tests call it directly) and now self-closes
if schema init fails.
Adds per-module regression tests asserting every opened connection is closed,
including the no-op-update and exception-mid-transaction paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a persisted, backend-confirmed provider/model lock for Hermes
Browser and other session API clients. A confirmed lock is an
execution contract rather than response metadata:
- POST /api/sessions/{session_id}/model validates and persists a
confirmed browser_model_lock (advertised in /v1/capabilities)
- session chat + chat/stream consume the persisted lock on body-only
follow-up turns; a confirmed lock wins over an older gateway session
/model override and the session-persisted model
- a later successful session /model switch explicitly clears and
replaces the lock while preserving lineage markers (_branched_from)
and invalidating cached system-prompt model/provider metadata
- ordinary one-off request overrides never replace a confirmed lock
- provider-resolution failure fails closed as a typed provider-auth
error (controlled response, never global-credential reuse)
- confirmed locks disable the global fallback model chain
- the completed agent's actual provider/model must match the locked
route or the turn fails with a runtime-mismatch error
- responses carry sanitized runtime metadata reporting actual vs
requested provider/model and lock state
Rebased onto the provider-aware request routing (#70853) and
session-model parity (#70931) that landed since the original branch;
the lock now slots into that precedence chain as the top rung.
Salvaged from PR #61236 by @abundantbeing.
