The Telegram gateway could go silently deaf for hours: the reconnect ladder
stalled mid-way (e.g. "attempt 4/10, reconnecting in 40s" then nothing) while
the process stayed active(running), so Restart=always never fired.
Root class: every recovery path — the ladder's re-entry
(_schedule_polling_recovery), the pending-update probe (_probe_pending_updates),
and PTB's error callback — gates new recovery on _polling_error_task.done(). If
that single task wedges on any hung await, all recovery returns early forever
and nothing retries.
The heartbeat loop is a separate task, so make it an independent, cause-agnostic
watchdog: if the same recovery task stays in-flight past
_POLLING_ERROR_TASK_STUCK_TIMEOUT (300s — well beyond a healthy ladder attempt's
bounded stop+drain+start+backoff), force a retryable-fatal so the background
reconnector rebuilds the adapter instead of relying on the frozen ladder. This
guarantees progress regardless of *where* the stall is (issue direction #1),
tracked locally so no task-assignment site needs to change.
Also salvages @koduri-mahesh-bhushan-chowdary's #66492 (drain-await timeout),
which closes the one concrete wedge vector documented in the incident
(_drain_polling_connections' unbounded shutdown()/initialize() on a wedged
CLOSE-WAIT pool). The watchdog covers the rest of the class.
Co-authored-by: Koduri Mahesh Bhushan Chowdary <mkoduri73@gmail.com>
_drain_polling_connections() awaited polling_req.shutdown() and
.initialize() without a timeout. When the getUpdates httpx connection is
wedged on a stale CLOSE-WAIT socket, that close can block forever, hanging
_handle_polling_network_error (the tracked _polling_error_task). The task
never completes, so every escalation path — _schedule_polling_recovery,
_probe_pending_updates, the heartbeat verifier — stays gated behind its
in-flight guard, the ladder freezes mid-way, _set_fatal_error is never
reached, and Restart=always never fires: the gateway is alive but silently
dead.
Wrap both drain awaits in asyncio.wait_for with a new module-level
_DRAIN_TIMEOUT (15.0s, matching _UPDATER_STOP_TIMEOUT), mirroring the
existing bounded stop()/start_polling() sites. On timeout the drain logs and
continues, so the handler task completes and the ladder always advances
toward the fatal-restart escalation.
Adds test_reconnect_continues_if_drain_hangs, which wedges the drain and
asserts the handler still reaches start_polling within a hard bound.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Round-robin configured Discord histories under the global scan cap, but move each channel/thread cursor only when its source message reaches successful final delivery.
Honor configured bot senders at shared ingress, fail closed when durable state is unavailable, and suppress duplicate reconnect work while a fresh queued/processing claim is active.
Admit recovered events without consuming live dedup first, bypass split-message debounce, report actual dispatch admission, and require explicit reply correlation before suppressing a missed request.
Preserve monotonic final-delivery completion, isolate recovery config and storage per adapter/profile, coalesce reconnect scans, release cancelled claims, bypass split-message debounce for historical events, and move bounded ledger setup into a short-timeout state module.
Include allowed mention-gated channels in default recovery scope, keep the newest bounded history window, narrow outage-message detection, avoid disabled-path ledger I/O, and persist forum replies.
Route recovered messages through the live Discord ingress policy, preserve dedup and completion invariants, bound and retain the recovery ledger, and expose the opt-in config with docs and backup coverage.
Gateway tests build adapters via object.__new__() without __init__ (the
documented bare-instance pattern), so the new _status_text dict must be
accessed through getattr guards in set_status_text, the _keep_typing
finally cleanup, and the Slack send_typing read — same treatment as
other post-hoc __init__ attributes. Fixes CI shard 2/8
(test_active_session_text_merge).
Builds on the salvaged typing_status_text plumbing (PR #62007): instead
of a static 'is thinking...', Slack's assistant status line now updates
live as the agent works — 'is running pytest tests/…', 'is reading
docs/api.md…' — and reverts to the static text between tool calls.
Mechanics:
- agent/display.py: build_status_phrase() derives a <=49-char present-
tense phrase from the existing _TOOL_VERBS table (+ 'is using <name>'
for plugin/MCP tools; None for _thinking).
- base adapter: supports_status_text capability flag + set_status_text()
per-chat store, cleared when the typing loop winds down.
- Slack adapter: send_typing() renders the live phrase when set, falling
back to typing_status_text then 'is thinking...'.
- gateway/run.py: progress_callback stashes the phrase on tool.started
and clears on tool.completed. Rendering rides the existing
_keep_typing refresh cadence — zero additional Slack API calls, no
rate-limit exposure. Works with tool_progress: off (Slack default);
the callback is now armed whenever the adapter supports status text.
- display.live_status config (full|verb|off, default full): 'verb' hides
argument previews for shared/customer-facing channels.
Also fixes a latent crash in the cherry-picked from_dict: malformed
non-dict 'extra' sections broke typing_status_text resolution (uses the
already-coerced extra dict).
Design notes: status text is a side-effect display channel only — never
enters the transcript, no prompt-cache impact. Lifecycle guarantees from
the stuck-status fix family are preserved (per-thread tracking,
clear-on-finish via existing stop_typing paths). Related: #45109
(closed; same direction via lifecycle states), #59010/#51363 (native
task cards — complementary, larger scope).
Adds PlatformConfig.typing_status_text for the two platforms that render
text for the working-state line: Slack's assistant.threads.setStatus
status (hardcoded 'is thinking...') and Google Chat's visible marker
message (hardcoded 'Hermes is thinking…'). None keeps each platform's
built-in default; to_dict omits the field when unset so existing configs
serialize unchanged. Plumbing mirrors typing_indicator exactly (typed
field, from_dict extra fallback, shared-key bridge).
Also documents that Slack's status line requires the assistant:write
scope — without it setStatus fails silently and Slack shows its own
generic placeholder, which previously made the behaviour undiagnosable
from config alone.
/simplify-code findings on the #66222 salvage:
- consume_detached_task_result moves to agent/async_utils.py (shared home);
gateway/run.py and the Discord adapter both had near-identical copies of
the same pattern (a third lives in the telegram adapter). One canonical
implementation, both new callsites import it.
- Reconnect backoff formula min(30 * 2^(n-1), 300) was copied verbatim at
3 sites in run.py (primary watcher x2, secondary-profile reconnect), the
third hardcoding the cap. Hoisted to module-level _reconnect_backoff()
with a single _RECONNECT_BACKOFF_CAP so a future tune can't silently
miss one path.
Behavior-preserving: 70 gateway teardown/liveness/reconnect tests green.
Replace REST-based Discord liveness probe with local WebSocket/heartbeat
state detection. REST success doesn't prove Gateway event delivery — a
half-closed WebSocket can leave Bot.start() alive while REST returns 200.
Now samples ready/open/ACK state and heartbeat latency; consecutive
unhealthy samples emit one retryable fatal code so GatewayRunner rebuilds
the adapter through the existing reconnect path.
Also fixes three lifecycle gaps in the recovery path:
1. asyncio.wait_for() can remain blocked if adapter cleanup swallows
cancellation — now uses bounded asyncio.wait() with task detachment.
2. Multiplexed secondary-profile adapters had no profile-scoped reconnect
owner — now uses one runner-owned reconnect slot per profile.
3. An in-flight turn could send its final text through the disconnected
adapter after a replacement was registered — now resolves the live
same-profile replacement for unsent final responses only (message IDs
never migrate, edits/deletes stay on the old transport).
Adds an opt-in Linux/systemd event-loop watchdog (gateway.systemd_watchdog_seconds,
default 0) for the failure mode where the whole asyncio loop stops making
progress and no in-process liveness task can run. stdlib-only sd_notify,
Type=notify/WatchdogSec generation, READY/STOPPING lifecycle.
Co-authored-by: 王鑫 <wx.xw@bytedance.com>
Bare /reasoning and /fast now render a native one-tap picker on
picker-capable platforms, with automatic fallback to the text status
card everywhere else — parity with the /model picker UX.
- gateway/slash_commands.py: generic send_choice_picker capability gate
(detected on the adapter type, like send_model_picker); selection and
typed arguments flow through one shared application path so they can
never diverge; choices built from VALID_REASONING_EFFORTS so future
levels appear automatically
- telegram: flat inline-keyboard picker (cp:<idx> callbacks), authorized
users only (same gate as approval buttons)
- discord: ChoicePickerView select menu, auth mirrors ExecApprovalView,
2-minute timeout with expiry edit
- matrix: reaction-based picker; reaction set extended to 12 slots to
fit the full effort ladder + subcommands
- locales: picker_title + choice labels in all 16 languages
- docs: ADDING_A_PLATFORM.md capability table
Closes#61110.
The Discord /reasoning command declared a single free-text `effort`
parameter, so the native UI funneled every invocation into that one box
and never surfaced the reset / show / hide subcommands the gateway
handler already supports. Replace the free-text param with an explicit
choices dropdown covering the effort levels plus reset/show/hide,
mirroring the existing /tokens and /voice commands. --global persistence
stays reachable by typing the command as plain text.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same bug class as the salvaged #65305/#65307: hmac.compare_digest (and
secrets.compare_digest) raise TypeError when given a str containing
non-ASCII characters, and these call sites feed it raw request input.
Compare as UTF-8 bytes everywhere:
- gateway/platforms/msgraph_webhook.py: clientState from request body
- gateway/platforms/whatsapp_cloud.py: hub.verify_token query param +
X-Hub-Signature-256 header (comment claimed 'works on str' — it
doesn't for non-ASCII)
- plugins/platforms/feishu: verification token + x-lark-signature
- plugins/platforms/raft: bridge token header
- plugins/platforms/line: X-Line-Signature
- plugins/platforms/sms: X-Twilio-Signature
- tools/code_execution_tool.py: sandbox RPC token (both loops)
Regression tests for the two gateway-core sites (msgraph, whatsapp).
Partial cherry-pick of a7ffbbff7 from PR #63256: secondary-profile
adapter creation errors no longer abort the whole secondary startup
(try/except around _create_adapter + loud warning on None return), and
Home Assistant's check_ha_requirements() becomes dep-only with the
credential moved to a new validate_ha_config() so secondary profiles
whose HASS_TOKEN lives in the profile secret scope are not silently
dropped by the registry gate.
Telegram diagnostic hunks and profile-label stamping dropped: the
regression they targeted does not exist on current main and they
conflict with the connect() teardown fence.
Secondary profiles under gateway multiplex keep tokens/allowlists in
profile secret_scope, not process os.environ. Auth and Slack were still
reading os.getenv, so Slack on a secondary profile failed allowlist and
socket mode. Webhook deliver also only looked at default adapters.
- Prefer get_secret for allowlists / allow-all flags (authz_mixin)
- Slack app token + allowlist via secret_scope with getenv fallback
- Wrap secondary profile message handlers in _profile_runtime_scope
before auth runs
- Resolve home-channel env from secret_scope / PlatformConfig
- Webhook deliver falls back to _profile_adapters for target platform
- Template key event_type for webhook prompts
Add a telegram.free_response_topics config list of '<chat_id>:<thread_id>'
entries (plus the TELEGRAM_FREE_RESPONSE_TOPICS env bridge) so a single
forum topic can be free-response — the bot replies without a mention —
without opening the whole chat via free_response_chats. A missing
message_thread_id is normalized to the General topic ('1') via
_effective_message_thread_id.
Re-ported from PR #36049 (by @wesleion): the original patched
gateway/platforms/telegram.py, which has since moved to
plugins/platforms/telegram/adapter.py with a second gating site
(_should_observe_unmentioned_group_message) and plugin-hook config
bridging (_apply_yaml_config). Both gating sites now honor
free_response_topics.
Salvaged-from: #36049
Telegram only auto-derives a voice/audio clip's duration from container
metadata for short recordings; clips longer than ~4:50 are delivered with
duration 0 and render as 0:00 in the player. Probe the length locally
(stdlib wave -> mutagen -> ffprobe) and pass duration explicitly to
sendVoice/sendAudio. Best-effort: when nothing can read the file we omit
duration and fall back to Telegram's prior behavior.
Extracts and hardens the Telegram-only part of the stale, Piper-bundled
PR #7815 (ffprobe-only, predates the send_voice retry/anchor refactor);
relates to #8508.
Extends @AlexFucuson9's 3-site fix (#58594) across the full adapter:
every logger call and SendResult.error that interpolates a raw PTB
exception now routes through _redact_telegram_error_text(). Covers
polling conflict/retry/network ladders, overflow-split edits, draft
sends, prompt/approval/clarify/picker sends, media send fallbacks,
media cache failures, reactions, and chat-info lookups (48 additional
sites). Telegram Bot API exceptions embed the token in the request URL
(/bot<TOKEN>/<method>), so any raw str(exc) is a leak surface.
Adds regression tests for SendResult.error redaction (update prompt,
clarify) and delete_message debug-log redaction.
Telegram Bot API URLs carry credentials in the path as
/bot<TOKEN>/<method>. Three error-handling paths logged raw exception
text that could include these URLs:
- sendRichMessage fallback (line 1603)
- editMessageText fallback (line 1709)
- polling reconnect warning (line 1902)
Replace raw / with which
uses the existing redact_sensitive_text(force=True) pipeline. This
matches the pattern already used by transient send failures, retry
errors, and legacy edit paths.
Fixes#58376
mark_awaiting_text is the 'Other (type answer)' mode-flip, not a send-time
setup call — invoking it in send_clarify forces the user's next message to
be captured as the clarify response, racing the button-click path and
bypassing the buttons entirely. Telegram calls it only in the 'other'
callback branch; do the same here.
Widen @lEWFkRAD's sidecar-headless fix (PR #54565) to the sibling spawn
sites: the npm ci / npm install self-heal runs in _reinstall_sidecar_deps
also popped a brief console window per run on Windows. Same
windows_hide_flags() helper (CREATE_NO_WINDOW only, so capture_output
stays usable).
plugins/platforms/photon/adapter.py launches the Node sidecar (and the
spectrum-ts mixed-attachment patch run) via subprocess without creationflags.
On Windows this opens a visible console window on every sidecar (re)start --
and because a failed sidecar is retried on a timer, it flashes repeatedly.
Wire windows_hide_flags() (hermes_cli/_subprocess_compat) into both spawns,
the same helper the discord and whatsapp adapters already use for their
sidecar spawns -- photon was the one platform adapter this pattern missed.
CREATE_NO_WINDOW only (no DETACHED_PROCESS) so the persistent sidecar's
stdin/stdout pipes stay usable for the supervisor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups on the salvaged #20096 profile-routing feature:
- _profile_name_for_source now returns None unless gateway.multiplex_profiles
is on. Routing stamps source.profile, which namespaces session/batch keys,
but the profile-scoped agent run only activates under multiplexing — without
the gate, configured routes with multiplexing off split batch/session keys
into agent:<profile> while the agent still ran from agent:main.
- Widen the profile-aware _text_batch_key fix from Discord to every adapter
that builds batch keys via build_session_key (telegram, whatsapp, matrix,
feishu, wecom, weixin) — routing is platform-generic, so the batch-key
namespace fix must be too.
- Downgrade the no-route-matched log from INFO to DEBUG (fired on every
unrouted inbound message).
- GatewayConfig.to_dict(): serialize profile_routes as plain dicts
(ProfileRoute dataclasses are not JSON-safe).
- Docs: correct the 'independent of multiplexing' claim in
docs/profile-routing.md (routing requires multiplexing), fix the
platform-only specificity row (0, not 1), and document profile_routes in
website/docs/user-guide/multi-profile-gateways.md.
- Tests: pin the multiplex gate (routes ignored when off, active when on,
build_source end-to-end stays in agent:main when off).
Two follow-ups observed after deploying profile routing:
1. sessions.profile_name was NULL even when the agent ran inside the
routed profile scope. _insert_session_row never wrote it,
get_or_create_session / reset_session never passed it through, and
the agent-side _ensure_db_session fallback had no way to read it.
- Declare profile_name TEXT in SCHEMA_SQL so _reconcile_columns
auto-adds it on existing DBs.
- _insert_session_row takes profile_name and writes it.
- SessionStore passes source.profile (or old_entry.origin.profile
on reset) into db_create_kwargs.
- _ensure_db_session reads the active profile via
get_active_profile_name() inside _profile_runtime_scope.
2. DiscordAdapter._text_batch_key called build_session_key without
profile=, so the batch key always landed in agent:main even when
the routed profile differed — diverging from the agent session
key namespace (agent:crypto-trader, agent:ai-expert, ...).
Pass event.source.profile through so both namespaces agree.
Live verification (jth-server-2, 2026-06-28): a test message in a
routed #coin thread produced agent:crypto-trader:discord🧵...
in the batch log and profile_name=crypto-trader in the sessions
row. Default-routed chat still produced agent:main / NULL.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Salvaged from #32340 by @LeonSGP43, adapted to the workspace-scoped
status tracking that landed in #63709:
- /stop with no running agent now best-effort clears the platform
status indicator, so a phantom 'is thinking...' left by a gateway
restart or a turn that died without a final send can always be
dismissed (#32295).
- SlackAdapter.stop_typing clears an untracked thread when the caller
names it explicitly in metadata — clearing an unset status is a
harmless no-op on Slack's side. The fallback is skipped when multiple
Slack Connect workspaces track the same channel+thread and no team_id
is given, preserving #63709's cross-workspace safety guarantee.
PTB's HTTPXRequest/BaseRequest use __slots__. On Python 3.13 their
instances no longer carry a __dict__, so the getUpdates progress
instrumentation's `request.do_request = wrapper` monkey-patch raises
`AttributeError: 'HTTPXRequest' object attribute 'do_request' is
read-only`, failing every Telegram connect (#64482). It only worked on
Python 3.12, where the instance still had a __dict__ — the Docker image
ships Python 3.13, so the release broke Telegram outright.
Re-tag the request to a thin `__slots__ = ()` subclass that overrides
do_request instead of mutating the instance. This preserves the exact
progress-observation semantics, works on both 3.12 and 3.13, and covers
the real request and the test doubles alike.
The docs state feedback_buttons requires rich_blocks: true, but
_maybe_blocks rendered full Block Kit whenever feedback_buttons alone
was enabled — implicitly turning on rich-block rendering the user never
opted into. Align the code with the documented contract and add a
regression test.
After #53825's fix removed the auto-rich table bypass from
_rich_delivery_enabled(), _content_is_pipe_table_primary() had zero
callers. Remove it and simplify _rich_delivery_enabled() to the bare
rich_messages opt-in check (content param no longer used).
Remove the pipe-table bypass from _rich_delivery_enabled() so that
rich_messages: false is fully honoured. Previously, pipe tables were
auto-routed to sendRichMessage regardless of the config flag, breaking
delivery on clients without Bot API 10.1 support (AyuGram, Telegram
Web, some desktop clients).
Fixes#53824
