The #63309 hang class — gateway stuck at 'Connecting to Telegram
(attempt 1/8)' with no retry, no timeout, for minutes — can only occur
when the event loop thread itself is blocked in a synchronous call:
_await_with_thread_deadline's timer fires off-loop, but its expiry
hand-off (call_soon_threadsafe) still needs the loop to run, and the
gateway's outer wait_for is a pure loop timer. When the loop is pinned,
every layer goes silent simultaneously and the process wedges with no
evidence of where.
Two changes:
1. Loop-blocked watchdog in _await_with_thread_deadline: a second
daemon timer fires one grace period (5s) after the deadline; if the
loop still hasn't processed the expiry, it logs a WARNING from the
timer thread and faulthandler-dumps all thread stacks to stderr —
converting the silent hang into a trace that names the exact
blocking frame. A threading.Event set by the expiry callback (and on
normal exit) keeps completed awaits from ever being misreported.
2. discover_fallback_ips: the system-resolver leg runs
socket.getaddrinfo in a worker thread with no timeout, and
asyncio.gather waited on it unboundedly — a wedged OS resolver
stalled discovery for minutes between the two startup log lines. Its
result only feeds a log message, so it no longer gates discovery:
DoH legs (already client-bounded) are gathered alone and the system
leg is awaited with a _DOH_TIMEOUT cap, best-effort.
Refs #63309
Tests: 3 watchdog regressions (blocked-loop dump fires; responsive-loop
timeout does not; completed await does not) + 2 hung-resolver
regressions (DoH results returned promptly; worst-case seed fallback
stays bounded).
httpx ignores the client-level `limits` kwarg when a custom `transport`
is supplied. The #31599 keepalive fix injected limits via
`httpx_kwargs[limits]`, but the fallback-IP branch also passes a
custom `TelegramFallbackTransport` — so the limits were silently
discarded and the inner AsyncHTTPTransport instances ran with httpx
defaults (keepalive_expiry=5.0), leaking CLOSE_WAIT fds.
Pass the tuned limits directly into `TelegramFallbackTransport`
via `transport_kwargs` so its inner transports honour keepalive_expiry.
Only affects the fallback-IP branch; proxy and direct-DNS branches
continue to use `_with_limits()` as before.
Fixes#58790
Salvage of PR #41284 onto current main. Relocates the last 9 inline messaging
adapters (+ satellites: telegram_network, feishu_comment/_rules/meeting_invite,
wecom_crypto, wecom_callback) from gateway/platforms/ into self-contained
bundled plugins under plugins/platforms/<x>/, discovered via the platform
registry. Strips the per-platform core touchpoints from gateway/run.py,
gateway/config.py, hermes_cli/gateway.py, hermes_cli/setup.py, and
tools/send_message_tool.py.
Carries forward the migration fixes (explicit enabled:false honored,
get_connected_platforms forces discovery, plugin is_connected via
gateway.get_env_value, logs --component gateway matches plugins.platforms.*,
matrix hidden on Windows).
Additionally ports config keys main added since the PR base: the matrix
plugin's _apply_yaml_config now also covers allowed_users,
ignore_user_patterns, process_notices, and session_scope (the inline
gateway/config.py matrix block gained these in the 1340 commits the PR sat
open; they would otherwise have been silently dropped on deletion).
discover_fallback_ips() filtered out any DoH-resolved IP that also appeared
in the system resolver's answer set, on the assumption that the system IP
was unreachable. When DoH and system DNS agreed (a common case), the
function returned the hardcoded _SEED_FALLBACK_IPS list instead — and on
networks where those seed addresses are not routable, the Telegram fallback
transport had nothing usable to retry against and polling failed.
Drop the system_ips exclusion so DoH-confirmed IPs are preserved regardless
of system DNS overlap. The TelegramFallbackTransport already tries the
primary path first via system DNS, then falls through to the IP-rewrite
path on connect failure; including the same IP in both lanes lets a
transient primary failure recover via the explicit IP route instead of
escalating to seed addresses.
Update the two tests that codified the old exclusion to reflect the new,
inclusion-by-default behaviour.
Fixes#14520
Group A (3 tests): 'No LLM provider configured' RuntimeError
- test_user_message_surrogates_sanitized, test_counters_initialized_in_init,
test_openai_prompt_tokens_unchanged
- Root cause: AIAgent.__init__ now requires base_url alongside api_key to
skip resolve_provider_client() (which returns None when API keys are
blanked in CI). Added base_url='http://localhost:1234/v1' to test
agent construction.
Group B (5 tests): Discord slash command auto-registration
- test_auto_registers_missing_gateway_commands, test_auto_registered_command_*,
test_register_skill_group_*
- Root cause: xdist workers that loaded a discord mock WITHOUT
app_commands.Command/Group caused _register_slash_commands() to fail
silently. Added comprehensive shared discord mock in
tests/gateway/conftest.py (same pattern as existing telegram mock).
Group C (5 errors): Discord reply mode 'NoneType has no DMChannel'
- All TestReplyToText tests
- Root cause: FakeDMChannel was not a subclass of real discord.DMChannel,
so isinstance() checks in _handle_message failed when running in full
suite (real discord installed). Made FakeDMChannel inherit from
discord.DMChannel when available. Removed fragile monkeypatch approach.
Group D (2 tests): detect_provider_for_model wrong provider
- test_openrouter_slug_match (got 'ai-gateway'), test_bare_name_gets_
openrouter_slug (got 'copilot')
- Root cause: ai-gateway, copilot, and kilocode are multi-vendor
aggregators that list other providers' models (OpenRouter-style slugs).
They were being matched in Step 1 before OpenRouter. Added all three
to _AGGREGATORS set so they're skipped like nous/openrouter.
Group E (1 test): model_flow_custom StopIteration
- test_model_flow_custom_saves_verified_v1_base_url
- Root cause: 'Display name' prompt was added after the test was written.
The input iterator had 5 answers but the flow now asks 6 questions.
Added 6th empty string answer.
Group F (1 test): Telegram proxy env assertion
- test_uses_proxy_env_for_primary_and_fallback_transports
- Root cause: _resolve_proxy_url() now checks TELEGRAM_PROXY first
(via resolve_proxy_url('TELEGRAM_PROXY')). Test didn't clear this
env var, allowing potential leakage from other tests in xdist workers.
Added TELEGRAM_PROXY to the cleanup list.
* fix: keep gateway running through telegram proxy failures
- continue gateway startup in degraded mode when Telegram cannot connect yet
- ensure Telegram fallback transport also honors proxy env vars
- support reconnect retries without taking down the whole gateway
* test(telegram): cover proxy env handling in fallback transport
---------
Co-authored-by: kufufu9 <pi@local>
* feat(telegram): auto-discover fallback IPs via DoH when api.telegram.org is unreachable
On some networks (university, corporate), api.telegram.org resolves to a
valid Telegram IP that is unreachable due to routing/firewall rules. A
different IP in the same Telegram-owned 149.154.160.0/20 block works fine.
This adds automatic fallback IP discovery at connect time:
1. Query Google and Cloudflare DNS-over-HTTPS for api.telegram.org A records
2. Exclude the system-DNS IP (the unreachable one), use the rest as fallbacks
3. If DoH is also blocked, fall back to a seed list (149.154.167.220)
4. TelegramFallbackTransport tries primary first, sticks to whichever works
No configuration needed — works automatically. TELEGRAM_FALLBACK_IPS env var
still available as manual override. Zero impact on healthy networks (primary
path succeeds on first attempt, fallback never exercised).
No new dependencies (uses httpx already in deps + stdlib socket).
* fix: share transport instance and downgrade seed fallback log to info
- Use single TelegramFallbackTransport shared between request and
get_updates_request so sticky IP is shared across polling and API calls
- Keep separate HTTPXRequest instances (different timeout settings)
- Downgrade "using seed fallback IPs" from warning to info to avoid
noisy logs on healthy networks
* fix: add telegram.request mock and discovery fixture to remaining test files
The original PR missed test_dm_topics.py and
test_telegram_network_reconnect.py — both need the telegram.request
mock module. The reconnect test also needs _no_auto_discovery since
_handle_polling_network_error calls connect() which now invokes
discover_fallback_ips().
---------
Co-authored-by: Mohan Qiao <Gavin-Qiao@users.noreply.github.com>
