The read-only premise only holds on Python 3.13+ where the full PTB request
MRO is slotted. CI runs 3.11/3.12 where BaseRequest instances still carry a
__dict__, so the unconditional pytest.raises failed. The behavioral half of
the test (subclass re-tag instruments and records) runs everywhere.
Adds a __slots__ request double that has no instance __dict__, mirroring
PTB's HTTPXRequest shape on Python 3.13. The test asserts the old
instance monkey-patch is rejected as read-only and that
_instrument_polling_request instead re-tags the class and still records
getUpdates progress. Fails on the pre-fix adapter with the exact
"'do_request' is read-only" AttributeError from the report.
When a user starts a chat without ever selecting a model (GUI Chat App
onboarding, provider-set-but-model-missing config, empty model.default),
every silent fallback path resolved to the first curated catalog entry —
anthropic/claude-fable-5, the priciest flagship. Users were silently
billed for the most expensive model without opting in.
- hermes_cli/models.py: add PREFERRED_SILENT_DEFAULT_MODEL (z-ai/glm-5.2)
+ pick_silent_default_model() helper; point the nous silent-default
override at it and add an openrouter override (previously resolved to
"" and let downstream paths land on the flagship).
- hermes_cli/web_server.py: /api/model/recommended-default (the endpoint
the Desktop onboarding confirm card reads) now picks GLM-5.2 when the
provider's list carries it instead of blindly taking entry [0].
- tui_gateway/server.py: _resolve_model()'s last-resort literal was
anthropic/claude-sonnet-4; now PREFERRED_SILENT_DEFAULT_MODEL.
- tests: update empty-model fallback tests for the new contract.
The just-merged auxiliary.<task>.reasoning_effort shorthand applied
ensemble-wide to MoA (one value for every advisor) — wrong granularity.
Per-slot preset config supersedes it:
moa:
presets:
deep_review:
reference_models:
- {provider: ..., model: ..., reasoning_effort: low}
- {provider: ..., model: ..., reasoning_effort: xhigh}
aggregator:
{provider: ..., model: ..., reasoning_effort: high}
- Remove reasoning_effort from the moa_reference/moa_aggregator
DEFAULT_CONFIG blocks; _get_task_extra_body now warns-and-ignores the
key on MoA tasks, pointing at the preset config
- Guard tests: MoA aux blocks must not regrow the key; task-level value
is rejected with the pointer warning
- Docs: configuration.md notes the MoA exception and links the MoA page
Phase 3 of relay-channel-context (gateway/agent side, single PR). The
connector (gateway-gateway #122/#123/#124) now attaches read-only
surrounding channel/group context to an addressed relay turn; this wires
the gateway to consume it.
- descriptor.py: additive optional supports_context (default False) on
CapabilityDescriptor. from_json already filters unknown keys, so this is
back-compat both directions within contract_version 1.
- ws_transport.py: _event_from_wire maps the connector's read-only
context[] array into the EXISTING MessageEvent.channel_context field via
a new _render_relay_context() helper — reusing the same read-only
injection path history-backfill uses (run.py prepends channel_context
ahead of the trigger message). Never raises; absent/empty/malformed ->
channel_context unset (byte-identical to today).
- docs/relay-connector-contract.md: document supports_context in the §2
descriptor table (fixes the contract-doc conformance test) + the
context/context_error inbound fields in §3.
- tests: descriptor default/round-trip/forward-compat; _render_relay_context
rendering + malformed-safe; _event_from_wire context->channel_context
mapping + the read-only invariant (trigger text untouched).
RELAY-ONLY: only gateway/relay/* + the shared MessageEvent consumption via
its existing channel_context field. No native adapter touched.
The inbound cron-fire verifier constructed a fresh PyJWKClient on every
fire, discarding the client's key cache and forcing a synchronous JWKS
HTTP GET to the portal on each fire. Under a burst of concurrent fires
(a hosted instance with several cron jobs firing in the same window) this
fanned out into N simultaneous JWKS fetches that the portal rate-limited
(HTTP 403 -> verification fails -> agent 401), or that blocked the event
loop long enough that the fire webhook could not return its 202 before
the relay's 30s timeout (observed in prod as relay 504s concentrated on
high-job-count instances).
Cache one PyJWKClient per JWKS URL at module scope (double-checked lock)
so the signing keys are reused across fires; NAS keys rotate rarely, so
the steady state is zero JWKS fetches per fire.
Regression test proves 5 fires -> 1 client construction (was 5).
Every auxiliary task block (vision, web_extract, compression,
title_generation, curator, background_review, moa_reference, ...) now
accepts a reasoning_effort shorthand:
auxiliary:
compression:
reasoning_effort: low
vision:
reasoning_effort: none
_get_task_extra_body() folds it into extra_body.reasoning, which every
auxiliary wire already translates: chat.completions passes it through,
the Codex Responses adapter maps it to top-level reasoning/include, and
the Anthropic auxiliary adapter now forwards it into
build_anthropic_kwargs(reasoning_config=...) (previously hardcoded None).
An explicit extra_body.reasoning on the same task wins over the
shorthand. Invalid levels are ignored with a warning. Empty string
(the shipped default) is a no-op — zero behavior change.
Config: reasoning_effort added to all 16 auxiliary task blocks in
DEFAULT_CONFIG (no version bump — deep-merge handles new keys).
Follow-ups on top of #64061's salvage:
- ResourceLink markers now point at mcp__<server>__read_resource (the
actual registered tool name via mcp_prefixed_tool_name) instead of a
nonexistent <server>_read_resource the agent could hallucinate-call.
- The isError path now surfaces EmbeddedResource .resource.text blocks
instead of dropping them, so error payloads carried in resources no
longer collapse to a bare 'MCP tool returned an error'. (Same-class
fix flagged in #64061 and independently addressed in #63576 by
@alauer.)
- 3 new error-path tests + updated ResourceLink wire-name assertion.
MCP tool results with non-image binary resources (PDFs, archives, office
docs) were silently dropped: the success path only handled TextContent and
ImageContent, so a PDF-returning MCP tool appeared to return metadata only.
- EmbeddedResource blob contents are decoded (50MB cap), materialized into
the Hermes document cache via cache_document_from_bytes (sanitized
filename, traversal-safe), and surfaced as a local-path marker the agent
can read with file/terminal tools.
- EmbeddedResource text contents are inlined directly.
- ResourceLink blocks preserve the URI and point the agent at the server's
read_resource tool; no arbitrary network fetch outside the MCP session.
- AudioContent blocks are cached via cache_audio_from_bytes as MEDIA: tags.
- read_resource blob contents are materialized the same way instead of
returning '[binary data, N bytes]'.
- Unsupported blocks are logged instead of silently discarded.
- Existing ImageContent MEDIA: behavior unchanged.
Reported by an enterprise customer; reproduced against an HTTP MCP server
returning application/pdf resources.
The docs state feedback_buttons requires rich_blocks: true, but
_maybe_blocks rendered full Block Kit whenever feedback_buttons alone
was enabled — implicitly turning on rich-block rendering the user never
opted into. Align the code with the documented contract and add a
regression test.
The terminal environment is shared process-globally (collapsed to the
default key), so env.cwd tracks the LAST session that ran a command.
_resolve_command_cwd() trusted env.cwd unconditionally — no ownership
check — so when session A left env.cwd pointing at A's checkout,
session B's first terminal command inherited A's stale cwd and ran in
the wrong workspace.
The file tools already solved this exact shared-env problem with
_live_cwd_if_owned() checking env.cwd_owner. The terminal tool never
got the same guard.
Fix: capture env.cwd_owner BEFORE the current session claims it, and
pass it as prev_owner to _resolve_command_cwd. When the previous owner
was a different session, env.cwd is stale — fall through to default_cwd
(the config/override cwd for this session) instead. Once the session
has claimed the env, subsequent calls in the same session still trust
env.cwd so in-session state survives.
A model response containing several parallel-safe reads plus one unsafe
tool used to lose ALL concurrency: _should_parallelize_tool_batch was
all-or-nothing, so a single barrier call (terminal, clarify, unknown
tool, malformed args) forced the entire batch onto the sequential path.
_plan_tool_batch_segments now splits the batch into ordered segments:
maximal contiguous runs of parallel-safe calls execute on the existing
concurrent path, barrier calls on the sequential path, strictly in the
model's emission order. Invariants preserved:
- one tool result per call, appended in emission order (segments are
contiguous, so no result reordering across a barrier)
- side-effect boundaries: no call starts before an earlier barrier ends
- overlapping file targets split into separate ordered parallel runs
- turn-end budget enforcement + /steer injection run exactly once per
batch (segment executors run with finalize=False; the segmented
dispatcher owns the whole-turn finalize)
- interrupt during segment k drains segments k+1..n with cancelled
results, keeping one result per tool_call_id
Homogeneous batches keep their original single-path dispatch (zero
behavior delta); _should_parallelize_tool_batch remains as a thin view
over the planner for existing callers and tests.
Two test mocks stubbed the old zero-arg signature; the chokepoint refactor
added an optional model param that call sites now pass. Swept the full test
tree for other stale stubs of the changed functions — the rest use
MagicMock/patch(return_value=...), which tolerate the new arg.
Collapse the six per-surface copies of override-then-global resolution
(CLI startup, gateway, TUI, cron, /model switch, fallback activation)
onto one shared resolve_reasoning_config() in hermes_constants.
Also fixes the gateway resolving reasoning against config model.default
instead of the session's effective model: after a session-only /model
switch, the switched model's override now applies (gateway message paths
pass the resolved session model through _resolve_session_reasoning_config;
/reasoning status reads the session model override).
Cleanup: drop docs/PER_MODEL_REASONING.md (duplicates the website docs
page), drop the change-detector _config_version test (no bump needed —
deep-merge handles new keys), remove a stale plan-reference comment.
Adds chokepoint contract tests (13) and gateway session-effective-model
regression tests (2).
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.
Example:
agent:
reasoning_effort: "medium" # global default
reasoning_overrides:
"openrouter/anthropic/claude-opus-4.5": "xhigh"
"openai/gpt-5": "low"
"claude-sonnet-4.6": "high" # bare model name also works
The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.
Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)
Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
+ _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
+ Re-resolves reasoning_config for the fallback model (best-effort)
Closes#21256 (per-model reasoning_effort defaults).
Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
Review findings from the 4-angle pass:
- Unknown-but-enabled effort levels now collapse to Solar's strongest
(high) instead of silently downgrading to the medium default — guards
against the next #62650-style vocabulary addition. Explicit-empty
effort keeps the medium default.
- fallback_models test now asserts the behavior contract (non-empty, no
denied families) instead of freezing the exact model tuple
(change-detector, AGENTS.md reject reason).
- Drop unused pytest import in test_upstage_provider.py.
Main added max/ultra effort levels (#62650) after this PR branched;
without the mapping 'ultra' silently fell through to the medium default.
Matches the xhigh/max collapse-to-strongest convention used by other
profiles.
Pin the Upstage default to the concrete solar-pro3 instead of the
solar-pro rolling alias:
- plugin fallback_models is now ("solar-pro3",); entry [0] is the setup default
- drop the "solar-pro" context-window fallback entry (solar-pro3 covers it)
- update the reasoning default-on docstring and profile tests accordingly
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`UPSTAGE_API_KEY` / `UPSTAGE_BASE_URL` were wired through the provider
resolver, auth registry, and the EnvPage grouping, but never added to
`OPTIONAL_ENV_VARS` in hermes_cli/config.py. The dashboard/desktop
Providers page builds its list from that catalog (`/api/env` iterates
`OPTIONAL_ENV_VARS`), so with no entry the keys were never emitted and
"Upstage Solar" never rendered — the EnvPage prefix group stayed empty.
Add both keys under `category: "provider"` (matching gmi/minimax) so they
show up in `hermes dashboard` / `hermes desktop` under "Upstage Solar".
Adds a regression test asserting the catalog contains them, mirroring the
existing GMI coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Invert the reasoning-support check from an allow-list (solar-pro,
solar-open) to a deny-list of the known non-reasoning families
(solar-mini, syn-pro). Newly released Solar models now get
reasoning_effort by default instead of having it silently dropped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds Upstage Solar as a bundled model-provider plugin. Solar exposes an
OpenAI-compatible chat-completions endpoint at https://api.upstage.ai/v1, so
the generic chat_completions transport handles request/response/streaming/tool
calls — the profile is the core integration.
Provider registration (Upstage isn't in models.dev, so each registry that does
not auto-wire from the plugin layer needs an explicit entry — same pattern as
nvidia/gmi):
- plugins/model-providers/upstage/: UpstageProfile + plugin.yaml. Picker default
and offline catalog list only the agentic Solar Pro models, led by `solar-pro`
(rolling alias for the latest Pro). default_aux_model empty so aux tasks use
the main model. `solar` alias. UPSTAGE_BASE_URL overrides the host.
- hermes_cli/providers.py: HERMES_OVERLAYS + label + `solar` alias, so
resolve_provider_full('upstage') resolves (without this, an explicit
`provider: upstage` in config was dropped and fell through to auto-detect).
- hermes_cli/auth.py: PROVIDER_REGISTRY entry + `solar` alias, so `hermes
doctor` / resolve_provider recognise upstage (the static-registry path the
lazy profile-extension doesn't reliably cover at validation time).
- hermes_cli/models.py: CANONICAL_PROVIDERS entry places Upstage Solar in the
curated picker order (above the auto-appended `custom`).
- agent/model_metadata.py: context-window fallbacks (/v1/models omits
context_length); `solar-pro` carries the 128K Pro context as the catch-all.
Reasoning: UpstageProfile.build_api_kwargs_extras wires Solar's top-level
`reasoning_effort` (low|medium|high; xhigh/max→high). Reasoning-capable families
are solar-pro* and solar-open*; solar-mini/syn-pro never receive it. Defaults ON
at medium when unset (matches the /reasoning "medium (default)" label);
`/reasoning none` disables; explicit/saved settings are honored. No
reasoning_content echo handling needed (unlike DeepSeek/Kimi).
Web dashboard:
- web/src/pages/EnvPage.tsx: add an "Upstage Solar" provider group so
UPSTAGE_API_KEY / UPSTAGE_BASE_URL appear under LLM Providers (not "Other").
Docs/tests:
- .env.example: documents UPSTAGE_API_KEY / UPSTAGE_BASE_URL.
- tests: profile wiring, reasoning_effort mapping (pro/open/mini, efforts,
disabled, default-on), provider-resolver regression (resolve_provider_full /
get_provider / solar alias / overlay), `solar-pro` default.
Testing: pytest tests/providers tests/plugins/model_providers
tests/hermes_cli/test_upstage_provider.py tests/run_agent/test_provider_parity.py
tests/hermes_cli/test_api_key_providers.py; ruff clean. Verified end-to-end:
`hermes doctor` shows "Upstage Solar", and live chat works via both
`--provider upstage` and `--provider solar`. Reasoning wire format per
https://console.upstage.ai/api/docs/for-agents/raw. Platforms tested: macOS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the cherry-picked empty-user-turn drop: the placeholder
introduced in 8582f35d9 fired for whitespace-only STRING turns too
(content=' ' flattens to non-stripping text but isn't in the
(None, '', []) exclusion set), fabricating an attachment note for a turn
that carried nothing. Gate the placeholder on isinstance(content, list)
so only genuinely structured (e.g. image-only) turns get it; empty and
whitespace-only string turns now fall through to the drop path.
Edge cases verified: trailing empty user turn still ends the view on the
synthetic advisory marker; an all-empty transcript degenerates to [].
MoA's _reference_messages() unconditionally appended every user-role
message to the advisory view sent to reference models, even when the
message content was an empty string or a non-string/multimodal payload
that the text-extraction step flattens to "".
Strict providers (Kimi/Moonshot, and others that enforce non-empty user
content) reject such a message with:
400 Invalid request: the message at position N with role 'user'
must not be empty
Lenient providers (DeepSeek) accept it, so an identical rendered view
passes on one reference and 400s on another within the same fan-out —
the user sees "kimi doesn't support MoA" when the real cause is an empty
user turn leaking into the advisory transcript.
Skip empty user turns, mirroring the existing behavior for empty
assistant turns (which are already dropped when they carry no parts).
The end-on-user invariant is preserved: the synthetic advisory-request
user turn is still appended when the view would otherwise end on an
assistant turn.
Adds a regression test asserting the advisory view contains no empty
user turn and still ends on a user turn.
The platform hint in PLATFORM_HINTS['telegram'] always encouraged rich
Markdown constructs (tables, task lists, math, collapsible details) even
when rich_messages: false (the default). This caused the agent to produce
formatting that MarkdownV2 cannot render, especially broken on Telegram Web.
Split the hint into a base hint (MarkdownV2-compatible) and a
TELEGRAM_RICH_MESSAGES_HINT extension. The extension is conditionally
appended in system_prompt.py only when
platforms.telegram.extra.rich_messages is true.
Fixes#57122
Remove the pipe-table bypass from _rich_delivery_enabled() so that
rich_messages: false is fully honoured. Previously, pipe tables were
auto-routed to sendRichMessage regardless of the config flag, breaking
delivery on clients without Bot API 10.1 support (AyuGram, Telegram
Web, some desktop clients).
Fixes#53824
Flaked 3 times today across 3 unrelated PRs (#64321, #64319, #64409),
on two different CI shards (slice 1 and slice 8), while passing
deterministically on local runs of the same SHAs. The test polls
process_registry.completion_queue for 5s waiting for a daemon-thread
completion event; since the durable completion delivery work
(67f4e1b4a, d0e9a42ce) the crashed-runner path also writes through the
sqlite-backed persistence layer, and on slow CI runners the in-memory
enqueue can lose the 5s race.
Coverage note: the durable-delivery suite in this file covers the
completed-runner and submit-failure paths through persistence, but not
a runner that raises mid-flight — that specific path loses its direct
test with this removal. A deterministic (non-racing) replacement can
follow separately if wanted.
CI test slices don't install python-telegram-bot (optional dep), causing
a ModuleNotFoundError on collection. Add pytest.importorskip('telegram')
before the PTB imports.
Review follow-up to the reasoning_config cache-parity fix:
- Only inherit the parent's reasoning_config when the fork runs on the
parent's model (not routed). On the routed aux path
(auxiliary.background_review.{provider,model}) the cache is cold
regardless, so parity buys nothing, and the parent's effort vocabulary
can be invalid for the routed model/provider: OpenRouter
extra_body.reasoning.effort is forwarded unclamped
(chat_completions.py) and codex_responses only maps max/ultra for
gpt-5.6 — an exotic parent effort routed to a strict provider could
400 the review. Mirrors the existing 'not _routed' gate on
_cached_system_prompt / session_start three lines below.
- Add a routed-path regression test asserting reasoning_config is
omitted from the fork kwargs when _resolve_review_runtime returns
routed=True.
- Extract the four copy-pasted recorder stubs in
test_background_review_cache_parity.py into a single
_make_recorder_class() factory so a new fork attribute needs one stub
edit, not four.
PR #17276 painstakingly pinned `_cached_system_prompt`, `session_start`,
`session_id`, and the toolset config on the background-review fork so its
outbound request body would byte-match the parent's and hit Anthropic's
exact-prefix cache. The contributor measured a ~26% end-to-end cost
reduction on Sonnet 4.5.
That optimization is currently being silently undone by a missing
`reasoning_config` kwarg. The fork's `AIAgent(...)` call omits it, so the
fork's `reasoning_config` defaults to `None`. `anthropic_adapter.build_anthropic_kwargs`
(line ~2165) then short-circuits the `thinking` / `output_config` block,
and the fork's request body lands in a DIFFERENT Anthropic cache namespace
from the parent's.
Result on the wire: 0 `cache_read_input_tokens`, full `cache_creation_input_tokens`
of the entire parent prefix — every single background review.
7 days of midagent.db traffic from one host running stock Hermes against
Anthropic Sonnet:
```
Background-review FIRST calls (the moment a review fork is born):
count = 68
cache_write tokens = 7,004,297
cache_read tokens = 1,016,335
Cost on Sonnet ($3.75/M write vs $0.30/M read):
Spent on these writes: $26.27
Cost if they had hit parent cache instead: $2.10
WASTED: $24.16 / week / user
```
That is from one user. Multiply by Hermes's installed base for the full
impact.
Tested against api.anthropic.com directly (see refs/api-tests/ in the
attached investigation repo if needed):
| pair | cache_r | cache_w |
|---------------------------------------------|---------|---------|
| parent fresh | 0 | 24,047 |
| parent same again | 24,047 | 0 |
| fork: appends 2 new tail msgs, thinking ON | 24,047 | 22 |
| fork: appends 2 new tail msgs, thinking OFF | 0 | 24,047 |
Same fork-shape request, only difference is `thinking`. With the fix,
the fork hits the parent's full prefix and only writes the delta
(the `Review the conversation above…` prompt block, ~3-5K tokens).
One line in `agent/background_review.py`: pass
`reasoning_config=getattr(agent, "reasoning_config", None)` to the
`AIAgent(...)` constructor of the review fork. A short comment block
above it explains why so the next person who reads this code doesn't
re-introduce the regression.
`tests/run_agent/test_background_review_cache_parity.py` already covers
the system-prompt / session-id / toolset-config parity contracts that
PR #17276 introduced. I added:
* a `reasoning_config` attribute to `_make_agent_stub` so the stub has
a non-None parent value the test can verify is propagated.
* `test_review_fork_inherits_parent_reasoning_config()` — asserts the
fork's `AIAgent(...)` kwargs carry the parent's `reasoning_config`.
Pre-fix this test fails with `None vs expected {'enabled': True, 'effort': 'medium'}`;
post-fix all 4 tests in the file pass.
```
$ python -m pytest tests/run_agent/test_background_review_cache_parity.py -v
test_review_fork_inherits_parent_cached_system_prompt PASSED
test_review_fork_pins_session_start_and_session_id PASSED
test_review_fork_inherits_parent_toolset_config PASSED
test_review_fork_inherits_parent_reasoning_config PASSED ← new
```
Also runs against the broader background-review test suite:
`test_background_review.py` (4), `test_background_review_summary.py` (8),
`test_background_review_toolset_restriction.py` (3) — 19/19 pass.
`agent/curator.py:1691` has the same omission for the umbrella-curation
fork, but curator's prompt is "curate all skills" — it shares no prefix
with any user conversation, so cache-parity is a non-issue there. Worth
auditing if the curator ever takes a parent conversation as input, but
not part of this PR.
The `agent/auxiliary_client.py:1006` `reasoning_config=None` hardcode is
intentional (title/summary one-shots on short prompts — per-call cost
of namespace flip is negligible) and is also out of scope.
prune_old_entries' active-process guard failed open: when
has_active_processes_fn raised, the except block logged at debug and
fell through to the age check, so sessions with live background
processes attached could still be pruned — violating the documented
invariant that such sessions are never dropped. Add a continue so an
exception in the safety check fails safe (the entry is kept).
Commit 6b408e131 fixed the session_key/session_id mismatch in this
same guard but left the exception path failing open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The #63309 hang class — gateway stuck at 'Connecting to Telegram
(attempt 1/8)' with no retry, no timeout, for minutes — can only occur
when the event loop thread itself is blocked in a synchronous call:
_await_with_thread_deadline's timer fires off-loop, but its expiry
hand-off (call_soon_threadsafe) still needs the loop to run, and the
gateway's outer wait_for is a pure loop timer. When the loop is pinned,
every layer goes silent simultaneously and the process wedges with no
evidence of where.
Two changes:
1. Loop-blocked watchdog in _await_with_thread_deadline: a second
daemon timer fires one grace period (5s) after the deadline; if the
loop still hasn't processed the expiry, it logs a WARNING from the
timer thread and faulthandler-dumps all thread stacks to stderr —
converting the silent hang into a trace that names the exact
blocking frame. A threading.Event set by the expiry callback (and on
normal exit) keeps completed awaits from ever being misreported.
2. discover_fallback_ips: the system-resolver leg runs
socket.getaddrinfo in a worker thread with no timeout, and
asyncio.gather waited on it unboundedly — a wedged OS resolver
stalled discovery for minutes between the two startup log lines. Its
result only feeds a log message, so it no longer gates discovery:
DoH legs (already client-bounded) are gathered alone and the system
leg is awaited with a _DOH_TIMEOUT cap, best-effort.
Refs #63309
Tests: 3 watchdog regressions (blocked-loop dump fires; responsive-loop
timeout does not; completed await does not) + 2 hung-resolver
regressions (DoH results returned promptly; worst-case seed fallback
stays bounded).
Exercises the actual ShellFileOperations.write_file and patch_replace
code paths (not just the helper in isolation) to verify that
safe-root denials surface 'outside HERMES_WRITE_SAFE_ROOT' and
credential-path denials surface 'protected system/credential file'.
Adapted from PR #55615 by @liuhao1024.
Return actionable errors when HERMES_WRITE_SAFE_ROOT blocks a path instead of
labeling every denial as a protected credential file. Wire the helper through
write_file, patch, delete/move, and the Copilot ACP shim; sync docs examples.
Regression tests for find_latest_gateway_session_for_peer and
SessionStore stale-routing self-heal when end_reason is ws_orphan_reap.
Pin manual approval mode in blocking E2E tests so smart aux-LLM
resolution does not flake CI.
