Extract the inline pure-tool-call tail check to a named helper using
flatten_message_text (canonical content extraction). Fix a SQLite
durability regression: the incremental tool-call persist
(conversation_loop.py:4990) stamps _DB_PERSISTED_MARKER on the assistant
row, so the next _persist_session flush skips it — the filled content
reaches the in-memory transcript but NOT the durable store, and /resume
reloads content="". Pop the marker so the next flush re-writes the row.
Tests pass, ruff clean.
`finalize_turn` guarantees the invariant "delivered final_response =>
assistant row in transcript" (#43849 / #44100) for recovery paths that
return a response without appending a closing assistant message. It
enforces it by checking only the tail's ROLE:
if _tail_role != "assistant":
messages.append({"role": "assistant", "content": final_response})
A tail that is a *pure tool-call turn* — `assistant(tool_calls=[...])`
with no text of its own — satisfies that check while carrying none of the
delivered answer. The append is skipped and the response is never
persisted, so the durable transcript ends at an assistant row the user
never saw as the reply. On the next turn the model replays the user
backlog and re-answers it: exactly the symptom the block was added to
prevent, just reached through a different tail shape.
Observed with the real finalizer: with messages ending
`user -> assistant(content="", tool_calls=[t1])` and
`final_response="Here is your answer."`, the persisted transcript keeps
`content=""` and the answer is absent.
Fill that row's empty content instead of appending. This keeps the
invariant without disturbing the tool-call structure and without creating
an assistant->assistant pair. A tail tool-call row that already carries
model text is left untouched, so no model output is ever overwritten.
Adds regression tests for both: the empty tool-call tail is filled, and a
tool-call tail with existing text is not clobbered.
The Telegram gateway could go silently deaf for hours: the reconnect ladder
stalled mid-way (e.g. "attempt 4/10, reconnecting in 40s" then nothing) while
the process stayed active(running), so Restart=always never fired.
Root class: every recovery path — the ladder's re-entry
(_schedule_polling_recovery), the pending-update probe (_probe_pending_updates),
and PTB's error callback — gates new recovery on _polling_error_task.done(). If
that single task wedges on any hung await, all recovery returns early forever
and nothing retries.
The heartbeat loop is a separate task, so make it an independent, cause-agnostic
watchdog: if the same recovery task stays in-flight past
_POLLING_ERROR_TASK_STUCK_TIMEOUT (300s — well beyond a healthy ladder attempt's
bounded stop+drain+start+backoff), force a retryable-fatal so the background
reconnector rebuilds the adapter instead of relying on the frozen ladder. This
guarantees progress regardless of *where* the stall is (issue direction #1),
tracked locally so no task-assignment site needs to change.
Also salvages @koduri-mahesh-bhushan-chowdary's #66492 (drain-await timeout),
which closes the one concrete wedge vector documented in the incident
(_drain_polling_connections' unbounded shutdown()/initialize() on a wedged
CLOSE-WAIT pool). The watchdog covers the rest of the class.
Co-authored-by: Koduri Mahesh Bhushan Chowdary <mkoduri73@gmail.com>
_drain_polling_connections() awaited polling_req.shutdown() and
.initialize() without a timeout. When the getUpdates httpx connection is
wedged on a stale CLOSE-WAIT socket, that close can block forever, hanging
_handle_polling_network_error (the tracked _polling_error_task). The task
never completes, so every escalation path — _schedule_polling_recovery,
_probe_pending_updates, the heartbeat verifier — stays gated behind its
in-flight guard, the ladder freezes mid-way, _set_fatal_error is never
reached, and Restart=always never fires: the gateway is alive but silently
dead.
Wrap both drain awaits in asyncio.wait_for with a new module-level
_DRAIN_TIMEOUT (15.0s, matching _UPDATER_STOP_TIMEOUT), mirroring the
existing bounded stop()/start_polling() sites. On timeout the drain logs and
continues, so the handler task completes and the ladder always advances
toward the fatal-restart escalation.
Adds test_reconnect_continues_if_drain_hangs, which wedges the drain and
asserts the handler still reaches start_polling within a hard bound.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hot-reload and multi-entry load_hermes_dotenv can hit the same UTF-32
file repeatedly; gate the refuse-to-mangle warning on a module-level
seen-set (house style: _WARNED_KEYS sibling) so logs are not spammed.
Notepad "Unicode" saves write UTF-16 with a BOM. The sanitizer decoded
those bytes as utf-8-sig with errors=replace, glued U+FFFD onto the first
key name, stripped NULs, and rewrote the mangled content permanently.
Sniff leading BOMs before any text decode (UTF-32 before UTF-16, because
UTF-32-LE's BOM starts with UTF-16-LE's FF FE). Decode UTF-16 correctly
and rewrite as clean UTF-8. Refuse UTF-32 (leave untouched + warning).
After errors=replace, do not persist a first line that starts with U+FFFD.
Does not touch _load_dotenv_with_fallback (#65124's surface).
_quote_env_value previously left internal spaces unquoted (only #/"/'
and leading/trailing whitespace triggered). Spaced macOS paths written
via hermes setup SSH / Google Chat SA path / hermes config set produced
lines that python-dotenv still parsed but shell `set -a; . file` word-split.
Extend needs_quoting with any(c.isspace()); escaping dialect unchanged.
The salvaged unit tests hand-construct completed futures to lock the
_run_on_mcp_loop contract. Add a live-loop test that reproduces the actual
field trigger: an inner asyncio.wait_for expiry stores a real TimeoutError
on a real future scheduled on the MCP loop. Asserts the fixed loop surfaces
it once, promptly -- not spinning to the outer deadline (which both leaked
memory and masked the real error behind the generic wrapper message).
A commit added a bullet and an em-dash inside two Write-Host/Write-Info string
literals in scripts/install.ps1. The file has no UTF-8 BOM, so Windows
PowerShell 5.1 (which the bootstrap runs the cached script under) reads it in
the system ANSI code page (CP1252), not UTF-8. The em-dash's UTF-8 tail byte
decodes to a smart close-quote (U+201D) that the tokenizer treats as a string
delimiter, prematurely closing the string and desyncing the parser -- surfacing
as the reported cascade of syntax errors at lines 1619/1770 and aborting the
Windows GUI installer before it does anything.
Non-ASCII bytes in '#' comments are harmless (skipped to end-of-line), so the
file carried em-dashes in comments for months; only the two chars in code
broke it. Convert all non-ASCII to ASCII equivalents (em-dash -> '--', already
this file's own comment convention; bullet -> '-') and add a source-level test
locking the pure-ASCII invariant, since Linux CI cannot run the PS installer.
Fixes#66994Fixes#67000
Bug 1 of #55048: when the MCP connection dropped (driver crash / restart),
_lifecycle_coro exited but left _started=True, so the next list_apps/capture
passed _require_started() and then operated on a None session — hanging
forever instead of reconnecting.
- _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead
session is re-enterable (idempotent no-op on the normal stop() path; atomic
bool write, safe from the bridge-loop thread without the lock stop() holds).
- call_tool() re-enters start() when the session isn't active, rebuilding it
before the call. The start_session/end_session handshake (driven by start()/
stop() themselves) is exempted so bootstrap doesn't recurse.
Tests: two cases in test_computer_use_delivery_ladder.py — finally resets
_started, and call_tool restarts a dead session exactly once. Full
computer_use suite green (233).
Refs #55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the
delivery_mode work in #67123.
_make_tui_argv() called _ensure_tui_workspace(tui_dir) unconditionally
before checking for a prebuilt bundle. That function sys.exit(1)s when
ui-tui/ doesn't exist, which it never does on a pip/pipx install — the
wheel ships hermes_cli/tui_dist/entry.js but never ships ui-tui/ at all
(that directory only exists in a git checkout).
Every dashboard Chat tab connection on a pip/pipx install therefore
hard-exited before ever reaching _find_bundled_tui(), surfacing as the
unhelpful "Chat unavailable: 1" banner despite having a fully valid
bundled entry.js on disk.
Move the bundled-wheel/HERMES_TUI_DIR shortcut ahead of the workspace
check. --dev is unaffected (it never uses the bundled path and still
requires the workspace), and the checkout-without-bundle path is
unaffected (bundled lookup returns None, falls through to the existing
git-restore/npm-install/build flow).
Adds a contributors/emails mapping for the original author.
Fixes#56665
Co-authored-by: lucaskvasirr <lucaskvasir@duck.com>
Fixes the two review defects that kept PR #29923 open, plus docs:
- gateway: exclude --once from the session-store write-through. The
once-override lived only in memory before, but the write-through
persisted it, so a gateway restart before the finally-restore
rehydrated a supposedly one-turn model permanently.
- TUI: skip _sync_agent_model_with_config while a one-turn restore is
pending. The once-model is deliberately not pinned as a session
model_override, so the config sync saw a model mismatch and clobbered
the once-override back to the config model before the turn ran.
- tests: real _handle_model_command drive asserting --once never
touches set_model_override while --session still does; restore-pop
idempotency.
- docs: /model --once in configuring-models.md with an honest
prompt-cache cost note (one-shot switch breaks the cached prefix
twice; wins for short sessions and cheap-to-expensive escalation).
Adds --once to /model across CLI, TUI, and gateway: switch model for the
next turn only, restoring the previous model in a finally block so
success, exception, and interrupt all revert. Parsing extends
parse_model_flags_detailed(); resolve_persist_behavior() treats --once
as a persistence opt-out; --global + --once is rejected.
Salvaged from PR #29923 (image-generation lane split to #59815 per
review; conflict resolution against current main by the maintainers).
Round-robin configured Discord histories under the global scan cap, but move each channel/thread cursor only when its source message reaches successful final delivery.
Honor configured bot senders at shared ingress, fail closed when durable state is unavailable, and suppress duplicate reconnect work while a fresh queued/processing claim is active.
Admit recovered events without consuming live dedup first, bypass split-message debounce, report actual dispatch admission, and require explicit reply correlation before suppressing a missed request.
Preserve monotonic final-delivery completion, isolate recovery config and storage per adapter/profile, coalesce reconnect scans, release cancelled claims, bypass split-message debounce for historical events, and move bounded ledger setup into a short-timeout state module.
Include allowed mention-gated channels in default recovery scope, keep the newest bounded history window, narrow outage-message detection, avoid disabled-path ledger I/O, and persist forum replies.
Route recovered messages through the live Discord ingress policy, preserve dedup and completion invariants, bound and retain the recovery ledger, and expose the opt-in config with docs and backup coverage.
Hermes' computer_use wrapper dropped cua-driver's structured action verdicts,
exposed no delivery_mode, and injected background-only guidance — so the agent
reported unverified no-ops as success and concluded cua-driver 'cannot drive'
Electron/Chromium surfaces (observed live on tldraw offline). Fixes#67052.
Phase A — preserve the result contract:
- ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode
- CuaDriverBackend._action() reads structuredContent (was data-only); a helper
normalizes it, additive and None-safe on old drivers
- _text_response surfaces the fields additively (ok stays transport-only)
Phase B — bounded, model-reachable foreground:
- delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher,
ABC, and all input methods
- foreground is capability-gated (input.delivery_mode); old drivers get a
structured foreground_unsupported refusal, never a silent background downgrade
- no automatic/hidden foreground retry — the model selects it from the signal
Phase C — guidance + isolation:
- system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from
background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven
by returned effect/escalation, not predicted from the app being Electron
- foreground approval scoped by (action, delivery_mode): a background approval
never silently authorizes foreground
- approval state keyed per session_id so concurrent gateway runs don't leak unlocks
Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/
unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating +
foreground_unsupported, and session-scoped foreground approval. Existing 265
computer_use tests still green.
Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click
returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground
request returned code='foreground_unsupported' — correct on a driver that predates
the input.delivery_mode capability.
Builds on the salvaged typing_status_text plumbing (PR #62007): instead
of a static 'is thinking...', Slack's assistant status line now updates
live as the agent works — 'is running pytest tests/…', 'is reading
docs/api.md…' — and reverts to the static text between tool calls.
Mechanics:
- agent/display.py: build_status_phrase() derives a <=49-char present-
tense phrase from the existing _TOOL_VERBS table (+ 'is using <name>'
for plugin/MCP tools; None for _thinking).
- base adapter: supports_status_text capability flag + set_status_text()
per-chat store, cleared when the typing loop winds down.
- Slack adapter: send_typing() renders the live phrase when set, falling
back to typing_status_text then 'is thinking...'.
- gateway/run.py: progress_callback stashes the phrase on tool.started
and clears on tool.completed. Rendering rides the existing
_keep_typing refresh cadence — zero additional Slack API calls, no
rate-limit exposure. Works with tool_progress: off (Slack default);
the callback is now armed whenever the adapter supports status text.
- display.live_status config (full|verb|off, default full): 'verb' hides
argument previews for shared/customer-facing channels.
Also fixes a latent crash in the cherry-picked from_dict: malformed
non-dict 'extra' sections broke typing_status_text resolution (uses the
already-coerced extra dict).
Design notes: status text is a side-effect display channel only — never
enters the transcript, no prompt-cache impact. Lifecycle guarantees from
the stuck-status fix family are preserved (per-thread tracking,
clear-on-finish via existing stop_typing paths). Related: #45109
(closed; same direction via lifecycle states), #59010/#51363 (native
task cards — complementary, larger scope).
Loader-level coverage for both YAML routes (top-level platform block via
the shared-key bridge; nested platforms.slack via _merge_platform_map),
per review — from_dict alone didn't exercise the bridge.
Adds PlatformConfig.typing_status_text for the two platforms that render
text for the working-state line: Slack's assistant.threads.setStatus
status (hardcoded 'is thinking...') and Google Chat's visible marker
message (hardcoded 'Hermes is thinking…'). None keeps each platform's
built-in default; to_dict omits the field when unset so existing configs
serialize unchanged. Plumbing mirrors typing_indicator exactly (typed
field, from_dict extra fallback, shared-key bridge).
Also documents that Slack's status line requires the assistant:write
scope — without it setStatus fails silently and Slack shows its own
generic placeholder, which previously made the behaviour undiagnosable
from config alone.
Adds two tests for the _background_review_write_guard manual-skill check:
- refuses delete on a skill with created_by=None (manually authored)
- allows delete on a skill with created_by='agent' (agent-created)
Replace REST-based Discord liveness probe with local WebSocket/heartbeat
state detection. REST success doesn't prove Gateway event delivery — a
half-closed WebSocket can leave Bot.start() alive while REST returns 200.
Now samples ready/open/ACK state and heartbeat latency; consecutive
unhealthy samples emit one retryable fatal code so GatewayRunner rebuilds
the adapter through the existing reconnect path.
Also fixes three lifecycle gaps in the recovery path:
1. asyncio.wait_for() can remain blocked if adapter cleanup swallows
cancellation — now uses bounded asyncio.wait() with task detachment.
2. Multiplexed secondary-profile adapters had no profile-scoped reconnect
owner — now uses one runner-owned reconnect slot per profile.
3. An in-flight turn could send its final text through the disconnected
adapter after a replacement was registered — now resolves the live
same-profile replacement for unsent final responses only (message IDs
never migrate, edits/deletes stay on the old transport).
Adds an opt-in Linux/systemd event-loop watchdog (gateway.systemd_watchdog_seconds,
default 0) for the failure mode where the whole asyncio loop stops making
progress and no in-process liveness task can run. stdlib-only sd_notify,
Type=notify/WatchdogSec generation, READY/STOPPING lifecycle.
Co-authored-by: 王鑫 <wx.xw@bytedance.com>
Second call site (non-streaming / gateway path) now flattens list-type
content with flatten_message_text before the inline <think> regex and the
surrogate sanitizer, matching the interim-text fix from the prior commit.
Adds regression tests (tests/run_agent/test_66267_multimodal_interim.py)
covering:
- build_assistant_message with list content does not raise TypeError
- inline <think> inside list content is extracted + stripped correctly
- _interim_assistant_visible_text is safe for tool messages (list content)
- duplicate_previous_interim dedup guards against tool messages
Verified the tests fail without the fix (TypeError: expected string... got
'list') and pass with it.
_bedrock_reasoning_stale_floor only matched dashed floor-table keys (opus
'claude-opus-4-6'), so sonnet reasoning models keyed with a dotted version in the
shared table ('claude-sonnet-4.5'/'4.6') got no reasoning floor from the dashed
Bedrock inference-profile id — premature stale-abort if the base timeout is set
below 180s. Generate both version-separator forms (digit-dash-digit <-> digit-dot-
digit, via lookbehind/lookahead so only version numbers flip) and try all
candidates; matches the table however each model is keyed, no edit to the shared
table. Verified: opus->240 (unchanged), sonnet-4.5/4.6->180, haiku->None. New
TestBedrockReasoningStaleFloor (8 cases).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- _spawn_supervised restart cap was lifetime-cumulative (never reset), so a
watcher crashing _MAX_SUPERVISED_RESTARTS times over a days-long process was
permanently abandoned despite the 'consecutive' wording. Make it time-based:
reset the attempt counter when the task ran healthily (>= _SUPERVISED_HEALTHY_SECS
= 300s) before crashing, so only rapid repeated crashes accumulate toward the
ceiling. Reword docstring/log. New regression: healthy-run-then-crash is not
abandoned.
- _derive_stream_stale_timeout read only 'model', so the reasoning stale-floor
never applied to Bedrock (payloads key the model as 'modelId', dotted
us.anthropic.claude-opus-4-6-v1:0 form). Resolve model||modelId and normalize
the Bedrock dotted/region-prefixed form to match the floor. (Sonnet's dotted
vs dashed floor-table key is a pre-existing table mismatch, left out of scope.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-applied against v0.18.2. Still unconditional KeepAlive <true/> with no
ThrottleInterval, and upstream's new restart_loop_guard only skips session
auto-resume (never throttles the boot), so the launchd respawn storm is
unguarded.
- launchd plist: add ThrottleInterval=30 + ExitTimeOut=25.
- status: record_start_and_check_storm — portable breaker (atomic gateway-starts.log,
backs off when too many starts land in a window); run_gateway sleeps it before
asyncio.run. Env HERMES_GATEWAY_MAX_STARTS (<=0 disables) / _START_WINDOW_S.
Separate file from upstream's restart_loop.json — no collision.
- tests: breaker (threshold/prune/atomic) + plist throttle keys.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-applied against v0.18.2. Upstream now wraps each watcher's INNER loop in
try/except (kept — not re-added), but the 9 long-lived watchers in start() are
still bare asyncio.create_task: no handle, no exception logging, no restart. A
raise in _platform_reconnect_watcher's OUTER while-loop / pre-try region still
dies silently, permanently losing platform reconnection. Add _spawn_supervised
(track + log + bounded-backoff restart; no respawn on clean return to avoid
busy-spin) and route all 9 watchers through it.
- tests: clean-return spawned exactly once; exception-restart bounded at ceiling+1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-applied against v0.18.2. Upstream now covers the OpenAI-path stall watchdog
(cross-turn give-up breaker #58962) and the tool-batch deadline, so those are
dropped. Two gaps remain:
- Bedrock streaming had NO liveness watchdog and was excluded from the #58962
breaker (it returns before _check_stale_giveup). Add an on_event hook to
stream_converse_with_callbacks (fires per yielded event = true wire-level
liveness), drive a stale timer from it, and wire Bedrock INTO the existing
breaker: entry _check_stale_giveup, _bump_stale_streak on stall, raise to end
the call (invalidate_runtime_client can't abort the in-flight botocore stream,
so the streak escalates across turns like the OpenAI path), and reset the
streak on success.
- local providers still got float('inf') stale timeout (watchdog disabled) — give
a finite ceiling (HERMES_LOCAL_STREAM_STALE_TIMEOUT, default 900s).
- tests: on_event per-event + swallow; Bedrock stall bumps streak + aborts;
pre-elevated streak aborts at entry; success resets streak.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
