The Telegram gateway could go silently deaf for hours: the reconnect ladder
stalled mid-way (e.g. "attempt 4/10, reconnecting in 40s" then nothing) while
the process stayed active(running), so Restart=always never fired.
Root class: every recovery path — the ladder's re-entry
(_schedule_polling_recovery), the pending-update probe (_probe_pending_updates),
and PTB's error callback — gates new recovery on _polling_error_task.done(). If
that single task wedges on any hung await, all recovery returns early forever
and nothing retries.
The heartbeat loop is a separate task, so make it an independent, cause-agnostic
watchdog: if the same recovery task stays in-flight past
_POLLING_ERROR_TASK_STUCK_TIMEOUT (300s — well beyond a healthy ladder attempt's
bounded stop+drain+start+backoff), force a retryable-fatal so the background
reconnector rebuilds the adapter instead of relying on the frozen ladder. This
guarantees progress regardless of *where* the stall is (issue direction #1),
tracked locally so no task-assignment site needs to change.
Also salvages @koduri-mahesh-bhushan-chowdary's #66492 (drain-await timeout),
which closes the one concrete wedge vector documented in the incident
(_drain_polling_connections' unbounded shutdown()/initialize() on a wedged
CLOSE-WAIT pool). The watchdog covers the rest of the class.
Co-authored-by: Koduri Mahesh Bhushan Chowdary <mkoduri73@gmail.com>
_drain_polling_connections() awaited polling_req.shutdown() and
.initialize() without a timeout. When the getUpdates httpx connection is
wedged on a stale CLOSE-WAIT socket, that close can block forever, hanging
_handle_polling_network_error (the tracked _polling_error_task). The task
never completes, so every escalation path — _schedule_polling_recovery,
_probe_pending_updates, the heartbeat verifier — stays gated behind its
in-flight guard, the ladder freezes mid-way, _set_fatal_error is never
reached, and Restart=always never fires: the gateway is alive but silently
dead.
Wrap both drain awaits in asyncio.wait_for with a new module-level
_DRAIN_TIMEOUT (15.0s, matching _UPDATER_STOP_TIMEOUT), mirroring the
existing bounded stop()/start_polling() sites. On timeout the drain logs and
continues, so the handler task completes and the ladder always advances
toward the fatal-restart escalation.
Adds test_reconnect_continues_if_drain_hangs, which wedges the drain and
asserts the handler still reaches start_polling within a hard bound.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Round-robin configured Discord histories under the global scan cap, but move each channel/thread cursor only when its source message reaches successful final delivery.
Honor configured bot senders at shared ingress, fail closed when durable state is unavailable, and suppress duplicate reconnect work while a fresh queued/processing claim is active.
Admit recovered events without consuming live dedup first, bypass split-message debounce, report actual dispatch admission, and require explicit reply correlation before suppressing a missed request.
Preserve monotonic final-delivery completion, isolate recovery config and storage per adapter/profile, coalesce reconnect scans, release cancelled claims, bypass split-message debounce for historical events, and move bounded ledger setup into a short-timeout state module.
Include allowed mention-gated channels in default recovery scope, keep the newest bounded history window, narrow outage-message detection, avoid disabled-path ledger I/O, and persist forum replies.
Route recovered messages through the live Discord ingress policy, preserve dedup and completion invariants, bound and retain the recovery ledger, and expose the opt-in config with docs and backup coverage.
Gateway tests build adapters via object.__new__() without __init__ (the
documented bare-instance pattern), so the new _status_text dict must be
accessed through getattr guards in set_status_text, the _keep_typing
finally cleanup, and the Slack send_typing read — same treatment as
other post-hoc __init__ attributes. Fixes CI shard 2/8
(test_active_session_text_merge).
Builds on the salvaged typing_status_text plumbing (PR #62007): instead
of a static 'is thinking...', Slack's assistant status line now updates
live as the agent works — 'is running pytest tests/…', 'is reading
docs/api.md…' — and reverts to the static text between tool calls.
Mechanics:
- agent/display.py: build_status_phrase() derives a <=49-char present-
tense phrase from the existing _TOOL_VERBS table (+ 'is using <name>'
for plugin/MCP tools; None for _thinking).
- base adapter: supports_status_text capability flag + set_status_text()
per-chat store, cleared when the typing loop winds down.
- Slack adapter: send_typing() renders the live phrase when set, falling
back to typing_status_text then 'is thinking...'.
- gateway/run.py: progress_callback stashes the phrase on tool.started
and clears on tool.completed. Rendering rides the existing
_keep_typing refresh cadence — zero additional Slack API calls, no
rate-limit exposure. Works with tool_progress: off (Slack default);
the callback is now armed whenever the adapter supports status text.
- display.live_status config (full|verb|off, default full): 'verb' hides
argument previews for shared/customer-facing channels.
Also fixes a latent crash in the cherry-picked from_dict: malformed
non-dict 'extra' sections broke typing_status_text resolution (uses the
already-coerced extra dict).
Design notes: status text is a side-effect display channel only — never
enters the transcript, no prompt-cache impact. Lifecycle guarantees from
the stuck-status fix family are preserved (per-thread tracking,
clear-on-finish via existing stop_typing paths). Related: #45109
(closed; same direction via lifecycle states), #59010/#51363 (native
task cards — complementary, larger scope).
Adds PlatformConfig.typing_status_text for the two platforms that render
text for the working-state line: Slack's assistant.threads.setStatus
status (hardcoded 'is thinking...') and Google Chat's visible marker
message (hardcoded 'Hermes is thinking…'). None keeps each platform's
built-in default; to_dict omits the field when unset so existing configs
serialize unchanged. Plumbing mirrors typing_indicator exactly (typed
field, from_dict extra fallback, shared-key bridge).
Also documents that Slack's status line requires the assistant:write
scope — without it setStatus fails silently and Slack shows its own
generic placeholder, which previously made the behaviour undiagnosable
from config alone.
/simplify-code findings on the #66222 salvage:
- consume_detached_task_result moves to agent/async_utils.py (shared home);
gateway/run.py and the Discord adapter both had near-identical copies of
the same pattern (a third lives in the telegram adapter). One canonical
implementation, both new callsites import it.
- Reconnect backoff formula min(30 * 2^(n-1), 300) was copied verbatim at
3 sites in run.py (primary watcher x2, secondary-profile reconnect), the
third hardcoding the cap. Hoisted to module-level _reconnect_backoff()
with a single _RECONNECT_BACKOFF_CAP so a future tune can't silently
miss one path.
Behavior-preserving: 70 gateway teardown/liveness/reconnect tests green.
Replace REST-based Discord liveness probe with local WebSocket/heartbeat
state detection. REST success doesn't prove Gateway event delivery — a
half-closed WebSocket can leave Bot.start() alive while REST returns 200.
Now samples ready/open/ACK state and heartbeat latency; consecutive
unhealthy samples emit one retryable fatal code so GatewayRunner rebuilds
the adapter through the existing reconnect path.
Also fixes three lifecycle gaps in the recovery path:
1. asyncio.wait_for() can remain blocked if adapter cleanup swallows
cancellation — now uses bounded asyncio.wait() with task detachment.
2. Multiplexed secondary-profile adapters had no profile-scoped reconnect
owner — now uses one runner-owned reconnect slot per profile.
3. An in-flight turn could send its final text through the disconnected
adapter after a replacement was registered — now resolves the live
same-profile replacement for unsent final responses only (message IDs
never migrate, edits/deletes stay on the old transport).
Adds an opt-in Linux/systemd event-loop watchdog (gateway.systemd_watchdog_seconds,
default 0) for the failure mode where the whole asyncio loop stops making
progress and no in-process liveness task can run. stdlib-only sd_notify,
Type=notify/WatchdogSec generation, READY/STOPPING lifecycle.
Co-authored-by: 王鑫 <wx.xw@bytedance.com>
The staleness check's bespoke mtime memo keyed only on the user
config.yaml, but load_config() merges the managed-scope config
(HERMES_MANAGED_DIR/config.yaml, /etc/hermes) whose leaf keys win. A
managed honcho.timeout with no user config.yaml made the memo cache
'no timeout' while _build resolved the managed value — the same
perpetual-rebuild mismatch this PR fixes for honcho.json. A managed
timeout edit was likewise invisible while the user file's mtime stayed
put.
load_config_readonly() is already cached on both files' signatures plus
the env-ref snapshot, so use it instead of duplicating that
invalidation logic; the defensive deepcopy the old memo existed to
avoid is skipped by the readonly variant. Drive the rebuild test
through a real config.yaml and add a HERMES_MANAGED_DIR regression
test covering stable reuse and managed-timeout edits.
The staleness check added in #66052 resolved the timeout from env,
config.yaml, and the default only, while the build path also reads the
honcho.json host block (timeout/requestTimeout). With a timeout
configured in honcho.json, the two permanently disagreed: every
no-config get_honcho_client() call — i.e. every HonchoSessionManager
.honcho property access — interpreted the mismatch as a config change
and tore down and rebuilt the client, defeating the singleton on the
hot path it was meant to protect.
Teach the check to read honcho.json through the same host-aware chain
as from_global_config, memoized on the file's mtime_ns so the per-call
cost stays one stat(). A genuine honcho.json timeout change is now also
detected, extending #57437 to that config surface.
Community feedback (@LSanapalli on X): the inline task-creation form is
cramped inside a ~280px column with no way to resize; board-level
workspace defaults can't be changed after board creation; and users
believe they must block a task, comment, then unblock just to talk to
a worker.
- Create-task dialog: replace the inline column form with a centered
modal (reuses hermes-kanban-dialog chrome, 36rem wide) with labeled
fields for title, assignee, priority, skills, workspace kind/path,
goal mode, and parent task. Same request shape; Enter/Escape behavior
preserved; submit disabled until a title is present.
- Board settings dialog: new Settings button in the board switcher opens
a modal to edit display name, description, and the board-level default
project directory (default_workdir). PATCH /boards/:slug now accepts
default_workdir (validated absolute existing dir; empty string clears;
omitted leaves unchanged) and returns the recomputed
default_workspace_kind so task-creation defaults follow immediately.
- Comment workflow hint: the task drawer's comment box now explains that
comments land on the thread immediately and reach the worker on its
next run/kanban_show() — no block/unblock dance needed — with a fuller
tooltip for when blocking IS the right tool.
- i18n: new keys optional in the kanban namespace with English fallbacks
in the bundle (established pattern; avoids churning 17 locale files).
- Docs: dashboard section updated for the dialog + Settings button.
Keep each xAI OAuth auth-add login as an independent manual device-code pool entry and recognize xAI personal-team spending-limit 403 responses as billing exhaustion. Preserve the structured top-level error message so the failed credential is quarantined and the next healthy account is selected without attempting a pointless token refresh.
Route direct xAI HTTP consumers through the credential pool as well. Proactive and 401-reactive refreshes update the exact issuing manual entry, preserve validated xAI base URL overrides, and serialize single-use refresh-token rotation across concurrent pool instances.
Follow-ups for the consolidated salvage:
- Memoize the config.yaml-derived timeout on the file's mtime_ns so the
rebuild-on-timeout-change check from PR #57437 costs one stat() per
get_honcho_client() call instead of a full YAML load on the hot path.
- hermes honcho status now displays the host-block-resolved
dialecticCadence (remnant from PR #63776, whose runtime fix landed
in #62290).
- AUTHOR_MAP entries for the salvaged contributor emails.
The Honcho client singleton cached the HTTP timeout at first build.
In long-lived processes (gateway, dashboard), changing the timeout
via config.yaml or HONCHO_TIMEOUT had no effect until restart.
Track the resolved timeout alongside the cached client and compare
on each get_honcho_client() call. When the timeout differs, reset
the singleton so the next call rebuilds with the new value.
Fixes#57347
Fixes Honcho client initialization for setup-generated configs that store
connection details in a named host block (e.g. "local"). Previously:
- base_url was only read from flat config root, not from host_block.
- resolve_active_host() ignored defaultHost and always used the Hermes
profile key ("hermes"), so the host block lookup returned {} and the
api_key was also lost.
FixesNousResearch/hermes-agent#61661
honcho_reasoning's minimal tier hard-caps Honcho's dialectic output at 250
tokens combined with the model's own hidden reasoning tokens. Confirmed via
direct honcho-api server logs that a multi-fact query ("summarize known
facts about this peer and communication preferences") run at
reasoning_level=minimal gets cut off mid chain-of-thought at exactly
output_tokens=250, before the model ever reaches a synthesized answer.
low/medium/high/max fall back to Honcho's much larger global dialectic
default and don't hit this cap. Since dialecticDynamic is on by default and
the calling model picks reasoning_level itself via this tool parameter, the
fix is to make the tradeoff explicit in the parameter description so the
model defaults to low unless the query is genuinely a single-fact lookup.
Schema-description-only change; the shared dialectic_max_chars truncation
cap in session.py's dialectic_query() is a separate fix (its own PR).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_fetch_session_context was using user_peer_id directly as observer
when calling _fetch_peer_context, bypassing _resolve_observer_target.
This caused honcho_context to return empty data because the observer
perspective was wrong (user instead of hermes/assistant).
Fixed by resolving observer via _resolve_observer_target(session, 'user'),
consistent with all other call sites (get_peer_card, honcho_search, etc.).
queryRewrite (default off) gates the latest-message rewrite so the
extra auxiliary LLM call is opt-in. firstTurnBaseWait and
firstTurnDialecticWait expose the turn-1 bounded waits in seconds
(0 disables). All three resolve host-block-first like every other
field. Also pins per-host timeout resolution with tests.
Move query_rewrite from the honcho plugin to plugins/memory/ and
rename the auxiliary task key honcho_query_rewrite ->
memory_query_rewrite so any memory provider can use the same
rewrite path and model/timeout config block. No behavior change.
dialecticMaxChars (default 600) is documented as the budget for the dialectic
supplement auto-injected into the system prompt every turn — a small recurring
cost that is correct to bound tightly. But dialectic_query() applied that cap
unconditionally, so explicit honcho_reasoning tool calls — where the model
deliberately spends a turn asking for a synthesized answer — were silently
truncated mid-word to 600 chars with a trailing " …", no error surfaced. The
full answer is returned by Honcho server-side; the clip happens client-side.
The auto-injection path already has its own token-based budget (contextTokens,
enforced in prefetch() via _truncate_to_budget), so the char cap's real job is a
cheap always-on guardrail for that recurring injection. Explicit tool results
are already bounded server-side by Honcho's dialectic MAX_OUTPUT_TOKENS and don't
need the injection cap — sibling tools (honcho_search, honcho_context) don't
post-clip their results either.
Add apply_injection_cap (default True, preserving current behavior) to
dialectic_query(); the honcho_reasoning tool handler passes False so it returns
Honcho's full synthesized answer. Auto-injection is unchanged. Tests cover both
the capped injection path and the uncapped tool path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
honcho_conclude's delete action was unreachable in practice: no tool ever
surfaced a real conclusion id for the model to pass as delete_id.
honcho_search only searches the separate Message resource space, and the
SDK's ConclusionScope.list()/.query() (which do return real Conclusion.id
values) were never wired into any tool.
Adds an optional list mode to honcho_conclude (query to search, omit to
browse recent conclusions), backed by a new
HonchoSessionManager.list_conclusions(). No new tool, no changes to the
create/delete signatures or their conclusions_of() routing.
injectionFrequency='first-turn' returned empty for the entire
prefetch_context() method on turns 2+, which blocked the dialectic
supplement from being consumed and injected. The dialectic has its
own cadence (dialecticCadence) and must continue to fire and inject
independently of the base context layer.
Now first-turn mode gates only Layer 1 (base context: representation
+ card), letting Layer 2 (dialectic supplement) flow through its
normal consumption path on every turn.
Also fixes all remaining tests that passed dialecticCadence via
cfg_extra={'raw': {...}} to use the typed dialectic_cadence field.
injectionFrequency, contextCadence, and dialecticCadence were read
only via raw.get() which checks root-level keys in honcho.json.
Settings placed inside hosts.<name> (the normal per-host location)
were silently ignored, falling back to defaults.
- Add typed dataclass fields: injection_frequency, context_cadence,
dialectic_cadence with host-block-first resolution chains matching
the pattern used by all other config fields.
- Update HonchoMemoryProvider.initialize() to read from typed cfg
fields instead of cfg.raw directly.
- Fix search_context tests that mocked _honcho directly — the
.honcho property getter calls get_honcho_client() which overwrites
the backing field, so use patch.object on the property instead.
- Add injection_frequency and context_cadence config override tests.
HonchoClientConfig timeout/requestTimeout resolution skipped the per-host config
block, silently dropping a host-scoped timeout and falling through to the global
config.yaml value (or the default). Add the host block at the front of the
resolution chain, consistent with every other field (base_url, api_key, etc.).
Symptom: Honcho logs show a dialectic answer was generated, but Hermes never
injects it — intermittently.
Root cause: the dialectic supplement that queue_prefetch() fires at the end of
turn N is stored pending (fired_at=N) for consumption by turn N+1's prefetch().
But prefetch()'s trivial-prompt guard returned early BEFORE the consumption
block. So when turn N+1's prompt was trivial ('ok', 'yes', 'continue', a slash
command), the ready result was never consumed, and a few turns later the
stale-discard guard dropped it. Generated by Honcho, never seen by the model.
The dependence on 'is the consuming turn trivial?' is why the loss looked random.
Fix: trivial turns now consume and inject a ready, non-stale pending result while
still spending no new work (no base-context fetch, no new dialectic fire). A
trivial ack shouldn't generate context, but it shouldn't destroy an answer
already computed for that exact turn. Extracted the pop+stale-check into a shared
_consume_pending_dialectic() used by both the trivial path and the normal path so
they age-check identically.
Preserved (covered by new tests): genuinely stale results are still discarded on
trivial turns; trivial turns still fire no new work; a trivial turn with nothing
pending still injects nothing.
Adds regression tests for inject-on-trivial and discard-stale-on-trivial.
A brand-new session injected no Honcho context on the user's first message —
the peer card/representation only showed up from turn 2 onward. The base-context
fetch was fired asynchronously and popped in the same synchronous pass, so it
always lost the race on turn 1 (a background thread can't finish inside one
pass), leaving the first response with zero recalled context.
Fetch the base layer (representation + card + summary) synchronously with a
bounded timeout on turn 1 so the peer card is injected immediately; subsequent
turns still consume the background-refreshed result primed by queue_prefetch().
The wait is bounded by _FIRST_TURN_BASE_TIMEOUT and tightened further by a small
configured request timeout (fail-fast deployments / tests).
Two related first-turn/dialectic reliability fixes ride along:
- first-turn dialectic no longer double-fires: if a prewarm .chat() thread is
already in flight from session init, turn 1 waits briefly for it instead of
firing a second (duplicate) call that also blocked the first response. The
first-turn wait is decoupled from a large host timeout (a 60s host timeout
must not block the first response for 60s) via _FIRST_TURN_DIALECTIC_CAP,
while still honoring a tight configured timeout.
- empty-pass propagation guard in multi-pass dialectic: at depth > 1 each pass
feeds the prior pass's output into the next prompt. If a pass returned empty
(e.g. a reasoning model that spent its whole budget thinking), the next prompt
carried a blank assessment (the "empty spot" seen in Honcho request logs). Now
only non-empty prior results feed dependent passes; if all priors are empty,
re-issue the base prompt instead of referencing nothing.
