The xapp-<num>-<hash> format used by Slack App-Level / Socket Mode
tokens was missing from both agent/redact.py prefix patterns and
gateway/run.py gateway secret patterns, so SLACK_APP_TOKEN values could
leak through to chat users even with security.redact_secrets enabled.
Adds an anchored xapp-\d+- pattern to both redaction paths.
The cross-process coherence guard (#45966) compares the session's
on-disk message_count against the snapshot stored next to the cached
agent, and rebuilds the agent on a mismatch. The guard is correct
when the cache snapshot and the live count both refer to the same
DB row. But the agent cache is keyed by session_key, which can
group multiple conversation threads (different session_ids) under
the same key — and the message_count values belong to DIFFERENT
DB rows.
When the user switches from session A to session B under the same
session_key, the cache hit returns A's cached agent. The guard then
compares A's snapshot count (A.message_count) against B's live count
(B.message_count) — they are NEVER equal because they track
different conversations — and invalidates the cache. Every session
switch busts the prompt cache and forces a fresh agent build. The
post-turn re-baseline (#46237) made it worse: it reads the live
count from the CURRENT session_entry.session_id, so each switch
overwrites the original snapshot with the new session's count,
causing the very next switch BACK to the original session to fire
the guard again.
This is the bug from #54947 (P0, sweeper:risk-session-state,
sweeper:risk-caching).
Fix:
* Record the snapshot's session_id alongside the message_count in
the cache tuple: (agent, sig, mc, session_id) — a 4-tuple. The
cache build at the AIAgent construction site stores the active
session_id.
* The cache-hit guard skips the cross-process count comparison
when the active session_id differs from the snapshot's
session_id — the comparison is meaningless across different DB
rows, so the agent is REUSED without invalidation. The cross-
process guard still fires when the session_id matches and the
live count differs (genuine cross-process write on the SAME
session).
* _refresh_agent_cache_message_count checks the snapshot's
session_id: when it differs from the current session_id, the
snapshot is intentionally left untouched (overwriting it would
corrupt the original conversation's baseline and cause the
switch-back to fire the guard). The legacy 3-tuple shape (no
session_id) is still re-baselined as before.
* Backward-compat:
- 2-tuple (agent, sig) — unchanged, opts out of the guard.
- 3-tuple (agent, sig, mc) — unchanged behavior, standard
cross-process check.
- pending sentinel — unchanged, untouched by re-baseline.
- new 4-tuple (agent, sig, mc, session_id) — full session_id-
aware guard with skip on mismatch.
Tests:
* tests/gateway/test_session_id_cache_coherence.py — 7 tests
covering L1-L5 from LAYERS.md:
- L1 session_id switch must REUSE
- L2 cache tuple records snapshot's session_id
- L3 re-baseline skips when session_id differs
- L4 same-session_id turns still re-baseline (#46237 holds)
- L5 legacy 2-tuples and pending sentinels untouched
- legacy 3-tuple (no session_id) still guarded (#45966 holds)
- 3-tuple transitions to 3-tuple (not 4-tuple) on re-baseline
No regressions in 70 existing tests in test_agent_cache.py or 137
related session tests. Co-authored with #52197 (deferred cleanup
of evicted agents); both fixes compose cleanly.
The cross-process cache-coherence guard (#45966) compares a session's
on-disk message_count against a snapshot stored next to the cached agent,
rebuilding the agent on a mismatch so a foreign writer (e.g. the dashboard
backend) can't leave the in-memory transcript stale.
On a fresh gateway conversation the post-turn re-baseline
(_refresh_agent_cache_message_count) ran BEFORE the first-turn `session_meta`
marker row was appended to the transcript. That append goes through
append_to_transcript -> append_message, which increments message_count
unconditionally. So the snapshot was left exactly one short of the live
count, and on turn 2 of every fresh conversation the guard mistook this
process's own session_meta write for a foreign write, evicting and rebuilding
the cached agent — silently busting the per-conversation prompt cache the
cache exists to protect.
Move the re-baseline to after the turn's full transcript persistence block
(including the session_meta append and the compression session_id swap). The
snapshot now matches the live count, so the guard fires only on genuinely
foreign writes. This also makes the call honor its own documented contract of
using the compaction-updated session_id.
Adds a regression test that drives the real _handle_message_with_agent
against a real SessionDB and asserts the invariant: after a fresh first turn,
snapshot == live message_count, so the next turn's guard reuses the cached
agent. Fails before this change, passes after.
The cross-process cache-coherence guard (#45966) re-baselines the cached
agent's message_count only on the external-turn boundary (#46237, at
_handle_message_with_agent). The in-band queued (/queue) follow-up recurses
into _run_agent mid-chain with the stale build-time snapshot, so the
follow-up's guard sees the first turn's own writes as a mismatch and rebuilds
the agent -- re-introducing the every-turn rebuild / prompt-cache destruction
#46237 set out to prevent, on the in-band path. Re-baseline before the
recursion, symmetric with the accepted external-path fix.
models_dev.py's fetch uses a synchronous requests.get(timeout=15). Called
from the async gateway message handlers, it blocked the event loop for up
to 15s, starving Discord heartbeats and causing ClientConnectionResetError
disconnects.
Adds get_model_context_length_async() which offloads the entire sync
resolution chain to a worker thread via asyncio.to_thread(), and switches
the two async gateway call sites (_prepare_inbound_message_text,
_handle_message_with_agent) to await it. The loop stays responsive; the
sync path remains the single source of truth for the cache.
Salvaged from PR #22753 by @itenev. Follow-up: dropped the unused
fetch_models_dev_async/lookup_models_dev_context_async aiohttp variants
from the original PR (dead code with zero callers that had drifted from
the sync cache logic) — the to_thread wrapper already runs the sync path
off-loop, so they were redundant.
A delayed fatal-error notification from an adapter instance that has
already been replaced by a successful reconnect (a different adapter
object now owns the platform slot) was still processed: it overwrote
the platform's runtime status back to retrying/fatal and could
re-queue an already-healthy platform for reconnection.
Snapshot the current owner of the platform slot at the top of
_handle_adapter_fatal_error and bail out before any side effect when
it belongs to a different, already-installed adapter.
_handle_polling_network_error's chained retry never updated
self._polling_error_task, so the reentrancy guard shared with the
heartbeat loop and the pending-updates probe went stale mid-recovery,
letting more than one recovery attempt run concurrently against the
same adapter. Combined with a TOCTOU window in
_handle_adapter_fatal_error (the adapter was only removed from
self.adapters in a finally block after awaiting disconnect()), two
concurrent fatal notifications for the same adapter could both pass
the "still installed" check and call disconnect() twice, which is
where the reported "'NoneType' object has no attribute 'updater'"
originates once self._app is cleared by the first call.
- Reassign the chained retry task to self._polling_error_task so the
guard reflects an in-flight recovery.
- Capture self._app in a local variable across the stop/start_polling
sequence instead of re-reading self._app between awaits.
- Claim (pop) the adapter from self.adapters before awaiting
disconnect() in _handle_adapter_fatal_error, not after, closing the
TOCTOU window for a concurrent notification on the same adapter.
The handoff seed path inlined its own int(chat_id) > 0 private-chat
check; delivery.py already had the identical heuristic. Promote it to
a public name and reuse it from both sites instead of duplicating.
When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue #18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).
Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.
Credit to @dontcallmejames for the diagnosis and original patch.
Mitigates indirect prompt injection (CWE-863) in Slack thread context.
When the bot is mentioned mid-thread for the first time, _fetch_thread_context
pulls the full thread via conversations.replies and prepends every reply to
the LLM prompt. Replies from senders not on the allowlist were rendered
identically to authorised senders, letting a third party in a shared channel
inject instructions the model might act on when answering the next authorised
message.
- BasePlatformAdapter.set_authorization_check / _is_sender_authorized, registered
by GatewayRunner._make_adapter_auth_check() with a closure over the existing
_is_user_authorized chain (platform/global/group allowlists, allow-all flags,
pairing store all stay the single source of truth — no env-var re-parsing).
- Tags non-bot thread messages whose sender fails the auth check with an
[unverified] prefix; strengthens the header with soft guidance only when at
least one unverified message is present, so setups without an allowlist see
no behaviour change.
- Wired into all three adapter-init sites in run.py (start, reconnect watcher,
restart) so the reconnect path is covered too.
Softened wording: adapted from the original [untrusted] tag to [unverified]
and non-accusatory header framing — the label reflects allowlist status, not
a judgment about the person. Adapter relocated to plugins/platforms/slack/
since the PR was authored.
Salvaged from #17059.
/queue rebuilt the queued MessageEvent with only text/type/source/
message_id/channel_prompt, silently dropping any photo, document, voice,
or reply context attached to the command. The deferred turn then ran with
the attachment lost. Carry the full payload through, and accept a /queue
that has media but no prompt text (e.g. "/queue" as an image caption).
Salvaged from #13913 by @ypwcharles — the gateway busy-session/queue
infrastructure was rewritten since that PR (Telegram moved to
plugins/platforms/, /queue now uses the FIFO chain), so the media fix is
reimplemented against the current handler; the PR's batching and
busy-bypass changes targeted code paths that no longer exist.
Co-authored-by: ypwcharles <92324143+ypwcharles@users.noreply.github.com>
Adds gateway.platform_connect_timeout (default 30s) to DEFAULT_CONFIG and
bridges it to the internal HERMES_GATEWAY_PLATFORM_CONNECT_TIMEOUT env var
at gateway startup, following the existing gateway_timeout config->env
pattern. The env var remains the manual-override escape hatch and wins if
set explicitly; otherwise config.yaml supplies the value. This closes the
issue's documentation/config-surface request (#19776 suggestion 2) on top
of the adapter ready-wait fix, so users no longer need an undocumented env
var to raise the Discord connect timeout.
Refs #19776
Follow-up to liuhao1024's #46924. Route plain-text approval replies
through the canonical /approve and /deny handlers (resolve thread, resume
typing, return localized confirmation) and deliver that confirmation back
to the user — previously a plain 'yes' resolved silently. Synthesize a
literal '/'-prefixed command so get_command_args() parses always/session
modifiers on every platform (is_command() only recognizes '/'). Add E2E
tests covering approve/deny/always/session vocab plus the no-pending and
unrelated-text fall-through cases.
When the agent is blocked waiting for a dangerous-command approval,
plain-text responses like "yes" or "approve" were being steered into
the running agent instead of being delivered to the approval handler.
This meant approval via messaging platforms (Signal, Telegram, etc.)
never succeeded — the user's response was consumed by the steer logic
and the approval timed out.
Add an early check in `_handle_active_session_busy_message` that routes
approval-like responses ("yes", "approve", "deny", etc.) to the
approval handler when `has_blocking_approval()` is true for the session.
Fixes#46866
(cherry picked from commit b37ec1e0fd)
The STT-failure enrichment templates injected setup instructions —
"no STT provider is configured", "a direct message has already been
sent", and a "hermes-agent-setup" skill mention — into the LLM-visible
prompt. That text persists in conversation history, so after one STT
failure the model kept volunteering Whisper/Vosk setup advice on every
later voice turn, even after transcription started working (observed in
prod on gpt-5-nano). The gateway also fired a hardcoded English notice
via _stt_adapter.send(), producing a second, wrong-language reply that
TTS then spoke aloud.
- Neutralize all enrichment templates: success passes the transcript
through as a plain quoted line; every failure branch emits a single
[voice message could not be transcribed] marker.
- Move the operator-facing failure cause to logger.info so it stays
diagnosable in container logs without leaking into the prompt.
- Remove the hardcoded English _stt_adapter.send() notice; the LLM now
produces one coherent reply in the user's language.
- Update the gateway STT tests to assert the neutral contract.
Co-authored-by: Hermes Agent <noreply@nousresearch.com>
Two independent bugs evicted the cached gateway AIAgent on every turn,
preventing the prompt cache from ever warming:
1. Model normalization mismatch: the post-run fallback-eviction check
compared _agent.model (stripped in AIAgent.__init__) against the raw
_resolve_gateway_model() config string. For vendor-prefixed config on
native providers (e.g. 'deepseek/deepseek-v4-pro' vs 'deepseek-v4-pro')
this was always unequal, so the agent was evicted after every
successful run. Normalize _cfg_model the same way (skip aggregators).
2. Discord triggering message_id leaked into the cached system prompt via
build_session_context_prompt()'s Discord IDs block. message_id changes
every turn, so the agent-cache signature (computed from the ephemeral
prompt) changed every Discord turn -> rebuild every message. The id is
now injected per-turn into the user message (where per-turn content
belongs and does not touch the cache signature); the cached IDs block
carries a static pointer to it, preserving reply/react/pin via the
discord tools.
Adapted from #28846. Bug #1 fix is the contributor's; bug #2 reworked to
be non-destructive (keeps the triggering-id capability instead of deleting
it). Redundant auto-reset eviction (already on main via #9893/#48031) and
the wrong-premise reset_context_note plumbing from the original PR were
dropped.
Co-authored-by: Hermes Agent <hermes@nousresearch.com>
GatewayStreamConsumer.run() processed queued deltas in an infinite loop
with no check on whether the session was still current. On /new or /stop
mid-stream, the consumer kept editing and delivering stale response
fragments alongside the 'Session reset!' ack.
PR #11016 (b7bdf32d) fixed the runner side via sentinel promotion/release
but left the stream consumer unguarded. Every other async callback in
run.py already bails via _run_still_current(); the stream consumer was
the only one missing it.
- stream_consumer.py: optional run_still_current callback, checked at the
top of the run() loop; returns early when the session is stale.
- run.py: pass the existing _run_still_current closure at both call sites
(proxy path and agent path).
- tests: TestRunStillCurrentGuard — immediate staleness, mid-stream
staleness, always-current, no-callback default, pending-finish.
Co-authored-by: jasonQin6 <39369769+jasonQin6@users.noreply.github.com>
* feat(display): friendly human-phrased tool labels for built-in tools
Built-in tools now render ChatGPT-style status verbs ('Searching the web
for ...', 'Reading <file>', 'Browsing <url>') on the CLI spinner and
gateway/desktop tool-progress instead of the raw tool name.
- agent/display.py: _TOOL_VERBS map + build_tool_label() + set/get
friendly-labels flag (default on). Custom/plugin/MCP tools fall back to
the raw preview; verbose gateway mode left untouched (debug surface).
- tool_executor.py / tui_gateway / gateway: route the three spinner sites,
the TUI _tool_ctx, and the gateway all/new progress line through the label.
- config: display.friendly_tool_labels (default True, per-platform aware).
Zero new core tool / schema footprint — pure display layer.
* docs: add PR infographic for friendly tool labels
* fix(display): preserve arg preview in gateway friendly labels + update tests
The first gateway pass re-derived the label from the callback's `args`, which
is empty ({}) at the gateway tool.started callsite — the command/query lives in
the `preview` string, so terminal rendered as a bare '💻 Running' and dedup
collapsed consecutive commands. Now the gateway prefixes the verb onto the
already-computed preview via get_tool_verb/tool_verb_connector/verb_drops_preview,
preserving the command/url/query. CLI spinner path (real args) keeps build_tool_label.
Tests: update test_run_progress_topics exact-format assertions to the friendly
form ('💻 Running pwd'), add a format-agnostic preview extractor for the
truncation tests (works for both quoted-legacy and verb-prefixed output).
* test(tui): update resume-display context to friendly tool label
_tool_ctx now uses build_tool_label, so the desktop resume-view context for a
search_files turn reads 'Searching files for resume' instead of the bare
'resume' preview — consistent with live tool-progress. Update the assertion.
* test(tui): harden no-race worker test against sibling shard leakage
test_session_create_no_race_keeps_worker_alive flaked under -j 8: a daemon
build thread leaked from a prior session.create test in the same shard process
fires close/unregister against its own (foreign) session_key after this test
patches the global approval hooks, polluting the captured lists. Scope the
assertions to this session's own session_key so the regression intent
(this session's worker/notify must survive) is preserved while the test
becomes immune to shard composition. Not related to friendly-tool-labels.
The topic-mode helpers (_telegram_topic_mode_enabled,
_recover_telegram_topic_thread_id, _record/_sync_telegram_topic_binding,
_is_telegram_topic_lane/_root_lobby, _normalize_source_for_session_key,
_telegram_topic_new_header, _schedule_telegram_topic_title_rename, and the
base.py _apply_topic_recovery hook) each run a synchronous SessionDB read or
write. They reach the event loop through async handlers, so a contended
state.db froze the loop the same way the handoff watcher did.
These helpers already run off-loop in the run_sync thread-pool closure, so
they are proven thread-safe there. Rather than colour them async, loop-side
callers now invoke them via asyncio.to_thread(...); the executor callers are
unchanged. Inside the helpers the SessionDB handle is unwrapped to the sync
door (getattr(db, '_db', db)) since they always run on a worker thread, and
AIAgent construction + query_session_listing are handed the sync SessionDB
directly. base.py wraps its single _apply_topic_recovery call in to_thread.
The guard is now alias-aware (catches db = getattr(self, '_session_db', None);
db.method(...)) and enforces the offload contract: the offloaded sync helpers
may never be called bare on the loop. Sibling test fixtures wrap their injected
SessionDB in AsyncSessionDB to match how the gateway holds it.
The migration's call-site sweep keyed on the literal self._session_db.
spelling and missed calls bound to a local first
(db = getattr(self, '_session_db', None); db.method(...)). Convert the
three in async contexts: get_telegram_topic_binding in the topic-rename
coroutine, and the two update_session_model sites on the model-switch path.
Add a generic suppress_notification flag to the drain-request marker. When a
drain that ends in process exit (e.g. a NAS auto-update image migration on the
always-on Hermes Cloud fleet) is flagged, the gateway skips ONLY the
home-channel 'gateway shutting down' broadcast — the operator-flavoured ping
that would otherwise fire on every routine auto-update, dozens of times a day.
The per-active-session interrupt ping is ALWAYS kept: on a drained shutdown
it's empty by construction, and in the force-interrupt (deadline-exceeded) case
it carries the user-valuable 'your task was cut off, message me to resume' hint.
The gateway stays agnostic about WHY a drain is quiet (generic boolean, not a
kind enum); the policy of which drain causes set the flag lives in the caller
(NAS). Default-false so legacy/operator drains behave exactly as before. The
reader reuses the NS-570 epoch-staleness check so an orphaned marker on the
durable volume can never silence a fresh gateway's legitimate broadcast.
- drain_control.py: write_drain_request gains suppress_notification; new
drain_notification_suppressed() reader (current-epoch + truthy flag).
- web_server.py: /api/gateway/drain reads + echoes the flag.
- run.py: _notify_active_sessions_of_shutdown skips the home-channel loop only.
Tests prove: flag round-trips; home-channel suppressed when set, kept when
unset; active-session ping always fires; stale/legacy/corrupt markers never
suppress.
Fixes#14238. During a compression/session split at the response
boundary, the interim callback delivered unrelated commentary, setting
response_previewed=True. The suppression logic treated that as proof the
final reply had been delivered and skipped the normal send — the response
was persisted to the child session but never sent to chat.
Only suppress the normal final send when the stream consumer confirms
final delivery (final_response_sent / final_content_delivered) or the
exact final response text was delivered as a preview.
Relocate marco0158's eviction into the dedicated auto-reset cleanup block
(single source of truth for dropping session-scoped transient state) and
add an AST invariant pinning _evict_cached_agent into that block. Add
AUTHOR_MAP entry for marco0158.
When a session is auto-reset by daily schedule, idle timeout, or suspended
state, the agent cache was not being cleared. This caused the old agent's
context_compressor._previous_summary to leak into the new session, mixing
old conversation history into new compaction summaries.
This was the root cause of the "skin making history" appearing after
compaction in fresh sessions reported by the user.
Follow-up to #9893 which only handled compression_exhausted case.
Changes:
- Add _evict_cached_agent(session_key) call after was_auto_reset check
- Covers daily, idle, and suspended auto-reset scenarios
- Matches the behavior of manual /reset command
Related tests: test_session_boundary_hooks, test_async_memory_flush,
test_session_reset_notify, test_session_reset_fix - all passing.
Two of the three fixes from PR #6660 (the cli.py reopen_session change is
moot — that raw _conn.execute reopen block no longer exists on main).
- gateway/run.py: stop sending raw type(e).__name__ and str(e)[:300] to
end users on chat platforms. Exception text from LLM providers can leak
API URLs, file paths, and partial credentials. Return a generic message;
keep curated status hints for known HTTP codes; full detail stays in logs.
- gateway/platforms/webhook.py: validate pr_number (positive int) and repo
(owner/name regex) before passing to the 'gh pr comment' subprocess.
Payload-controlled values could otherwise inject gh flags (--help, a
different --repo). List-form subprocess means this is arg injection, not
shell injection, but validation is still correct.
Co-authored-by: aaronagent <1115117931@qq.com>
* fix(security): redact secrets in background process + foreground env-dump output
Terminal-output redaction was incomplete (#43025):
- Gap 1: process(action=poll/log/wait) returned background stdout verbatim —
no redaction at all. A background printenv/server/test emitting a key leaked
raw to the model, session.db, and CLI display. Same for the gateway
background-process watcher's completion/progress notifications.
- Gap 2: the foreground terminal path hardcoded code_file=True, which skips the
ENV-assignment pass, so an opaque token (no vendor prefix) from env/printenv
leaked even there.
Adds agent.redact.redact_terminal_output(output, command) as the single policy
for ALL terminal-output surfaces: env-dump commands (env/printenv/set/export/
declare) get the ENV-assignment pass (code_file=False) to mask opaque tokens;
other commands stay on code_file=True to avoid false positives on source dumps.
Wired into terminal_tool, process_registry (_handle_process boundary), and the
gateway watcher. Respects security.redact_secrets (no force) — opt-out preserved.
* docs: add infographic for #43025 terminal-output redaction fix
Config-backed quick_commands bypassed the admin-only slash gate. The
early gate in _handle_message only fires for registry-known commands
(is_gateway_known_command), but quick_commands are never in the gateway
registry, so they reached the type:exec dispatch sink unchecked. An
allowlisted non-admin gateway user could invoke admin-only quick
commands — including shell exec in the gateway process — even when the
operator set allow_admin_from / user_allowed_commands to lock them out.
Apply _check_slash_access(source, command) at the quick_commands
dispatch site (the single exec chokepoint, cold-path only) using the
raw typed name. Admins and users with the command in
user_allowed_commands still run it; backward-compat (no policy set)
is unaffected.
Fixes#44727.
Co-authored-by: maxpetrusenko <max.petrusenko.agent@gmail.com>
Co-authored-by: zapabob <1920071390@campus.ouj.ac.jp>
#35994 moved /new reset cleanup off the loop, but _cleanup_agent_resources
(agent.close() subprocess teardown; shutdown_memory_provider() plugin IO) was
still called INLINE on the event loop from three other sites:
- _session_expiry_watcher (5-min idle sweep) — live loop
- _handle_message_with_agent cache-hygiene re-eviction — live loop
- _finalize_shutdown_agents / stop() idle-cache loop — shutdown
A wedged memory provider on any of these froze the loop: bot goes silent,
runtime-status updated_at heartbeat stops advancing, and SIGTERM can't be
serviced (requires kill -9) — exactly the #53175 zombie pattern.
Adds _cleanup_agent_resources_off_loop: a bounded (30s) worker-thread offload
mirroring the #35994 reset fix, and routes all four sites through it.
Extend the gateway noisy-status filter beyond Telegram so internal
compression lifecycle messages stay in logs instead of spamming Discord,
Slack, and other messaging channels.
A custom_providers config that names the model under model.name (or
model.model) resolved to an empty model, so the API request went out
with model= — HTTP 400 from OpenAI-compatible backends. Display paths
(hermes status/dump) already read model.name and showed the model,
making the failure silent.
The model id was read via 'default or model' at ~14 independent sites
(cli, gateway, cron, curator, oneshot, fallback, profiles, ...), none
of which honored 'name'. Rather than patch every site, canonicalize at
the single load/save chokepoint: _normalize_root_model_keys() now
promotes model.model/model.name -> model.default (precedence
default > model > name) and drops the stale alias, so every reader —
present and future — sees a populated default and config.yaml is
migrated canonical on next save. The gateway, which bypasses
load_config(), replays the same normalization in _load_gateway_config().
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
Credit: root-cause analysis and fix direction from @Bartok9 (#34502,
first) and @v86861062 (#34527).
A restart now interrupts in-flight agents immediately rather than holding
the gateway open for a grace window. The previous 180s default coupled two
independently-set timers: the gateway's own drain timer and systemd's
TimeoutStopSec. On a stale unit where TimeoutStopSec < drain, systemd
SIGKILLed the gateway mid-cleanup, leaving a stale lock that made the next
startup exit immediately ('already running') — an infinite crash loop under
Restart=on-failure (#31981).
Setting drain to 0 makes the mismatch structurally impossible: with drain 0
the generated unit gets TimeoutStopSec=90 against a near-instant drain, so
systemd never kills mid-cleanup. Contract: restart the gateway, in-flight
work stops. A grace window large enough to 'save' a long agent turn would
have to outlast an unbounded task, which is impossible.
Also fixes the stale-unit warning's suggested command
(hermes gateway service install --replace -> hermes gateway install --force);
the former subcommand does not exist.
Closes#31981
Salvage of NousResearch/hermes-agent#41498 (0-CYBERDYNE-SYSTEMS-0).
- Leave response_previewed false on partial_stream_recovery so gateway
fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
not only for empty or very short fragments (#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
bounded wait of restart_drain_timeout + 5s.
A WebUI/TUI session whose last turn died mid-tool-loop (stale-timeout kill,
interrupt, or process restart before the tool result was written) persists a
dangling assistant(tool_calls) or interrupted assistant->tool tail. The
messaging gateway already strips these tails before replay (the #49201 fix),
but the TUI/WebUI resume path fed db.get_messages_as_conversation() straight
in as the agent's conversation_history with no cleanup. The model re-issued
the unanswered call on every resume -- including after a full WebUI + Gateway
restart, since the poison lives in the SessionDB, not memory -- leaving the
session permanently 'thinking'. Only deleting the session recovered it.
- Extract the two strippers + helper from gateway/run.py into a shared
agent/replay_cleanup.py (sanitize_replay_history wraps both).
- gateway/run.py re-exports under the historical private names; messaging
behavior unchanged.
- Both TUI cold-resume sites now sanitize the model-fed history while leaving
the display transcript untouched, so the user still sees their full history.
Verified E2E against a real SessionDB: dangling and interrupted tails are
stripped from the model feed, healthy mid-progress tool sequences are
preserved, and the display transcript is always the full raw history.
Follow-up to the cherry-picked #29212 (#29177):
- Promote the 24h stale-process threshold to config.yaml
(session_reset.bg_process_max_age_hours) instead of a hardcoded
constant. 0 disables the cutoff (legacy: any live process blocks reset).
Wired through GatewayConfig.default_reset_policy in gateway/run.py.
- Bug 2: process(action=list) now resolves the gateway session_key from
the contextvar and surfaces session-scoped background processes (a
forgotten preview server under a different task), flagged
session_scoped — so the agent/user can discover and kill the blocker.
Previously the task-scoped list returned [] and the blocker was invisible.
- Tests: config round-trip for the new field, cross-task list visibility.
- Docs: messaging session-reset section.
Background processes (e.g. http.server preview) that Hermes starts and
forgets about previously blocked session idle/daily reset indefinitely.
The reset guard in session.py checked has_active_for_session() with no
max age — a 3-day-old preview server blocked reset the same as a task
started 30 seconds ago.
Changes:
- Add max_active_age parameter to has_active_for_session() in
process_registry.py. Processes older than this threshold are ignored.
- Add MAX_ACTIVE_PROCESS_AGE constant (24h / 86400s).
- Wire max_active_age into the gateway's session store callback in
run.py so stale processes no longer block session lifecycle.
- Add debug logging when reset is skipped due to active processes.
- Add 3 tests covering recent, stale, and legacy (None) max age.
Fixes#29177
A document attached alongside an image in the same Discord message was
swept into the vision pipeline and 400'd the whole turn ("Could not
process image"), and was simultaneously never surfaced to the agent as a
readable file. Restores the "any file type works" contract for mixed
messages and fixes the HTTP 400.
Bug 1 — mixed attachments: the inbound routing loop keyed image/audio/video
classification off the message-level type (PHOTO/VOICE/AUDIO), so a doc in
a PHOTO message landed in image_paths and poisoned the vision call. The
document context-note path was gated on message_type == DOCUMENT, so that
same doc never reached the agent at all. Now classification is
per-attachment (trust each attachment's own MIME; fall back to the
message-level type only when MIME is unknown), via shared _event_media_is_*
helpers used by both _build_media_placeholder and the main inbound loop.
The document note now fires for any non-image/audio/video attachment
regardless of message-level type.
Bug 2 — uncommon formats: AVIF/HEIC/BMP/TIFF/ICO produced the same generic
400 because providers only accept PNG/JPEG/GIF/WEBP. image_routing now
transcodes those to PNG via Pillow before declaring media_type, skipping
cleanly (logged) if Pillow/plugins are missing. SVG is vector — Pillow
can't rasterize it — so it's skipped rather than transcoded.
Closes#25935.
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: cypres0099 <74935762+cypres0099@users.noreply.github.com>
The gateway banner promises 'chat responses are scrubbed before delivery',
but _redact_gateway_user_facing_secrets used a divergent 6-pattern subset that
leaked credential shapes the comprehensive agent.redact catches — notably the
GitHub fine-grained PAT (github_pat_...) and the Telegram bot-token shape
(bot<digits>:<token>), the gateway's own credential type.
_redact_gateway_user_facing_secrets now delegates to
agent.redact.redact_sensitive_text(force=True) — the same Tirith-grade redactor
already applied to logs, tool output, and approval-command prompts — so the
outbound LLM-response path (final_response -> _sanitize_gateway_final_response)
masks the full credential set. The narrow local pattern set is kept as a
fail-soft second pass. force=True honors redaction even when
security.redact_secrets is off, matching _redact_approval_command.
Test: regression guard parametrizing all 5 issue shapes x every chat surface;
asserts secret body never reaches the user and surrounding prose survives. The
existing bearer-token test's marker assertion is loosened from the literal
'[REDACTED]' to mask-agnostic (the redactor masks as '***'/partial) — it
asserts the security invariant, not the implementation's mask string.
The main stop loop in _stop_impl() awaited adapter.cancel_background_tasks()
and adapter.disconnect() with no timeout, for both the primary and the
secondary-profile (multiplex) adapter maps. A half-dead platform — a wedged
Feishu/Lark WebSocket thread blocked on network I/O is the reported case —
makes one of those awaits block forever, so the process never exits. systemd
then SIGKILLs it after TimeoutStopSec, skipping atexit PID-file cleanup, and
the next start dies with 'PID file race lost' and enters a restart loop.
The per-adapter timeout infra already existed on main
(_adapter_disconnect_timeout_secs / HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT,
default 5s) but was only wired into _safe_adapter_disconnect, which the
teardown path never calls.
Add _bounded_adapter_teardown(): wraps BOTH cancel_background_tasks() and
disconnect() in the existing timeout budget, logs and forces forward progress
on timeout, and never raises. Both teardown loops now route through it, so the
stop sequence always completes regardless of any adapter's internal behavior
and PID-file cleanup runs.
Original report + fix direction by @happy5318 (#14128, #14130); this widens it
to cover cancel_background_tasks(), the multiplex loop, and the config knob.
Co-authored-by: happy5318 <happy5318@users.noreply.github.com>
display.thinking_progress is documented as independent of tool_progress —
users can keep tool progress quiet while opting into mid-turn assistant
scratch-text bubbles. But two gates were keyed on tool_progress_enabled alone,
so with tool_progress:off the _thinking relay was silently dead even when
thinking_progress:true:
1. agent.tool_progress_callback was set to None unless tool_progress_enabled,
so the callback that queues _thinking text never fired.
2. The send_progress_messages drain task was only started when
tool_progress_enabled, so even queued messages had no consumer.
Both now gate on needs_progress_queue (tool_progress OR thinking_progress) —
the same condition that already decides whether to create the progress queue
at all. No effect when both are off (queue is None) or when tool_progress is
on (unchanged).
Tests: _thinking relays with thinking_progress:on/tool_progress:off, and is
suppressed when thinking_progress:off. Full progress-topics suite: 35 pass.
* fix(windows): stop terminal-window popups from background spawns
Native-Windows desktop/gateway users saw cmd/conhost windows flash on
gateway restart, image paste, the dashboard Projects tree, voice notes,
and ~5 min after closing the app (detached cron). Two root causes:
- Console-subsystem exes (taskkill, schtasks, wmic, netstat, tasklist,
agent-browser, git, ffmpeg, powershell, git-bash) spawned via raw
subprocess allocate a fresh console when the launching process has
none (pythonw desktop backend / detached gateway) - even with output
captured.
- uv venv pythonw shims re-exec console python.exe, so Python children
get a console regardless of how they're launched.
Fixes:
- Single hidden-spawn primitive (_subprocess_compat.run/.popen) that ORs
CREATE_NO_WINDOW on Windows, no-op on POSIX. Route every Hermes-owned
console-exe spawn through it.
- FreeConsole() catch-all in hermes_bootstrap: any Python child that
exclusively owns an auto-allocated console detaches it at startup
(GetConsoleProcessList()==1 gate leaves shared interactive consoles
untouched).
- Replace PowerShell/wmic gateway PID scans with in-process psutil.
- Skip schtasks queries on non-interactive desktop restarts.
- Prefer native agent-browser .exe over .cmd shims.
- Guard test bans raw subprocess spawns of the Windows-only console
tools repo-wide so the popup class can't regress.
* fix(windows): scope FreeConsole to background entry points; fix merge fallout
Console detach review (per #53810 feedback): GetConsoleProcessList()==1 can't
tell a uv pythonw->python phantom console apart from a user opening the
interactive CLI/TUI in its own fresh console (double-click, shortcut, ConPTY) —
both report a single attached process with a tty. Running FreeConsole() in the
import-time bootstrap therefore risked detaching a legitimately-interactive
terminal.
- Extract FreeConsole into explicit hermes_bootstrap.detach_orphan_console();
remove it from apply_windows_utf8_bootstrap() (import side effect).
- Call it only from known background mains: gateway run, dashboard backend
(start_server, what the desktop spawns), cron standalone, tui_gateway entry,
slash worker. Interactive CLI/TUI never calls it.
- Behavior-contract tests: frees only when solo owner, leaves shared console,
no-op without console / on POSIX, and asserts it's not an import side effect.
Merge fallout from origin/main (#53791):
- local.py: 3-way merge left a dangling **_popen_kwargs (NameError crashing
every terminal init). _subprocess_compat.popen already hides the window, so
drop it.
- discord adapter: merge stacked an undefined windows_hide_flags() onto the
primitive call; drop the redundant arg.
- test_gateway: scan now goes psutil-first (zero spawn); rewrite the
case-variant test to drive that production path.
* test(claw): mock _subprocess_compat.run seam for Windows process scan
claw.py's Windows tasklist/powershell scan routes through the hidden-spawn
primitive; the tests still patched claw_mod.subprocess, so on win32 the mock
was never hit and real spawns returned nothing. Patch the actual seam.
Add an explicit _closing guard to both owned executors so the
recreate-on-shutdown path only recovers from an *external* teardown of
the loop default — never resurrects a pool the gateway/adapter itself
stopped. _shutdown_*executor() sets the flag; _get_*executor() raises if
closing; feishu connect() re-arms on reconnect. Updates the gateway
recreate test to assert the refusal contract and adds feishu coverage.
