The shutdown banner promised "send any message after restart to resume
where you left off" but the code did the opposite: a drain-timeout
restart skipped the .clean_shutdown marker, which made the next startup
call suspend_recently_active(), which marked the session suspended,
which made get_or_create_session() spawn a fresh session_id with a
'Session automatically reset. Use /resume...' notice — contradicting
the banner.
Introduce a resume_pending state on SessionEntry that is distinct from
suspended. Drain-timeout shutdown flags active sessions resume_pending
instead of letting startup-wide suspension destroy them. The next
message on the same session_key preserves the session_id, reloads the
transcript, and the agent receives a reason-aware restart-resume
system note that subsumes the existing tool-tail auto-continue note
(PR #9934).
Terminal escalation still flows through the existing
.restart_failure_counts stuck-loop counter (PR #7536, threshold 3) —
no parallel counter on SessionEntry. suspended still wins over
resume_pending in get_or_create_session() so genuinely stuck sessions
converge to a clean slate.
Spec: PR #11852 (BrennerSpear). Implementation follows the spec with
the approved correction (reuse .restart_failure_counts rather than
adding a resume_attempts field).
Changes:
- gateway/session.py: SessionEntry.resume_pending/resume_reason/
last_resume_marked_at + to_dict/from_dict; SessionStore
.mark_resume_pending()/clear_resume_pending(); get_or_create_session()
returns existing entry when resume_pending (suspended still wins);
suspend_recently_active() skips resume_pending entries.
- gateway/run.py: _stop_impl() drain-timeout branch marks active
sessions resume_pending before _interrupt_running_agents();
_run_agent() injects reason-aware restart-resume system note that
subsumes the tool-tail case; successful-turn cleanup also clears
resume_pending next to _clear_restart_failure_count();
_notify_active_sessions_of_shutdown() softens the restart banner to
'I'll try to resume where you left off' (honest about stuck-loop
escalation).
- tests/gateway/test_restart_resume_pending.py: 29 new tests covering
SessionEntry roundtrip, mark/clear helpers, get_or_create_session
precedence (suspended > resume_pending), suspend_recently_active
skip, drain-timeout mark reason (restart vs shutdown), system-note
injection decision tree (including tool-tail subsumption), banner
wording, and stuck-loop escalation override.
Based on #11984 by @maxchernin. Fixes#8259.
Some providers (MiniMax M2.7 via NVIDIA NIM) resend the full function
name in every streaming chunk instead of only the first. The old
accumulator used += which concatenated them into 'read_fileread_file'.
Changed to simple assignment (=), matching the OpenAI Node SDK, LiteLLM,
and Vercel AI SDK patterns. Function names are atomic identifiers
delivered complete — no provider splits them across chunks, so
concatenation was never correct semantics.
Pass 3 of `_prune_old_tool_results` previously shrunk long `function.arguments`
blobs by slicing the raw JSON string at byte 200 and appending the literal
text `...[truncated]`. That routinely produced payloads like::
{"path": "/foo.md", "content": "# Long markdown
...[truncated]
— an unterminated string with no closing brace. Strict providers (observed
on MiniMax) reject this as `invalid function arguments json string` with a
non-retryable 400. Because the broken call survives in the session history,
every subsequent turn re-sends the same malformed payload and gets the same
400, locking the session into a re-send loop until the call falls out of
the window.
Fix: parse the arguments first, shrink long string leaves inside the parsed
structure, and re-serialise. Non-string values (paths, ints, booleans, lists)
pass through intact. Arguments that are not valid JSON to begin with (rare,
some backends use non-JSON tool args) are returned unchanged rather than
replaced with something neither we nor the provider can parse.
Observed in the wild: a `write_file` with ~800 chars of markdown `content`
triggered this on a real session against MiniMax-M2.7; every turn after
compression got rejected until the session was manually reset.
Tests:
- 7 direct tests of `_truncate_tool_call_args_json` covering valid-JSON
output, non-JSON pass-through, nested structures, non-string leaves,
scalar JSON, and Unicode preservation
- 1 end-to-end test through `_prune_old_tool_results` Pass 3 that
reproduces the exact failure payload shape from the incident
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(kimi): force fixed temperature on kimi-k2.* models (k2.5, thinking, turbo)
The prior override only matched the literal model name "kimi-for-coding",
but Moonshot's coding endpoint is hit with real model IDs such as
`kimi-k2.5`, `kimi-k2-turbo-preview`, `kimi-k2-thinking`, etc. Those
requests bypassed the override and kept the caller's temperature, so
Moonshot returns HTTP 400 "invalid temperature: only 0.6 is allowed for
this model" (or 1.0 for thinking variants).
Match the whole kimi-k2.* family:
* kimi-k2-thinking / kimi-k2-thinking-turbo -> 1.0 (thinking mode)
* all other kimi-k2.* -> 0.6 (non-thinking / instant mode)
Also accept an optional vendor prefix (e.g. `moonshotai/kimi-k2.5`) so
aggregator routings are covered.
* refactor(kimi): whitelist-match kimi coding models instead of prefix
Addresses review feedback on PR #12144.
- Replace `startswith("kimi-k2")` with explicit frozensets sourced from
Moonshot's kimi-for-coding model list. The prefix match would have also
clamped `kimi-k2-instruct` / `kimi-k2-instruct-0905`, which are the
separate non-coding K2 family with variable temperature (recommended 0.6
but not enforced — see huggingface.co/moonshotai/Kimi-K2-Instruct).
- Confirmed via platform.kimi.ai docs that all five coding models
(k2.5, k2-turbo-preview, k2-0905-preview, k2-thinking, k2-thinking-turbo)
share the fixed-temperature lock, so the preview-model mapping is no
longer an assumption.
- Drop the fragile `"thinking" in bare` substring test for a set lookup.
- Log a debug line on each override so operators can see when Hermes
silently rewrites temperature.
- Update class docstring. Extend the negative test to parametrize over
kimi-k2-instruct, Kimi-K2-Instruct-0905, and a hypothetical future
kimi-k2-experimental name — all must keep the caller's temperature.
- /retry: use session['history'] instead of non-existent
agent.conversation_history; truncate history at last user message
to match CLI retry_last() behavior; add history_lock safety
- /plan: pass user instruction (arg) to build_plan_path instead of
session_key; add runtime_note so agent knows where to save the plan
- ANSI tool results: render full text via <Ansi wrap=truncate-end>
instead of slicing raw ANSI through compactPreview (which cuts
mid-escape-sequence producing garbled output)
- Move _PENDING_INPUT_COMMANDS frozenset to module level
- Use get_skill_commands() (cached) instead of scan_skill_commands()
(rescans disk) in slash.exec skill interception
- Add 3 retry tests: happy path with history truncation verification,
empty history error, multipart content extraction
- Update test mock target from scan_skill_commands to get_skill_commands
Additional TUI fixes discovered in the same audit:
1. /plan slash command was silently lost — process_command() queues the
plan skill invocation onto _pending_input which nobody reads in the
slash worker subprocess. Now intercepted in slash.exec and routed
through command.dispatch with a new 'send' dispatch type.
Same interception added for /retry, /queue, /steer as safety nets
(these already have correct TUI-local handlers in core.ts, but the
server-side guard prevents regressions if the local handler is
bypassed).
2. Tool results were stripping ANSI escape codes — the messageLine
component used stripAnsi() + plain <Text> for tool role messages,
losing all color/styling from terminal, search_files, etc. Now
uses <Ansi> component (already imported) when ANSI is detected.
3. Terminal tab title now shows model + busy status via useTerminalTitle
hook from @hermes/ink (was never used). Users can identify Hermes
tabs and see at a glance whether the agent is busy or ready.
4. Added 'send' variant to CommandDispatchResponse type + asCommandDispatch
parser + createSlashHandler handler for commands that need to inject
a message into the conversation (plan, queue fallback, steer fallback).
Two TUI fixes:
1. Hyperlinks are now clickable (Cmd+Click / Ctrl+Click) in terminals
that support OSC 8. The markdown renderer was rendering links as
plain colored text — now wraps them in the existing <Link> component
from @hermes/ink which emits OSC 8 escape sequences.
2. Skill slash commands (e.g. /hermes-agent-dev) now work in the TUI.
The slash.exec handler was delegating to the _SlashWorker subprocess
which calls cli.process_command(). For skills, process_command()
queues the invocation message onto _pending_input — a Queue that
nobody reads in the worker subprocess. The skill message was lost.
Now slash.exec detects skill commands early and rejects them so
the TUI falls through to command.dispatch, which correctly builds
and returns the skill payload for the client to send().
* feat(steer): /steer <prompt> injects a mid-run note after the next tool call
Adds a new slash command that sits between /queue (turn boundary) and
interrupt. /steer <text> stashes the message on the running agent and
the agent loop appends it to the LAST tool result's content once the
current tool batch finishes. The model sees it as part of the tool
output on its next iteration.
No interrupt is fired, no new user turn is inserted, and no prompt
cache invalidation happens beyond the normal per-turn tool-result
churn. Message-role alternation is preserved — we only modify an
existing role:"tool" message's content.
Wiring
------
- hermes_cli/commands.py: register /steer + add to ACTIVE_SESSION_BYPASS_COMMANDS.
- run_agent.py: add _pending_steer state, AIAgent.steer(), _drain_pending_steer(),
_apply_pending_steer_to_tool_results(); drain at end of both parallel and
sequential tool executors; clear on interrupt; return leftover as
result['pending_steer'] if the agent exits before another tool batch.
- cli.py: /steer handler — route to agent.steer() when running, fall back to
the regular queue otherwise; deliver result['pending_steer'] as next turn.
- gateway/run.py: running-agent intercept calls running_agent.steer(); idle-agent
path strips the prefix and forwards as a regular user message.
- tui_gateway/server.py: new session.steer JSON-RPC method.
- ui-tui: SessionSteerResponse type + local /steer slash command that calls
session.steer when ui.busy, otherwise enqueues for the next turn.
Fallbacks
---------
- Agent exits mid-steer → surfaces in run_conversation result as pending_steer
so CLI/gateway deliver it as the next user turn instead of silently dropping it.
- All tools skipped after interrupt → re-stashes pending_steer for the caller.
- No active agent → /steer reduces to sending the text as a normal message.
Tests
-----
- tests/run_agent/test_steer.py — accept/reject, concatenation, drain,
last-tool-result injection, multimodal list content, thread safety,
cleared-on-interrupt, registry membership, bypass-set membership.
- tests/gateway/test_steer_command.py — running agent, pending sentinel,
missing steer() method, rejected payload, empty payload.
- tests/gateway/test_command_bypass_active_session.py — /steer bypasses
the Level-1 base adapter guard.
- tests/test_tui_gateway_server.py — session.steer RPC paths.
72/72 targeted tests pass under scripts/run_tests.sh.
* feat(steer): register /steer in Discord's native slash tree
Discord's app_commands tree is a curated subset of slash commands (not
derived from COMMAND_REGISTRY like Telegram/Slack). /steer already
works there as plain text (routes through handle_message → base
adapter bypass → runner), but registering it here adds Discord's
native autocomplete + argument hint UI so users can discover and
type it like any other first-class command.
base.py's _keep_typing refresh loop calls send_typing every ~2s while
the agent is processing. If signal-cli returns NETWORK_FAILURE for the
recipient (offline, unroutable, group membership lost), the unmitigated
path was a WARNING log every 2 seconds for as long as the agent stayed
busy — a user report showed 1048 warnings in 41 minutes for one
offline contact, plus the matching volume of pointless RPC traffic to
signal-cli.
- _rpc() accepts log_failures=False so callers can route repeated
expected failures (typing) to DEBUG while keeping send/receive at
WARNING.
- send_typing() tracks consecutive failures per chat. First failure
still logs WARNING so transport issues remain visible; subsequent
failures log at DEBUG. After three consecutive failures we skip the
RPC during an exponential cooldown (16s, 32s, 60s cap) so we stop
hammering signal-cli for a recipient it can't deliver to. A
successful sendTyping resets the counters.
- _stop_typing_indicator() clears the backoff state so the next agent
turn starts fresh.
E2E simulation against the reported 41-minute window: RPCs drop from
1230 to 45 (-96%), log lines from 1048 WARNINGs to 1 WARNING + 44
DEBUGs.
Credits kshitijk4poor (#12056) for the _rpc log_failures kwarg idea;
the broader restructure in that PR (nested per-chat loop inside
send_typing) is avoided here in favour of stateful backoff that
preserves base.py's existing _keep_typing architecture.
Twelve tests under TestCJKSearchFallback guarding:
- CJK detection across Chinese/Japanese/Korean/Hiragana/Katakana ranges
(including the full Hangul syllables block \uac00-\ud7af, to catch
the shorter-range typo from one of the duplicate PRs)
- Substring match for multi-char Chinese, Japanese, Korean queries
- Filter preservation (source_filter, exclude_sources, role_filter)
in the LIKE path — guards against the SQL-builder bug from another
duplicate PR where filter clauses landed after LIMIT/OFFSET
- Snippet centered on the matched term (instr-based substr window),
not the leading 200 chars of content
- English fast-path untouched
- Empty/no-match cases
- Mixed CJK+English queries
Also:
- hermes_state.py: LIKE-fallback snippet is now
`substr(content, max(1, instr(content, ?) - 40), 120)`, centered on
the match instead of the whole-content default. Credit goes to
@iamagenius00 for the snippet idea in PR #11517.
- scripts/release.py: add @iamagenius00 to AUTHOR_MAP so future
release attribution resolves cleanly.
Refs #11511, #11516, #11517, #11541.
Co-authored-by: iamagenius00 <iamagenius00@users.noreply.github.com>
When streaming died after text was already delivered to the user but
before a tool-call's arguments finished streaming, the partial-stream
stub at the end of _interruptible_streaming_api_call silently set
`tool_calls=None` on the returned message and kept `finish_reason=stop`.
The agent treated the turn as complete, the session exited cleanly with
code 0, and the attempted action was lost with zero user-facing signal.
Live-observed Apr 2026 with MiniMax M2.7 on a ~6-minute audit task:
agent streamed 'Let me write the audit:', started emitting a write_file
tool call, MiniMax stalled for 240s mid-arguments, the stale-stream
detector killed the connection, the stub fired, session ended, no file
written, no error shown.
Fix: the streaming accumulator now records each tool-call's name into
`result['partial_tool_names']` as soon as the name is known. When the
stub builder fires after a partial delivery and finds any recorded tool
names, it appends a human-visible warning to the stub's content — and
also fires it as a live stream delta so the user sees it immediately,
not only in the persisted transcript. The next turn's model also sees
the warning in conversation history and can retry on its own. Text-only
partial streams keep the original bare-recovery behaviour (no warning).
Validation:
| Scenario | Before | After |
|---------------------------------------------|---------------------------|---------------------------------------------|
| Stream dies mid tool-call, text already sent | Silent exit, no indication | User sees ⚠ warning naming the dropped tool |
| Text-only partial stream | Bare recovered text | Unchanged |
| tests/run_agent/test_streaming.py | 24 passed | 26 passed (2 new) |
Weaker models (Gemma-class) repeatedly rediscover and forget that
execute_code uses a different CWD and Python interpreter than terminal(),
causing them to flip-flop on whether user files exist and to hit import
errors on project dependencies like pandas.
Adds a new 'code_execution.mode' config key (default 'project') that
brings execute_code into line with terminal()'s filesystem/interpreter:
project (new default):
- cwd = session's TERMINAL_CWD (falls back to os.getcwd())
- python = active VIRTUAL_ENV/bin/python or CONDA_PREFIX/bin/python
with a Python 3.8+ version check; falls back cleanly to
sys.executable if no venv or the candidate fails
- result : 'import pandas' works, '.env' resolves, matches terminal()
strict (opt-in):
- cwd = staging tmpdir (today's behavior)
- python = sys.executable (today's behavior)
- result : maximum reproducibility and isolation; project deps
won't resolve
Security-critical invariants are identical across both modes and covered by
explicit regression tests:
- env scrubbing (strips *_API_KEY, *_TOKEN, *_SECRET, *_PASSWORD,
*_CREDENTIAL, *_PASSWD, *_AUTH substrings)
- SANDBOX_ALLOWED_TOOLS whitelist (no execute_code recursion, no
delegate_task, no MCP from inside scripts)
- resource caps (5-min timeout, 50KB stdout, 50 tool calls)
Deliberately avoids 'sandbox'/'isolated'/'cloud' language in tool
descriptions (regression from commit 39b83f34 where agents on local
backends falsely believed they were sandboxed and refused networking).
Override via env var: HERMES_EXECUTE_CODE_MODE=strict|project
Seven test files were asserting against older function signatures and
behaviors. CI has been red on main because of accumulated test debt
from other PRs; this catches the tests up.
- tests/agent/test_subagent_progress.py: _build_child_progress_callback
now takes (task_index, goal, parent_agent, task_count=1); update all
call sites and rewrite tests that assumed the old 'batch-only' relay
semantics (now relays per-tool AND flushes a summary at BATCH_SIZE).
Renamed test_thinking_not_relayed_to_gateway → test_thinking_relayed_to_gateway
since thinking IS now relayed as subagent.thinking.
- tests/tools/test_delegate.py: _build_child_agent now requires
task_count; add task_count=1 to all 8 call sites.
- tests/cli/test_reasoning_command.py: AIAgent gained _stream_callback;
stub it on the two test agent helpers that use spec=AIAgent / __new__.
- tests/hermes_cli/test_cmd_update.py: cmd_update now runs npm install
in repo root + ui-tui/ + web/ and 'npm run build' in web/; assert
all four subprocess calls in the expected order.
- tests/hermes_cli/test_model_validation.py: dissimilar unknown models
now return accepted=False (previously True with warning); update
both affected tests.
- tests/tools/test_registry.py: include feishu_doc_tool and
feishu_drive_tool in the expected builtin tool set.
- tests/gateway/test_voice_command.py: missing-voice-deps message now
suggests 'pip install PyNaCl' not 'hermes-agent[messaging]'.
411/411 pass locally across these 7 files.
hermes update no longer dies when the controlling terminal closes
(SSH drop, shell close) during pip install. SIGHUP is set to SIG_IGN
for the duration of the update, and stdout/stderr are wrapped so writes
to a closed pipe are absorbed instead of cascading into process exit.
All update output is mirrored to ~/.hermes/logs/update.log so users can
see what happened after reconnecting.
SIGINT (Ctrl-C) and SIGTERM (systemd) are intentionally still honored —
those are deliberate cancellations, not accidents. In gateway mode the
helper is a no-op since the update is already detached.
POSIX preserves SIG_IGN across exec(), so pip and git subprocesses
inherit hangup protection automatically — no changes to subprocess
spawning needed.
When a Telegram /restart fires and PTB's graceful-shutdown `get_updates`
ACK call times out ("When polling for updates is restarted, updates may
be received twice" in gateway.log), the new gateway receives the same
/restart again and restarts a second time — a self-perpetuating loop.
Record the triggering update_id in `.restart_last_processed.json` when
handling /restart. On the next process, reject a /restart whose
update_id <= the recorded one as a stale redelivery. 5-minute staleness
guard so an orphaned marker can't block a legitimately new /restart.
- gateway/platforms/base.py: add `platform_update_id` to MessageEvent
- gateway/platforms/telegram.py: propagate `update.update_id` through
_build_message_event for text/command/location/media handlers
- gateway/run.py: write dedup marker in _handle_restart_command;
_is_stale_restart_redelivery checks it before processing /restart
- tests/gateway/test_restart_redelivery_dedup.py: 9 new tests covering
fresh restart, redelivery, staleness window, cross-platform,
malformed-marker resilience, and no-update_id (CLI) bypass
Only active for Telegram today (the one platform with monotonic
cross-session update ordering); other platforms return False from
_is_stale_restart_redelivery and proceed normally.
* fix(interrupt): propagate to concurrent-tool workers + opt-in debug trace
interrupt() previously only flagged the agent's _execution_thread_id.
Tools running inside _execute_tool_calls_concurrent execute on
ThreadPoolExecutor worker threads whose tids are distinct from the
agent's, so is_interrupted() inside those tools returned False no matter
how many times the gateway called .interrupt() — hung ssh / curl / long
make-builds ran to their own timeout.
Changes:
- run_agent.py: track concurrent-tool worker tids in a per-agent set,
fan interrupt()/clear_interrupt() out to them, and handle the
register-after-interrupt race at _run_tool entry. getattr fallback
for the tracker so test stubs built via object.__new__ keep working.
- tools/environments/base.py: opt-in _wait_for_process trace (ENTER,
per-30s HEARTBEAT with interrupt+activity-cb state, INTERRUPT
DETECTED, TIMEOUT, EXIT) behind HERMES_DEBUG_INTERRUPT=1.
- tools/interrupt.py: opt-in set_interrupt() trace (caller tid, target
tid, set snapshot) behind the same env flag.
- tests: new regression test runs a polling tool on a concurrent worker
and asserts is_interrupted() flips to True within ~1s of interrupt().
Second new test guards clear_interrupt() clearing tracked worker bits.
Validation: tests/run_agent/ all 762 pass; tests/tools/ interrupt+env
subset 216 pass.
* fix(interrupt-debug): bypass quiet_mode logger filter so trace reaches agent.log
AIAgent.__init__ sets logging.getLogger('tools').setLevel(ERROR) when
quiet_mode=True (the CLI default). This would silently swallow every
INFO-level trace line from the HERMES_DEBUG_INTERRUPT=1 instrumentation
added in the parent commit — confirmed by running hermes chat -q with
the flag and finding zero trace lines in agent.log even though
_wait_for_process was clearly executing (subprocess pid existed).
Fix: when HERMES_DEBUG_INTERRUPT=1, each traced module explicitly sets
its own logger level to INFO at import time, overriding the 'tools'
parent-level filter. Scoped to the opt-in case only, so production
(quiet_mode default) logs stay quiet as designed.
Validation: hermes chat -q with HERMES_DEBUG_INTERRUPT=1 now writes
'_wait_for_process ENTER/EXIT' lines to agent.log as expected.
* fix(cli): SIGTERM/SIGHUP no longer orphans tool subprocesses
Tool subprocesses spawned by the local environment backend use
os.setsid so they run in their own process group. Before this fix,
SIGTERM/SIGHUP to the hermes CLI killed the main thread via
KeyboardInterrupt but the worker thread running _wait_for_process
never got a chance to call _kill_process — Python exited, the child
was reparented to init (PPID=1), and the subprocess ran to its
natural end (confirmed live: sleep 300 survived 4+ min after SIGTERM
to the agent until manual cleanup).
Changes:
- cli.py _signal_handler (interactive) + _signal_handler_q (-q mode):
route SIGTERM/SIGHUP through agent.interrupt() so the worker's poll
loop sees the per-thread interrupt flag and calls _kill_process
(os.killpg) on the subprocess group. HERMES_SIGTERM_GRACE (default
1.5s) gives the worker time to complete its SIGTERM+SIGKILL
escalation before KeyboardInterrupt unwinds main.
- tools/environments/base.py _wait_for_process: wrap the poll loop in
try/except (KeyboardInterrupt, SystemExit) so the cleanup fires
even on paths the signal handlers don't cover (direct sys.exit,
unhandled KI from nested code, etc.). Emits EXCEPTION_EXIT trace
line when HERMES_DEBUG_INTERRUPT=1.
- New regression test: injects KeyboardInterrupt into a running
_wait_for_process via PyThreadState_SetAsyncExc, verifies the
subprocess process group is dead within 3s of the exception and
that KeyboardInterrupt re-raises cleanly afterward.
Validation:
| Before | After |
|---------------------------------------------------------|--------------------|
| sleep 300 survives 4+ min as PPID=1 orphan after SIGTERM | dies within 2 s |
| No INTERRUPT DETECTED in trace | INTERRUPT DETECTED fires + killing process group |
| tests/tools/test_local_interrupt_cleanup | 1/1 pass |
| tests/run_agent/test_concurrent_interrupt | 4/4 pass |
Extend forum support from PR #10145:
- REST path (_send_discord): forum thread creation now uploads media
files as multipart attachments on the starter message in a single
call. Previously media files were silently dropped on the forum
path.
- Websocket media paths (_send_file_attachment, send_voice, send_image,
send_animation — covers send_image_file, send_video, send_document
transitively): forum channels now go through a new _forum_post_file
helper that creates a thread with the file as starter content,
instead of failing via channel.send(file=...) which forums reject.
- _send_to_forum chunk follow-up failures are collected into
raw_response['warnings'] so partial-send outcomes surface.
- Process-local probe cache (_DISCORD_CHANNEL_TYPE_PROBE_CACHE) avoids
GET /channels/{id} on every uncached send after the first.
- Dedup of TestSendDiscordMedia that the PR merge-resolution left
behind.
- Docs: Forum Channels section under website/docs/user-guide/messaging/discord.md.
Tests: 117 passed (22 new for forum+media, probe cache, warnings).
Follow-up to #11909: surface the legacy-unit warning where users are most
likely to see it. After a 'hermes update', if a pre-rename hermes.service
is still installed alongside the current hermes-gateway.service, print
the list of legacy units + the 'hermes gateway migrate-legacy' command.
Profile-safe: reuses _find_legacy_hermes_units() which is an explicit
allowlist of hermes.service only — profile units never match.
Platform-gated: only prints on systemd hosts (the rename is Linux-only).
Non-blocking: just prints, never prompts, so gateway-spawned
hermes update --gateway runs aren't affected.
* fix(gateway): detect legacy hermes.service units from pre-rename installs
Older Hermes installs used a different service name (hermes.service) before
the rename to hermes-gateway.service. When both units remain installed, they
fight over the same bot token — after PR #5646's signal-recovery change,
this manifests as a 30-second SIGTERM flap loop between the two services.
Detection is an explicit allowlist (no globbing) plus an ExecStart content
check, so profile units (hermes-gateway-<profile>.service) and unrelated
third-party services named 'hermes' are never matched.
Wired into systemd_install, systemd_status, gateway_setup wizard, and the
main hermes setup flow — anywhere we already warn about scope conflicts now
also warns about legacy units.
* feat(gateway): add migrate-legacy command + install-time removal prompt
- New hermes_cli.gateway.remove_legacy_hermes_units() removes legacy
unit files with stop → disable → unlink → daemon-reload. Handles user
and system scopes separately; system scope returns path list when not
running as root so the caller can tell the user to re-run with sudo.
- New 'hermes gateway migrate-legacy' subcommand (with --dry-run and -y)
routes to remove_legacy_hermes_units via gateway_command dispatch.
- systemd_install now offers to remove legacy units BEFORE installing
the new hermes-gateway.service, preventing the SIGTERM flap loop that
hits users who still have pre-rename hermes.service around.
Profile units (hermes-gateway-<profile>.service) remain untouched in
all paths — the legacy allowlist is explicit (_LEGACY_SERVICE_NAMES)
and the ExecStart content check further narrows matches.
* fix(gateway): mark --replace SIGTERM as planned so target exits 0
PR #5646 made SIGTERM exit the gateway with code 1 so systemd's
Restart=on-failure revives it after unexpected kills. But when a user has
two gateway units fighting for the same bot token (e.g. legacy
hermes.service + hermes-gateway.service from a pre-rename install), the
--replace takeover itself becomes the 'unexpected' SIGTERM — the loser
exits 1, systemd revives it 30s later, and the cycle flaps indefinitely.
Before calling terminate_pid(), --replace now writes a short-lived marker
file naming the target PID + start_time. The target's shutdown_signal_handler
consumes the marker and, when it names this process, leaves
_signal_initiated_shutdown=False so the final exit code stays 0.
Staleness defences:
- PID + start_time combo prevents PID reuse matching an old marker
- Marker older than 60s is treated as stale and discarded
- Marker is unlinked on first read even if it doesn't match this process
- Replacer clears the marker post-loop + on permission-denied give-up
Cherry-picked from #10985 by pedh, adapted to current main:
* Keeps main's full group-chat gating (require_mention + allowed_users +
free_response_chats + mention_patterns) — PR's simpler subset dropped.
* Keeps main's fire-and-forget process() dispatch + session_webhook
fallback for SDK >= 0.24.
* Picks up PR's REQUIRES_EDIT_FINALIZE capability flag on
BasePlatformAdapter + finalize kwarg on edit_message(), plumbed through
stream_consumer. Default False so Telegram/Slack/Discord/Matrix stay
on the zero-overhead fast path.
* DingTalk AI Card lifecycle: per-chat _message_contexts, two-card flow
(tool-progress + final response) with sibling auto-close driven by
reply_to, idempotent 🤔Thinking → 🥳Done swap, $alibabacloud-dingtalk$
for media URL resolution (replaces raw HTTP that was 403-ing).
* pyproject: dingtalk extra now dingtalk-stream>=0.20,<1 +
alibabacloud-dingtalk>=2.0.0 + qrcode.
Closes#10991
Co-authored-by: pedh
ShellFileOperations captured the terminal env's cwd at __init__ time and
used that stale value for every subsequent _exec() call. When the user
ran `cd` via the terminal tool, `env.cwd` updated but `ops.cwd` did not.
Relative paths passed to patch_replace / read_file / write_file / search
then targeted the ORIGINAL directory instead of the current one.
Observed symptom in agent sessions:
terminal: cd .worktrees/my-branch
patch hermes_cli/main.py <old> <new>
→ returns {"success": true} with a plausible unified diff
→ but `git diff` in the worktree shows nothing
→ the patch landed in the main repo's checkout of main.py instead
The diff looked legitimate because patch_replace computes it from the
IN-MEMORY content vs new_content, not by re-reading the file. The
write itself DID succeed — it just wrote to the wrong directory's copy
of the same-named file.
Fix: _exec() now resolves cwd from live sources in this order:
1. Explicit `cwd` arg (if provided by the caller)
2. Live `self.env.cwd` (tracks `cd` commands run via terminal)
3. Init-time `self.cwd` (fallback when env has no cwd attribute)
Includes a 5-test regression suite covering:
- cd followed by relative read follows live cwd
- the exact reported bug: patch_replace with relative path after cd
- explicit cwd= arg still wins over env.cwd
- env without cwd attribute falls back to init-time cwd
- patch_replace success reflects real file state (safety rail)
Co-authored-by: teknium1 <teknium@nousresearch.com>
persist_nous_credentials() now accepts an optional label kwarg which
gets embedded in providers.nous under the 'label' key.
_seed_from_singletons() prefers the embedded label over the
auto-derived label_from_token() fingerprint when materialising the
pool entry, so re-seeding on every load_pool('nous') preserves the
user's chosen label.
auth_commands.py threads --label through to the helper, restoring
parity with how other OAuth providers (anthropic, codex, google,
qwen) honor the flag.
Tests: 4 new (embed, reseed-survives, no-label fallback, end-to-end
through auth_add_command). All 390 nous/auth/credential_pool tests
pass.
Review feedback on the original commit: the helper wrote a pool entry
with source `manual:device_code` while `_seed_from_singletons()` upserts
with `device_code` (no `manual:` prefix), so the pool grew a duplicate
row on every `load_pool()` after login.
Normalise: the helper now writes `providers.nous` and delegates the pool
write entirely to `_seed_from_singletons()` via a follow-up
`load_pool()` call. The canonical source is `device_code`; the helper
never materialises a parallel `manual:device_code` entry.
- `persist_nous_credentials()` loses its `label` and `source` kwargs —
both are now derived by the seed path from the singleton state.
- CLI and web dashboard call sites simplified accordingly.
- New test `test_persist_nous_credentials_idempotent_no_duplicate_pool_entries`
asserts that two consecutive persists leave exactly one pool row and
no stray `manual:` entries.
- Existing `test_auth_add_nous_oauth_persists_pool_entry` updated to
assert the canonical source and single-entry invariant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`hermes auth add nous --type oauth` only wrote credential_pool.nous,
leaving providers.nous empty. When the Nous agent_key's 24h TTL expired,
run_agent.py's 401-recovery path called resolve_nous_runtime_credentials
(which reads providers.nous), got AuthError "Hermes is not logged into
Nous Portal", caught it as logger.debug (suppressed at INFO level), and
the agent died with "Non-retryable client error" — no signal to the
user that recovery even tried.
Introduce persist_nous_credentials() as the single source of truth for
Nous device-code login persistence. Both auth_commands (CLI) and
web_server (dashboard) now route through it, so pool and providers
stay in sync at write time.
Why: CLI-provisioned profiles couldn't recover from agent_key expiry,
producing silent daily outages 24h after first login. PR #6856/#6869
addressed adjacent issues but assumed providers.nous was populated;
this one wasn't being written.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: aggregator users (OpenRouter / Nous Portal) running 'auto'
routing for auxiliary tasks — compression, vision, web extraction,
session search, etc. — got routed to a cheap provider-side default
model (Gemini Flash). Non-aggregator users already got their main
model. Behavior was inconsistent and surprising — users picked
Claude / GPT / their preferred model, but side tasks ran on
Gemini Flash.
After: 'auto' means "use my main chat model" for every user,
regardless of provider type. Only when the main provider has no
working client does the fallback chain run (OpenRouter → Nous →
custom → Codex → API-key providers). Explicit per-task overrides
in config.yaml (auxiliary.<task>.provider / .model) still win —
they are a hard constraint, not subject to the auto policy.
Vision auto-detection follows the same policy: try main provider +
main model first (with _PROVIDER_VISION_MODELS overrides preserved
for providers like xiaomi and zai that ship a dedicated multimodal
model distinct from their chat model). Aggregator strict vision
backends are fallbacks, not the primary path.
Changes:
- agent/auxiliary_client.py: _resolve_auto() drops the
`_AGGREGATOR_PROVIDERS` guard. resolve_vision_provider_client()
auto branch unifies aggregator and exotic-provider paths —
everyone goes through resolve_provider_client() with main_model.
Dead _AGGREGATOR_PROVIDERS constant removed (was only used by
the guard we just removed).
- hermes_cli/main.py: aux config menu copy updated to reflect
the new semantics ("'auto' means 'use my main model'").
- tests/agent/test_auxiliary_main_first.py: 12 regression tests
covering OpenRouter/Nous/DeepSeek main paths, runtime-override
wins, explicit-config wins, vision override preservation for
exotic providers, and fallback-chain activation when the main
provider has no working client.
Co-authored-by: teknium1 <teknium@nousresearch.com>
Follow-up polish on top of the cherry-picked #11023 commit.
- feishu_comment_rules.py: replace import-time "~/.hermes" expanduser fallback
with get_hermes_home() from hermes_constants (canonical, profile-safe).
- tools/feishu_doc_tool.py, tools/feishu_drive_tool.py: drop the
asyncio.get_event_loop().run_until_complete(asyncio.to_thread(...)) dance.
Tool handlers run synchronously in a worker thread with no running loop, so
the RuntimeError branch was always the one that executed. Calls client.request
directly now. Unused asyncio import removed.
- tests/gateway/test_feishu.py: add register_p2_customized_event to the mock
EventDispatcher builder so the existing adapter test matches the new handler
registration for drive.notice.comment_add_v1.
- scripts/release.py: map liujinkun@bytedance.com -> liujinkun2025 for
contributor attribution on release notes.
- Full comment handler: parse drive.notice.comment_add_v1 events, build
timeline, run agent, deliver reply with chunking support.
- 5 tools: feishu_doc_read, feishu_drive_list_comments,
feishu_drive_list_comment_replies, feishu_drive_reply_comment,
feishu_drive_add_comment.
- 3-tier access control rules (exact doc > wildcard "*" > top-level >
defaults) with per-field fallback. Config via
~/.hermes/feishu_comment_rules.json, mtime-cached hot-reload.
- Self-reply filter using generalized self_open_id (supports future
user-identity subscriptions). Receiver check: only process events
where the bot is the @mentioned target.
- Smart timeline selection, long text chunking, semantic text extraction,
session sharing per document, wiki link resolution.
Change-Id: I31e82fd6355173dbcc400b8934b6d9799e3137b9
Follow-up to the cherry-picked contributor fix:
- Extract `_remember_chat_req_id()` and bound it at DEDUP_MAX_SIZE like
`_reply_req_ids` — the unbounded dict would grow forever on a long-
running gateway with many chats.
- Move the cache write to AFTER the group/DM policy check so we don't
cache req_ids from blocked senders.
- Revert the undocumented `is_group` change: the contributor flipped
`chattype == 'group'` to `bool(chatid)`, which wasn't mentioned in
the PR description and weakens the signal (chattype is the explicit
hint; relying on chatid presence assumes DMs never carry it). Keep
the original check.
- Drop the defensive `getattr(self, '_last_chat_req_ids', {})` reads
at both send sites — the attribute is initialized in __init__.
- Update `test_send_uses_passive_reply_stream_...` → `_markdown_...`
to match the new msgtype, and add a new TestWeComZombieSessionFix
class covering device_id presence in subscribe, per-chat req_id
caching + bounding, blocked-sender cache exclusion, and the group
APP_CMD_RESPONSE fallback path.
Previously users had to hand-edit config.yaml to route individual auxiliary
tasks (vision, compression, web_extract, etc.) to a specific provider+model.
Add a first-class picker reachable from the bottom of the existing `hermes
model` provider list.
Flow:
hermes model
→ Configure auxiliary models...
→ <task picker: 9 tasks, shows current setting inline>
→ <provider picker: authenticated providers + auto + custom>
→ <model picker: curated list + live pricing>
The aux picker does NOT re-run credential/OAuth setup; users authenticate
providers through the normal `hermes model` flow, then route aux tasks to
them here. `list_authenticated_providers()` gates the list to providers
the user has configured.
Also:
- 'Cancel' entry relabeled 'Leave unchanged' (sentinel still 'cancel'
internally, so dispatch logic is unchanged)
- 'Reset all to auto' entry to bulk-clear aux overrides; preserves
user-tuned timeout / download_timeout values
- Adds `title_generation` task to DEFAULT_CONFIG.auxiliary — the task
was called from agent/title_generator.py but was missing from defaults,
so config-backed timeout overrides never worked for it
Co-authored-by: teknium1 <teknium@nousresearch.com>
Both fixes close process leaks observed in production (18+ orphaned
agent-browser node daemons, 15+ orphaned paste.rs sleep interpreters
accumulated over ~3 days, ~2.7 GB RSS).
## agent-browser daemon leak
Previously the orphan reaper (_reap_orphaned_browser_sessions) only ran
from _start_browser_cleanup_thread, which is only invoked on the first
browser tool call in a process. Hermes sessions that never used the
browser never swept orphans, and the cross-process orphan detection
relied on in-process _active_sessions, which doesn't see other hermes
PIDs' sessions (race risk).
- Write <session>.owner_pid alongside the socket dir recording the
hermes PID that owns the daemon (extracted into _write_owner_pid for
direct testability).
- Reaper prefers owner_pid liveness over in-process _active_sessions.
Cross-process safe: concurrent hermes instances won't reap each
other's daemons. Legacy tracked_names fallback kept for daemons
that predate owner_pid.
- atexit handler (_emergency_cleanup_all_sessions) now always runs
the reaper, not just when this process had active sessions —
every clean hermes exit sweeps accumulated orphans.
## paste.rs auto-delete leak
_schedule_auto_delete spawned a detached Python subprocess per call
that slept 6 hours then issued DELETE requests. No dedup, no tracking —
every 'hermes debug share' invocation added ~20 MB of resident Python
interpreters that stuck around until the sleep finished.
- Replaced the spawn with ~/.hermes/pastes/pending.json: records
{url, expire_at} entries.
- _sweep_expired_pastes() synchronously DELETEs past-due entries on
every 'hermes debug' invocation (run_debug() dispatcher).
- Network failures stay in pending.json for up to 24h, then give up
(paste.rs's own retention handles the 'user never runs hermes again'
edge case).
- Zero subprocesses; regression test asserts subprocess/Popen/time.sleep
never appear in the function source (skipping docstrings via AST).
## Validation
| | Before | After |
|------------------------------|---------------|--------------|
| Orphan agent-browser daemons | 18 accumulated| 2 (live) |
| paste.rs sleep interpreters | 15 accumulated| 0 |
| RSS reclaimed | - | ~2.7 GB |
| Targeted tests | - | 2253 pass |
E2E verified: alive-owner daemons NOT reaped; dead-owner daemons
SIGTERM'd and socket dirs cleaned; pending.json sweep deletes expired
entries without spawning subprocesses.
Two accretion-over-time leaks that compound over long CLI / gateway
lifetimes. Both were flagged in the memory-leak audit.
## file_tools._read_tracker
_read_tracker[task_id] holds three sub-containers that grew unbounded:
read_history set of (path, offset, limit) tuples — 1 per unique read
dedup dict of (path, offset, limit) → mtime — same growth pattern
read_timestamps dict of resolved_path → mtime — 1 per unique path
A CLI session uses one stable task_id for its lifetime, so these were
uncapped. A 10k-read session accumulated ~1.5MB of tracker state that
the tool no longer needed (only the most recent reads are relevant for
dedup, consecutive-loop detection, and write/patch external-edit
warnings).
Fix: _cap_read_tracker_data() enforces hard caps on each container
after every add. Defaults: read_history=500, dedup=1000,
read_timestamps=1000. Eviction is insertion-order (Python 3.7+ dict
guarantee) for the dicts; arbitrary for the set (which only feeds
diagnostic summaries).
## process_registry._completion_consumed
Module-level set that recorded every session_id ever polled / waited /
logged. No pruning. Each entry is ~20 bytes, so the absolute leak is
small, but on a gateway processing thousands of background commands
per day the set grows until process exit.
Fix: _prune_if_needed() now discards _completion_consumed entries
alongside the session dict evictions it already performs (both the
TTL-based prune and the LRU-over-cap prune). Adds a final
belt-and-suspenders pass that drops any dangling entries whose
session_id no longer appears in _running or _finished.
Tests: tests/tools/test_accretion_caps.py — 9 cases
* Each container bound respected, oldest evicted
* No-op when under cap (no unnecessary work)
* Handles missing sub-containers without crashing
* Live read_file_tool path enforces caps end-to-end
* _completion_consumed pruned on TTL expiry
* _completion_consumed pruned on LRU eviction
* Dangling entries (no backing session) cleared
Broader suite: 3486 tests/tools + tests/cli pass. The single flake
(test_alias_command_passes_args) reproduces on unchanged main — known
cross-test pollution under suite-order load.
