Main folded slash_worker.close() into _finalize_session (the single
_finalized-guarded chokepoint) while #42143 was open. The rebase
conflicted with the PR's worker-close in _teardown_session. Keep both —
they target the same #38095 leak and _SlashWorker.close() is
idempotent (_closed/poll()-guarded) — so callers reaching
_teardown_session without the real _finalize_session (and the PR's own
tests, which monkeypatch _finalize_session out) still reap the worker.
Same for _shutdown_sessions, now routed through the unified
_close_session_by_id funnel.
Salvaged from #35626 (banditburai) and re-scoped after maintainers landed the
parent-death watchdog (slash_worker.py) and PTY process-group teardown
(pty_bridge.py) directly on main. Those pieces are intentionally NOT included
here — this carries only what is still missing:
- C1 disconnect reap: ws.py's `finally` only re-pointed the dead transport at
stdio. `_close_sessions_for_transport` now reaps `close_on_disconnect`
sessions and schedules the grace-reap for the rest, offloaded via
`asyncio.to_thread` so the blocking worker.close() + DB write never stalls
the uvicorn loop.
- C2 create/close orphan race: `_attach_worker` stores the worker iff
`_sessions.get(sid) is session` under the lock (else closes it), applied at
every spawn site incl. the post-turn `_restart_slash_worker`.
- Single idempotent teardown funnel: session.close, WS disconnect, the
generous-TTL idle reaper, shutdown, and the WS grace-reap all reach
`_close_session_by_id` → `_teardown_session`; `_finalized`/`_closed` flags
make concurrent/double teardown a no-op. `_sessions_lock` upgraded to RLock.
- uvicorn `ws_ping_interval/timeout=20s` so a half-open socket (reverse-proxy
524) becomes a `WebSocketDisconnect` and the C1 path runs.
Plus two review-driven hardening fixes (mine):
- `session.active_list` now skips `_finalized` sessions so the footer
"N sessions" count reflects attachable sessions instead of only ever
growing until restart (#38950). Keys on `_finalized` only, NOT the stdio
sentinel, so a standalone `hermes --tui` session stays visible.
- `_schedule_ws_orphan_reap._reap` pops via `_close_session_by_id`
(under `_sessions_lock`) instead of `_sessions.pop` under the unrelated
`_session_resume_lock` (#39591); the resume_lock now only guards the orphan
re-check against `session.resume`.
- Float env knobs (`HERMES_SLASH_WATCHDOG_*`, `HERMES_TUI_SESSION_TTL_S`)
parse with a fallback helper so a malformed value can't crash the worker at
import.
Fixes#32377Fixes#38950
Addresses #22855
Co-authored-by: banditburai <123342691+banditburai@users.noreply.github.com>
Co-authored-by: kshitijk4poor <82637225+kshitijk4poor@users.noreply.github.com>
resolve_provider() auto-detection only checked OPENROUTER_API_KEY/
OPENAI_API_KEY env vars, never the credential pool. A key added via
`hermes auth add openrouter` (manual pool entry, no env var) was invisible:
the provider failed to resolve or resolved with an empty api_key, so
requests went out with no Authorization header and OpenRouter returned
"HTTP 401: Missing Authentication header" while `hermes auth list` showed
the credential. Closes#42130.
- auth.py: check load_pool("openrouter").has_credentials() after the env check
- dump.py: `debug share` shows 'openrouter set (auth pool)' instead of the
misleading 'not set' when the key lives in the pool
- add regression tests (pool credential auto-detects; empty pool still raises)
Lift the 18 _model_flow_* provider-setup wizard functions out of hermes_cli/main.py
into hermes_cli/model_setup_flows.py. Behavior-neutral; main.py 14050 -> 11479 LOC.
select_provider_and_model (the dispatcher) STAYS in main.py and re-imports the
flows via an explicit 'from hermes_cli.model_setup_flows import (...)' block, so
both its bare-name calls and existing test monkeypatches targeting
hermes_cli.main._model_flow_* keep resolving against main's namespace unchanged.
Imports: 3 neutral deps (argparse, os, subprocess) at the module top; the 14
main.py-internal helpers the flows call (_prompt_api_key, _save_custom_provider,
the reasoning-effort/stepfun/qwen helpers, _run_anthropic_oauth_flow, ...) are
lazy-imported per-flow (from hermes_cli.main import ...) so the new module never
imports main at module scope -> no import cycle.
Repointed one source-inspection change-detector (test_setup_ollama_cloud_force_refresh)
to read the module the ollama-cloud branch moved to.
Validation: 6563/6563 hermes_cli tests pass; live flow-dispatch probe confirms the
lazy main-internal imports resolve at runtime.
Lift the 5 agent-construction/session-resume methods out of HermesCLI into
hermes_cli/cli_agent_setup_mixin.py:CLIAgentSetupMixin. Behavior-neutral; cli.py
14139 -> 13492 LOC.
Methods moved (~647 LOC): _ensure_runtime_credentials, _resolve_turn_agent_config,
_init_agent, _preload_resumed_session, _display_resumed_history. All self.* calls
resolve unchanged via the MRO (HermesCLI(CLIAgentSetupMixin, CLICommandsMixin)).
Import split (same recipe as #41942): 2 neutral deps (sys, _escape) imported at
the mixin module top; 12 cli.py-internal helpers/constants (AIAgent, ChatConsole,
CLI_CONFIG, _cprint, _DIM, _RST, _accent_hex, ...) imported lazily per-method
(from cli import ...) so the mixin never imports cli at module scope -> no cycle.
Repointed one source-inspection change-detector (test_callable_api_key.py) to read
the mixin file where the method now lives.
Two narrow Windows desktop fixes:
1. tools/process_registry.py — PTY stdin writes are now platform-aware.
pywinpty (Windows) expects str; ptyprocess (POSIX) expects bytes.
Previously bytes was unconditionally passed, producing a TypeError on
Windows ("'bytes' object cannot be converted to 'PyString'").
2. tui_gateway/server.py + ws.py — Detached WebSocket sessions now park on
a _DropTransport sink instead of _stdio_transport. In the desktop the
gateway runs in-process and stdout is captured by Electron into
desktop.log, so falling back to stdio leaked raw JSON-RPC frames into
the desktop log after WS disconnects. Orphan-reap semantics are
preserved via _ws_session_is_orphaned.
Verified on a Windows desktop install:
- pywinpty 2.0.15 rejects bytes / accepts str — reproduced exactly
- Focused suite green (write_stdin × 2, write_json_drops_detached_ws_frames,
ws_orphan_reap × 2)
- All 6 CI test shards green, e2e green, nix (ubuntu/macos) green
Salvage commit (21be7ca) fixes the new test referencing an undefined
_ThreadUnsafeStdout — uses the existing _ChunkyStdout helper.
A Responses-API-shaped payload carrying instructions=/input=/store=/
parallel_tool_calls= can reach the native Anthropic messages.stream() /
messages.create() call under a rare api_mode-flip race (e.g. a concurrent
auxiliary vision call mutating a shared agent between the kwargs build and
the stream dispatch). The Anthropic SDK rejects these with a non-retryable
TypeError that kills the whole turn and propagates the entire fallback chain.
Add sanitize_anthropic_kwargs() at both Anthropic dispatch sites: it drops
the Responses-only keys in place and logs a WARNING (with #31673 breadcrumb)
when one is present, so the underlying race stays visible in the wild
instead of being silently papered over.
* fix(plugins): add thread-safe lazy-singleton helpers, fix honcho TOCTOU (#24759)
get_honcho_client() and fal's _load_fal_client() used unlocked
check-then-init: racing threads both ran the expensive build and the
loser's client (open connection) leaked.
Rather than one-off locks, add plugins/plugin_utils.py with two
reusable primitives every plugin author can drop in:
- lazy_singleton: decorator for zero-arg accessors
- SingletonSlot: manual slot for config-keyed accessors (first wins)
Both use double-checked locking; factory runs at most once; failed
builds aren't cached. honcho is the reference consumer; fal's sibling
TOCTOU gets a matching double-checked lock. Plugin dev guide documents
the pattern so future plugins don't reintroduce the race.
Closes#24759
* test(honcho): update reset test for SingletonSlot internals
test_reset_clears_singleton poked the removed _honcho_client module
global directly. Assert through the slot's public peek() surface
instead, matching the #24759 refactor.
Add a best-effort `commits` list (sha/summary/author/at) to the update-check
response for git/pip installs that are behind upstream, so the desktop's
remote update overlay can show what's changed before applying.
Additive and non-breaking: existing consumers (legacy dashboard, tests using
subset assertions) ignore the new field. Leaves the shared check_for_updates()
int contract untouched — commits come from a separate best-effort git call.
Fold the slash-worker subprocess close into _finalize_session itself —
the single _finalized-guarded session-end chokepoint — instead of
relying on each caller (_teardown_session, _shutdown_sessions) to close
it separately. A future code path that finalizes a session directly can
no longer reintroduce the #38095 worker leak.
Idempotent: _SlashWorker.close() is poll()-guarded and _finalize_session
short-circuits on _finalized, so the existing teardown paths are
unaffected. Drops the now-redundant separate close() in
_shutdown_sessions.
Note: the active leak this issue reported was already fixed on main
(WS-orphan reaper #38591, _restart_slash_worker close, atexit shutdown).
This addresses the residual defense-in-depth gap the reporter correctly
identified in their follow-up comment.
Daemon thread polls _is_orphaned (original ppid check + psutil create_time PID-reuse
guard, no PR_SET_PDEATHSIG). On orphan, drains an in-flight command up to a grace
window then os._exit(0). Started before the HermesCLI build to cover the spawn window.
Task: swl-qrf.8
Long-lived gateways under heavy cron/build workloads grow steadily (~18 MB/hr
post-phantom-dispatch-fix) and eventually need a restart-or-OOM. Four retention
sites, all confirmed live on current main:
1. _evict_cached_agent() (/model, /reasoning, codex-runtime, /undo, etc.) popped
the cache entry without releasing the agent's OpenAI client, httpx transport,
SSL context, or conversation history. Only /new cleaned up first. Now releases
clients on a daemon thread, matching _enforce_agent_cache_cap.
2. _release_evicted_agent_soft() now clears _session_messages after
release_clients() — tool outputs (file reads, terminal output, search results)
can be tens of MB per 100+-tool-call session; the list is rebuilt from
persisted session JSON on resume, so dropping it on soft eviction is safe.
3. The session-expiry watcher (permanent finalization) now drops the session's
per-session control dicts (_session_model_overrides, _session_reasoning_overrides,
_pending_approvals, _update_prompt_pending, _pending_model_notes). These leaked
one entry per session per gateway lifetime. NOTE: this is the session-finalize
path, NOT idle agent-cache eviction — an idle-evicted session is still alive and
rebuilds its agent from these overrides, so pruning them there would silently
reset a user's /model choice.
4. _tool_defs_cache is now bounded (_TOOL_DEFS_CACHE_MAX=8) with oldest-first
eviction instead of growing unboundedly across the distinct toolset/config
fingerprints a gateway sees over its lifetime.
Salvaged from #25318 by Michael Steuer (@mssteuer); fix 3 redirected from the
idle-sweep to the session-finalize lifecycle, magic number 8 lifted to a named
constant, test ported.
Fixes#19251
Co-authored-by: Michael Steuer <michael@make.software>
test_concurrent_compressions_same_session_serialize relied on a
time.sleep(0.25) inside the stubbed compressor to make the two threads
overlap inside the per-session lock window. Under CI CPU starvation that
sleep is insufficient: one thread can acquire -> compress -> rotate ->
RELEASE the lock before the other reaches try_acquire, so both acquire on
the shared session_id and both compress (the recurring 'Expected exactly
one agent to compress, got 2' failure on shard test (1)).
Replace the timing dependency with a threading.Barrier(2) wrapped around
the shared db's try_acquire_compression_lock: both threads rendezvous
immediately before the real (atomic) acquire, guaranteeing genuine
simultaneous contention regardless of scheduling. The real lock logic is
unchanged and still picks exactly one winner — this only fixes the test's
overlap guarantee. Restored after join so the post-join lock-leak
assertion hits the unwrapped method.
Verified: 20/20 plain + 15/15 under all-core CPU stress (load avg ~4.6),
where the old version flaked.
#41076 makes `hermes plugins list` discover nested category plugins (e.g.
observability/nemo_relay). This adds the missing enable/disable mutation path
so those plugins can actually be toggled, and fixes two incomplete-update
breakages on the #41076 base.
Before: `hermes plugins enable nemo_relay` -> "Plugin 'nemo_relay' is not
installed or bundled." (exit 1), because cmd_enable/cmd_disable went through
_plugin_exists(), which only checked top-level plugins/<name>/.
Changes:
- Add _resolve_plugin_key(): resolve a bare manifest/leaf name OR a full
path-derived key (observability/nemo_relay) to the canonical key the runtime
loader gates on, reusing #41076's _discover_all_plugins(). A bare leaf name
ambiguous across two categories resolves to None rather than silently picking
one.
- cmd_enable/cmd_disable resolve first, persist the canonical key, and drop any
stale legacy bare-name alias so the enabled/disabled lists can't drift into a
contradictory state. _plugin_exists delegates to the same resolver.
- Fix#41076 base breakages: _discover_all_plugins now returns 6-tuples, but
web_server._merged_plugins_hub() still unpacked 5 (ValueError on the
dashboard plugins-hub endpoint) and several test_plugins_cmd_list.py fixtures
were still 5-tuples. Both updated; the hub status check is now key-aware.
Verified e2e on the real CLI + runtime loader (isolated HERMES_HOME):
`hermes plugins enable nemo_relay` writes observability/nemo_relay to
config.yaml and the loader then loads it (enabled=True, error=None); a stale
bare-name alias is cleared on disable; the dashboard _merged_plugins_hub() runs
without crashing. Adds resolution + enable/disable tests; full
tests/hermes_cli/test_plugins_cmd* + web_server plugin tests green.
Follow-up to #41076 (#41066). Branched from that PR's head.
Salvaged from #6600 (@kristianvast) — re-scoped to the voice half only and
rebased onto current main. The cascading-interrupt hang half of the original
PR landed independently in dd0d1222a, so this carries ONLY Problem 1.
When a voice/audio message arrives while the agent is busy on the same
session, it hit the interrupt path with empty text because STT only ran after
the running-agent guard — the voice was effectively lost. Now we transcribe
audio BEFORE signaling the agent (and on the fresh-message path), echo the raw
transcript back to the user (🎙️), and _enrich_message_with_transcription
returns (text, transcripts) so callers can echo. A new
_dequeue_pending_with_transcription drives the post-agent drain the same way.
Reapplied onto _prepare_inbound_message_text (inbound enrichment was extracted
from the inline dispatch block since the original PR).
Co-authored-by: Kristian Vastveit <kristian@agrointel.no>
When agent.interrupt() fires during an active LLM call, the main poll loop
force-closes the worker-local httpx client to stop token generation. That
raises a transport error (RemoteProtocolError) on the worker thread — the
EXPECTED consequence of our own close, not a network bug.
The streaming retry loop misclassified it as a transient connection error
and retried; each doomed retry stalled for the full stream-stale timeout
(up to 300s). Because the gateway caches AIAgent instances per session, the
stale worker outlived the interrupted turn and raced the next turn's request
on shared client state — the root of the multi-minute cascading-interrupt
hang reported in the wild.
Fix: a request-local _request_cancelled token set by the poll loop right
before the force-close, in both interruptible_api_call (non-streaming) and
interruptible_streaming_api_call. The worker's exception handler checks the
token and exits cleanly — no retry, no fallback, no 'reconnecting' status —
instead of treating the forced error as transient. The token is request-
local (not agent._interrupt_requested, which is cleared at turn boundaries)
so a stale worker outliving its turn still recognizes its own forced close.
Original diagnosis and fix by @kristianvast (PR #6600), against the then-
inline methods in run_agent.py. Those were since extracted into
agent/chat_completion_helpers.py, so the fix is reapplied there.
Co-authored-by: Kristian Vastveit <kristianvast@users.noreply.github.com>
A misconfigured/slow external memory provider could hold the agent in
the 'running' state for minutes after the final response was delivered.
MemoryManager.sync_all / queue_prefetch_all looped provider.sync_turn /
queue_prefetch INLINE on the turn-completion path; a provider making a
blocking network/daemon call (a broken Hindsight daemon was observed
blocking ~298s before failing) blocked run_conversation from returning.
Because every interface (CLI, TUI, gateway) marks the agent 'running'
until run_conversation returns, the agent stayed busy for the full block
and any follow-up message triggered an aggressive interrupt that dropped
the message.
Dispatch provider sync/prefetch to a lazily-created single-worker
background executor. sync_all / queue_prefetch_all return immediately;
work completes (or fails, logged) in the background. A single worker
serializes writes so turn N lands before turn N+1. flush_pending()
provides a barrier for session boundaries and deterministic tests.
shutdown_all() drains the executor with a bounded timeout so a wedged
provider can never hang teardown.
Builtin-only / no-provider sessions spawn no executor (zero new threads
in the common case).
Review feedback (#40998): `rm -rf` / `Remove-Item -Recurse -Force` on the
install dir is destructive -- a user might still want whatever is there.
Rename the broken checkout to a timestamped `<dir>.broken-<ts>` backup and
re-clone fresh, so nothing is ever deleted. Transient cleanup of a clone
attempt that fails within the same run is left as-is.
Behavioral coverage for install.sh's clone_repo() guard (removes a
commit-less checkout, keeps a real one, ignores a non-repo dir) plus a
contract check that install.ps1's repo-validity gate requires a resolvable
HEAD.
Lift the `_handle_*_command` cluster (2,077 LOC) out of HermesCLI into
hermes_cli/cli_commands_mixin.py; HermesCLI now inherits CLICommandsMixin so
every self.<handler> call resolves unchanged via the MRO. Behavior-neutral.
Import discipline mirrors gateway/slash_commands.py (PR #41886): neutral deps
imported at the mixin module top level; cli.py-internal helpers/constants
(_cprint, _ACCENT, save_config_value, ...) imported lazily inside each handler
via 'from cli import ...' so the mixin never imports cli at module scope.
cli.py 16215 -> 14139 LOC. One test mock repointed (cli.is_browser_debug_ready
-> hermes_cli.cli_commands_mixin.is_browser_debug_ready).
* fix(cli): set PYTHON env for node-gyp native builds on NixOS
node-gyp (triggered by node-pty during npm ci) looks for python3 on
PATH, which fails on NixOS because python3 lives in the nix store and
is not on the system PATH.
Add _nixos_build_env() — a two-tier helper that detects NixOS and:
1. Fast path: hermes venv python3 (~0s)
2. Fallback: nix-shell which python3 (~2-5s)
Wire it into _run_npm_install_deterministic() via a new env= parameter,
then pass it through cmd_gui() and _update_node_dependencies().
Non-NixOS systems: _nixos_build_env() returns None, behavior unchanged.
* fix(cli): merge _nixos_build_env() with os.environ, fix NixOS detection, add explicit return None
- Critical fix: both Tier 1 (venv) and Tier 2 (nix-shell) now return
{**os.environ, "PYTHON": ...} instead of {"PYTHON": ...} — subprocess.run
with env= replaces the entire environment, so the old code wiped PATH
and broke npm/node on NixOS entirely.
- Uses re.search(r"^ID=nixos$", ...) for anchored NixOS detection instead
of unanchored substring match (could match ID_LIKE=...nixos).
- Removes redundant Path.exists() guard before read_text(); just catches
OSError (one filesystem read instead of two).
- Adds explicit return None at end of function for type-hint consistency.
test_gateway_run_clamped read gateway/run.py asserting the /usage stats handler
clamps pct with min(100, ...). That handler moved to gateway/slash_commands.py
in this PR's extraction; repoint the guard so it still fires on clamp removal.
tests/run_agent/ + tests/gateway/ 8024 passed / 0 failed.
Tests for the extracted handlers mocked symbols at gateway.run.*; the handlers
now resolve top-level-imported deps (atomic_json_write, fetch_account_usage,
render_account_usage_lines) and __file__ from gateway.slash_commands. Repoint
those mocks. run.py-resident methods (_increment_restart_failure_counts,
_clear_restart_failure_count) keep their gateway.run.atomic_json_write mock —
only the moved handlers' mocks change.
tests/gateway/ 6415 passed / 0 failed.
A one-off transient transport failure (streaming-close / incomplete
chunked read / 5xx / 408) on an auxiliary LLM call escalated straight to
provider/model fallback (or, for context compression, dropped the summary
and entered cooldown), even when an immediate retry on the same provider
would have succeeded.
Add a single same-target retry at the top of call_llm() and
async_call_llm() — before the existing except-chain — gated on a new
_is_transient_transport_error() that reuses the canonical
_is_connection_error() detector plus a 5xx/408 status check. A second
failure (or any non-transient error: auth, other 4xx, malformed payload)
falls through to first_err and the existing fallback handling unchanged.
This lives in call_llm so every auxiliary task (compression, memory flush,
title generation, session search, vision) shares one transient-retry
surface, rather than each caller re-implementing it. The context
compressor needs no change — it calls call_llm and inherits the retry; its
existing fallback-to-main path (#18458) now composes naturally (retry the
aux model once, then fall back to main only if the retry also fails).
Co-authored-by: ARegalado1 <alberto.regalado@ymail.com>
Under systemd's Restart=always, --replace turns every restart into a
self-kill loop: the new instance reads gateway.pid, kills the previous
process, writes its own PID, and on the next restart the cycle repeats.
A process supervisor owns the lifecycle — --replace is for manual
one-shot takeovers and fights the supervisor.
Remove --replace from both the system-level and user-level systemd
ExecStart lines. The --replace flag stays available for manual
'hermes gateway run --replace' and on the macOS launchd fallback path
(#23387), which is a deliberate manual takeover, not a supervised unit.
Also drop RestartMaxDelaySec / RestartSteps from the templates — they
require systemd v255+ and are silently ignored on older versions. The
_strip_optional_systemd_directives normalizer stays so existing installs
whose on-disk unit still carries those directives aren't flagged as
outdated.
Credit: reported and diagnosed by @Skippy-the-Magnificent-one (PR #37145);
reimplemented here under project authorship because the original commit
was authored under a non-existent email.
Completes the worktree-misroute fix from #35399, which made misroutes
visible (resolved_path) but did not prevent them: its divergence warning
only fired once a terminal command had populated the live cwd registry.
A fresh worktree session (registry still empty) with a stale TERMINAL_CWD='.'
got neither a worktree anchor nor a warning, so a relative write_file/patch
silently landed in the MAIN checkout.
Two changes in tools/file_tools.py:
- Treat sentinel TERMINAL_CWD values ('', '.', './', 'auto', 'cwd') and any
relative value as UNSET rather than a literal anchor. Previously '.' was
joined onto the process cwd, silently routing edits to wherever the process
happened to be (the main repo, in a worktree session). The gateway already
sanitizes the same set at import time; the file-tool layer now matches.
- New _authoritative_workspace_root(): prefers the live terminal cwd, else a
sentinel-free absolute TERMINAL_CWD (the worktree path cli.py/main.py set
for -w). _resolve_base_dir() and _path_resolution_warning() both use it, so
a worktree session resolves into — and warns about escaping — the worktree
from the very first write, before any cd has run.
Validation: 11 new/parametrized tests (sentinel handling, empty-registry
anchoring, early divergence warning, live-cwd precedence). 32/32 pass under
scripts/run_tests.sh. Live E2E: relative write in an empty-registry worktree
session lands in the worktree, main untouched.
When --replace force-kills an unresponsive old gateway, SIGKILL can fail
to reap it (uninterruptible sleep, zombie-reaping parent, etc.). The old
code unconditionally cleared the PID file and scoped locks and started a
fresh instance anyway, leaving two live gateways fighting over the same
bot token — a duplicate-gateway failure mode of #19471.
Re-verify the process is actually gone (via the Windows-safe _pid_exists
helper) after the force-kill; if it still appears alive, clear the
takeover marker and abort the replacement instead of duplicating.
Co-authored-by: Hermes <noreply@nousresearch.com>
gateway/run.py is the largest god file (20k LOC, GatewayRunner with 220
methods). This lifts the cohesive kanban-watcher cluster — _kanban_notifier_watcher,
_kanban_dispatcher_watcher, _kanban_advance/unsub/rewind, _deliver_kanban_artifacts
(~1,035 LOC, 6 methods) — into gateway/kanban_watchers.py as a mixin that
GatewayRunner inherits.
Mixin (not free functions) because the methods use only self state: inheriting
keeps every self._kanban_* call site working unchanged via the MRO, making this
a behavior-neutral move. The methods' lazy imports (_kb, _decomp, _load_config,
Platform) travel with them; the mixin needs only stdlib + a matching
logging.getLogger('gateway.run').
run.py 20187 -> 19157 LOC; GatewayRunner direct methods 220 -> 214.
Behavior-neutral: gateway test suite 6582 passed / 0 failed; start() still wires
both watchers via self._kanban_*; MRO resolves all 6 to the mixin. One test
(corrupt-board quarantine retry) keyed its time-travel mock on the caller's
filename being gateway/run.py — updated to also accept gateway/kanban_watchers.py.
Establishes the mixin-extraction pattern for further GatewayRunner decomposition
(the 2406-LOC _run_agent and 1164-LOC _handle_message remain, but their callback
closures need a context-object redesign — deferred).
When register_task_env_overrides is called with only a 'cwd' key
(ACP adapter workspace tracking), the task_id should collapse to
'default' so all interactive surfaces (TUI, gateway, dashboard)
share one long-lived container.
Previously, any override registration — even CWD-only — caused
_resolve_container_task_id to return the session key unchanged,
spinning up a separate container per session. This made it
impossible to authenticate into external services once and have
that auth available across all surfaces.
Now only overrides containing isolation keys (docker_image,
modal_image, singularity_image, daytona_image, env_type) trigger
per-task container isolation.
Fixes#37361
Subcommands whose handler was a closure defined inside main() — memory, acp,
tools, insights, skills, pairing, plugins, mcp, claw — have their handler
promoted to a top-level function and their parser block extracted into
hermes_cli/subcommands/<name>.py (build_<name>_parser, injected handler).
These 9 had zero closure-over-main-locals, so promotion is a pure relocation.
acp/mcp parser blocks use the shared add_accept_hooks_flag helper.
main() 1798 -> 954 LOC (71% below the 3297 Phase-2 starting point);
add_parser calls in main.py 89 -> 28.
Deferred: sessions, computer-use, secrets handlers reference <name>_parser
(for a no-subcommand print_help fallback) — left in place to avoid the
_self_parser indirection; minority, low value.
Behavior-neutral: all 9 subcommands' --help (incl nested subactions) byte-
identical to pre-extraction (diff-verified). tests/hermes_cli/ 6519 passed /
0 failed; new test_subcommands_followup.py covers the 9 builders.
run_conversation's inner retry loop tracked recovery state in ~15 scattered
bare booleans (per-provider OAuth refresh guards, format-recovery guards,
restart signals). They are now fields on a single TurnRetryState dataclass the
loop mutates in place (_retry.<flag>), giving the recovery bookkeeping a named,
testable home.
Loop-control vars (retry_count, max_retries, max_compression_attempts) stay as
plain locals — they're while-mechanics, not recovery bookkeeping.
Behavior-neutral: pure local→attribute rewrite of 42 references; kwarg NAMES
preserved (e.g. has_retried_429=_retry.has_retried_429). Live simple + tool
turns OK.
Validation: tests/run_agent/ 1615 passed / 0 failed under per-file process
isolation; new test_turn_retry_state.py pins the field contract.
When context compaction rotates agent.session_id, it updates the gateway/tools
session context (set_current_session_id -> HERMES_SESSION_ID env + ContextVar)
but never updates the separate logging session context. The [session_id] tag on
log lines comes from hermes_logging._session_context (set once per turn in
conversation_loop.py), so post-compaction log lines in the same turn carry the
STALE old id while the message/DB/gateway state carry the new one — breaking log
correlation exactly at the compaction boundary.
Call hermes_logging.set_session_context(agent.session_id) alongside the existing
set_current_session_id, guarded so a logging failure can't regress the routing
update. Logs-only; no runtime or caching impact.
Refs #34089
The curator's idle-archival path (apply_automatic_transitions under
prune_builtins) could archive the bundled `plan` skill, killing the
/plan slash command silently — typing /plan then returned 'Unknown
command' with no signal that a skill had vanished. The archived skill's
hash stays in .bundled_manifest, so 'hermes update' wouldn't re-seed it.
Add PROTECTED_BUILTIN_SKILLS ({plan}) enforced at the master gate
is_curation_eligible() (covers archive_skill + the transition walk) and
in the candidate enumerator (so the LLM consolidation pass never sees
them). Immune to prune_builtins, pin state, and LLM judgment.
Closes#33617. Adds additive _meta.hermes.sessionProvenance to ACP session
surfaces so clients can detect compression-driven internal session rotation
without parsing status text, guessing from token drops, or reading state.db.
Derived on demand from the existing compression chain (parent_session_id /
end_reason) — no new persisted state, no schema change, no ACP protocol change.
ACP session_id stays the stable client handle.
- acp_adapter/provenance.py: derive provenance from SessionDB
- server.py: attach _meta to new/load/resume responses; emit a
session_info_update when the internal head rotates during a prompt
VolcEngine's api/plan endpoint occasionally leaks raw XML attribute
fragments into tool_use.name when its protocol-translation layer
converts the model's native XML-style tool emission to Anthropic
Messages tool_use blocks, producing names like:
terminal" parameter="command" string="true
execute_code" parameter="code" string="true
session_search" parameter="session_id" string="true
The corruption happens server-side at the provider, but it breaks
every tool call for affected users — no normalization rule in
repair_tool_call can rescue them, so each request runs through three
retries and then aborts as partial.
Add an early sanitizer in agent_runtime_helpers.repair_tool_call that
trims at the first ' " ', " ' ", '<', or '>' character (idx > 0
only) so the rest of the existing repair pipeline (lowercase /
snake_case / fuzzy match) can resolve the cleaned name normally.
Whitespace is deliberately NOT a separator — the legitimate
"write file" -> write_file repair path (covered by
test_space_to_underscore) must keep working.
Tests: 11 new regression cases in TestVolcEngineXmlPollution
covering all three observed polluted names, CamelCase + pollution
mix, single-quote variants, angle-bracket variants, clean-name
passthrough, and the whitespace-preservation guard. All 18 pre-
existing repair tests still pass (29 total in the file).
The auxiliary Codex adapter maintained its own chat->Responses conversion
loop that forwarded every non-system message's role verbatim into
Responses input[]. When flush_memories()/compression replayed session
history containing assistant tool_calls + role=tool results, those tool
messages leaked into the request and the Responses API rejected them with
HTTP 400: Invalid value: 'tool'.
Route _CodexCompletionsAdapter.create() through the same shared converter
the main agent transport uses (_chat_messages_to_responses_input), so tool
calls become function_call items and tool results become function_call_output
items with a valid call_id. Single conversion path means no future drift.
Also remove the now-dead _convert_content_for_responses() helper — its only
caller was the private conversion loop this change deletes.
Co-authored-by: ProgramCaiCai <techxacm@gmail.com>
Batch extraction of every remaining subcommand whose handler is top-level and
whose parser block is pure argparse: model, setup, postinstall, whatsapp, slack,
login, logout, auth, status, webhook, hooks, doctor, security, dump, debug,
backup, import, config, version, update, uninstall, dashboard, gui, logs,
prompt-size.
Each becomes hermes_cli/subcommands/<name>.py with build_<name>_parser() and an
injected handler (no main import). dashboard also injects cmd_dashboard_register
for its nested 'register' action.
Behavior-neutral: all 25 subcommands' --help output (and nested subaction help)
diff-verified byte-identical to pre-extraction. Two RawDescriptionHelpFormatter
epilogs (debug, logs) needed their multi-line string interiors preserved at
column 0 — caught by the --help diff, not compile.
main() 3297 -> 1798 LOC across this PR; add_parser calls in main.py 179 -> 89.
Validation: tests/hermes_cli/ 6476 passed / 0 failed under per-file process
isolation; new test_subcommands_batch.py smoke-tests all 25 builders + the
dashboard two-handler case.
Follow-on to the cron extraction in the same Phase 2 PR. Same pattern:
per-group build_<name>_parser() functions with injected handlers, no main
import.
- subcommands/profile.py: build_profile_parser (190-line block out of main()).
- subcommands/gateway.py: build_gateway_parser (gateway + proxy, 238-line block;
they shared one inline section). Imports argparse for SUPPRESS defaults.
- main(): two more inline blocks become single builder calls.
Behavior-neutral: 'profile [sub] --help' and 'gateway/proxy [sub] --help'
byte-identical to pre-extraction (diff-verified).
main() now 2723 LOC (was 3297 at Phase 2 start); add_parser calls in main.py
179 -> 141.
Validation: tests/hermes_cli/ 6476 passed / 0 failed under per-file process
isolation; new builder unit tests cover subactions, aliases, dispatch, flags.
Phase 2 of the god-file decomposition plan. main()'s argparse tree is 179
inline add_parser calls in one 3,297-line function. This establishes the
hermes_cli/subcommands/ package and extracts the first group (cron) as the
proof-of-pattern:
- hermes_cli/subcommands/_shared.py: shared parser helpers (add_accept_hooks_flag),
re-exported from main.py for backwards compat.
- hermes_cli/subcommands/cron.py: build_cron_parser(subparsers, cmd_cron=...).
Handler injected so the module never imports main (cycle avoidance).
- main()'s ~155-line inline cron block becomes one build_cron_parser() call.
Behavior-neutral: 'hermes cron create --help' output is byte-identical to
origin/main. main() 3297 -> 3143 LOC.
Validation: tests/hermes_cli/ 6466 passed / 0 failed under per-file process
isolation; new test_subcommands_cron.py covers subactions, aliases, options,
no-agent tristate, injected dispatch, and --accept-hooks.
Phase 1 of the god-file decomposition plan. run_conversation's ~470-line
once-per-turn setup block (stdio guarding, retry-counter resets, user-message
sanitization, todo/nudge hydration, system-prompt restore-or-build,
crash-resilience persistence, preflight compression, the pre_llm_call hook, and
external-memory prefetch) is moved verbatim into build_turn_context(), which
returns a TurnContext dataclass the loop unpacks.
Behavior-neutral move-and-name refactor: the builder mutates `agent` exactly as
the inline code did; only the locals the loop reads back are returned.
- run_conversation: 4602 -> 4217 LOC (-385)
- agent/conversation_loop.py: 4965 -> ~4580 LOC
- new agent/turn_context.py: focused, dependency-injected, unit-tested in isolation
Tests: tests/run_agent/ 1570 passed / 0 failed under per-file process isolation.
Relocation follow-ups: 413_compression mocks now patch both module references;
nudge/on_turn_start source-inspection guards point at the extracted module.
* fix(memory): make overflow errors instruct in-turn consolidation + retry
When bounded memory is full, the add/replace overflow errors now explicitly
tell the model to consolidate (merge/remove/shorten) and retry the write in
the same turn, matching the documented behavior. The replace-overflow path
now also echoes current_entries + usage for parity with add-overflow, so the
model has the same context to act on.
Closes#23378 (working-as-documented; this sharpens runtime to match docs).
* fix(memory): broaden overflow remediation hint beyond 'stale'
Say 'stale or less important' — entries don't have to be stale to be the
right ones to drop when making room.
Session-scoped /model and /reasoning overrides were silently lost on
Telegram DM/forum topics and after compression session splits (#30479).
Root cause: _handle_message_with_agent rewrites source.thread_id via
_recover_telegram_topic_thread_id (lobby/stripped reply -> the user's
bound topic) before deriving the session key. The /model and /reasoning
handlers derived their override key from the raw inbound event.source,
skipping that recovery, so the override was stored under one key and the
next message turn read a different key.
Fix: add _normalize_source_for_session_key (applies the same recovery a
message turn does) and use it in both handlers before deriving the key.
session_id rotation on compression was never the cause — overrides are
keyed by the durable session_key; the split path preserves it.
Author: teknium1 <127238744+teknium1@users.noreply.github.com>
