On fresh RHEL/Debian SSH sessions without linger, `systemctl --user
start hermes-gateway` fails with 'Failed to connect to bus: No medium
found' because /run/user/$UID/bus doesn't exist. Setup previously
showed a raw CalledProcessError and continued claiming success, so the
gateway never actually started.
systemd_start() and systemd_restart() now call _preflight_user_systemd()
for the user scope first:
- Bus socket already there → no-op (desktop / linger-enabled servers)
- Linger off → try loginctl enable-linger (works when polkit permits,
needs sudo otherwise), wait for socket
- Still unreachable → raise UserSystemdUnavailableError with a clean
remediation message pointing to sudo loginctl + hermes gateway run
as the foreground fallback
Setup's start/restart handlers and gateway_command() catch the new
exception and render the multi-line guidance instead of a traceback.
* fix(gateway): detect legacy hermes.service units from pre-rename installs
Older Hermes installs used a different service name (hermes.service) before
the rename to hermes-gateway.service. When both units remain installed, they
fight over the same bot token — after PR #5646's signal-recovery change,
this manifests as a 30-second SIGTERM flap loop between the two services.
Detection is an explicit allowlist (no globbing) plus an ExecStart content
check, so profile units (hermes-gateway-<profile>.service) and unrelated
third-party services named 'hermes' are never matched.
Wired into systemd_install, systemd_status, gateway_setup wizard, and the
main hermes setup flow — anywhere we already warn about scope conflicts now
also warns about legacy units.
* feat(gateway): add migrate-legacy command + install-time removal prompt
- New hermes_cli.gateway.remove_legacy_hermes_units() removes legacy
unit files with stop → disable → unlink → daemon-reload. Handles user
and system scopes separately; system scope returns path list when not
running as root so the caller can tell the user to re-run with sudo.
- New 'hermes gateway migrate-legacy' subcommand (with --dry-run and -y)
routes to remove_legacy_hermes_units via gateway_command dispatch.
- systemd_install now offers to remove legacy units BEFORE installing
the new hermes-gateway.service, preventing the SIGTERM flap loop that
hits users who still have pre-rename hermes.service around.
Profile units (hermes-gateway-<profile>.service) remain untouched in
all paths — the legacy allowlist is explicit (_LEGACY_SERVICE_NAMES)
and the ExecStart content check further narrows matches.
* fix(gateway): mark --replace SIGTERM as planned so target exits 0
PR #5646 made SIGTERM exit the gateway with code 1 so systemd's
Restart=on-failure revives it after unexpected kills. But when a user has
two gateway units fighting for the same bot token (e.g. legacy
hermes.service + hermes-gateway.service from a pre-rename install), the
--replace takeover itself becomes the 'unexpected' SIGTERM — the loser
exits 1, systemd revives it 30s later, and the cycle flaps indefinitely.
Before calling terminate_pid(), --replace now writes a short-lived marker
file naming the target PID + start_time. The target's shutdown_signal_handler
consumes the marker and, when it names this process, leaves
_signal_initiated_shutdown=False so the final exit code stays 0.
Staleness defences:
- PID + start_time combo prevents PID reuse matching an old marker
- Marker older than 60s is treated as stale and discarded
- Marker is unlinked on first read even if it doesn't match this process
- Replacer clears the marker post-loop + on permission-denied give-up
Salvaged from PR #10643 by kshitijk4poor, updated for current main.
Root causes fixed:
1. Telegram xdist mock pollution — new tests/gateway/conftest.py with shared
mock that runs at collection time (prevents ChatType=None caching)
2. VIRTUAL_ENV env var leak — monkeypatch.delenv in _detect_venv_dir tests
3. Copilot base_url missing — add fallback in _resolve_runtime_from_pool_entry
4. Stale vision model assertion — zai now uses glm-5v-turbo
5. Reasoning item id intentionally stripped — assert 'id' not in (store=False)
6. Context length warning unreachable — pass base_url to AIAgent in test
7. Kimi provider label updated — 'Kimi / Kimi Coding Plan' matches models.py
8. Google Workspace calendar tests — rewritten for current production code,
properly mock subprocess on api_module, removed stale +agenda assertions
9. Credential pool auto-seeding — mock _select_pool_entry / _resolve_auto /
_import_codex_cli_tokens to prevent real credentials from leaking into tests
Previously, systemd_restart() sent SIGUSR1 to the gateway, printed
'restart requested', and returned immediately. The gateway still
needed to drain active agents, exit with code 75, wait for systemd's
RestartSec=30, and start the new process. The user saw 'success' but
the gateway was actually down for 30-60 seconds.
Now the SIGUSR1 path blocks with progress feedback:
Phase 1 — wait for old process to die:
⏳ User service draining active work...
Polls os.kill(pid, 0) until ProcessLookupError (up to 90s)
Phase 2 — wait for new process to become active:
⏳ Waiting for hermes-gateway to restart...
Polls systemctl is-active + verifies new PID (up to 60s)
Success:
✓ User service restarted (PID 12345)
Timeout:
⚠ User service did not become active within 60s.
Check status: hermes gateway status
Check logs: journalctl --user -u hermes-gateway --since '2 min ago'
The reload-or-restart fallback path (line 1189) already blocks because
systemctl reload-or-restart is synchronous.
Test plan:
- Updated test to verify wait-for-restart behavior
- All 118 gateway CLI tests pass
Centralize container detection in hermes_constants.is_container() with
process-lifetime caching, matching existing is_wsl()/is_termux() patterns.
Dedup _is_inside_container() in config.py to delegate to the new function.
Add _run_systemctl() wrapper that converts FileNotFoundError to RuntimeError
for defense-in-depth — all 10 bare subprocess.run(_systemctl_cmd(...)) call
sites now route through it.
Make supports_systemd_services() return False in containers and when
systemctl binary is absent (shutil.which check).
Add Docker-specific guidance in gateway_command() for install/uninstall/start
subcommands — exit 0 with helpful instructions instead of crashing.
Make 'hermes status' show 'Manager: docker (foreground)' and 'hermes dump'
show 'running (docker, pid N)' inside containers.
Fix setup_gateway() to use supports_systemd instead of _is_linux for all
systemd-related branches, and show Docker restart policy instructions in
containers.
Replace inline /.dockerenv check in voice_mode.py with is_container().
Fixes#7420
Co-authored-by: teknium1 <teknium1@users.noreply.github.com>
- add all_profiles=False to find_gateway_pids() and
kill_gateway_processes() so hermes update and gateway stop --all
can still discover processes across all profiles
- narrow bare 'except Exception' to (OSError, subprocess.TimeoutExpired)
- update test mocks to match new signatures
In Docker, HERMES_HOME=/opt/data (set in Dockerfile) and users mount
their .hermes directory to /opt/data. However, profile operations used
Path.home() / '.hermes' which resolves to /root/.hermes in Docker —
an ephemeral container path, not the mounted volume.
This caused:
- Profiles created at /root/.hermes/profiles/ (lost on container recreate)
- active_profile sticky file written to wrong location
- profile list looking at wrong directory
Fix: Add get_default_hermes_root() to hermes_constants.py that detects
Docker/custom deployments (HERMES_HOME outside ~/.hermes) and returns
HERMES_HOME as the root. Also handles Docker profiles correctly
(<root>/profiles/<name> → root is grandparent).
Files changed:
- hermes_constants.py: new get_default_hermes_root()
- hermes_cli/profiles.py: _get_default_hermes_home() delegates to shared fn
- hermes_cli/main.py: _apply_profile_override() + _invalidate_update_cache()
- hermes_cli/gateway.py: _profile_suffix() + _profile_arg()
- Tests: 12 new tests covering Docker scenarios
launchd_stop() previously used `launchctl kill SIGTERM` which only
signals the process. Because the plist has KeepAlive.SuccessfulExit=false,
launchd immediately respawns the gateway — making `hermes gateway stop`
a no-op that prints '✓ Service stopped' while the service keeps running.
Switch to `launchctl bootout` which unloads the service definition so
KeepAlive can't trigger. The process exits and stays stopped until
`hermes gateway start` (which already handles re-bootstrapping unloaded
jobs via error codes 3/113).
Also adds _wait_for_gateway_exit() after bootout to ensure the process
is fully gone before returning, and tolerates 'already unloaded' errors.
Fixes: .env changes not taking effect after gateway stop+restart on macOS.
The root cause was that stop didn't actually stop — the respawned process
loaded the old env before the user's restart command ran.
When installing a system service via sudo, ExecStart, WorkingDirectory,
VIRTUAL_ENV, and PATH entries were not remapped to the target user's
home — only HERMES_HOME was. This caused the service to fail with
status=200/CHDIR because the target user cannot access /root/.
Adds _remap_path_for_user() helper and applies it to all path variables
in the system branch of generate_systemd_unit().
Closes#6989
- Add 7 unit tests for _profile_arg: default home, named profile,
hash path, nested path, invalid name, systemd integration, launchd integration
- Add stt.local.language to config.yaml (empty = auto-detect)
- Both STT code paths now read config.yaml first, env var fallback,
then default (auto-detect for faster-whisper, 'en' for CLI command)
- HERMES_LOCAL_STT_LANGUAGE env var still works as backward-compat fallback
launchctl kickstart returns exit code 113 ("Could not find service") when
the plist exists but the job hasn't been bootstrapped into the runtime domain.
The existing recovery path only caught exit code 3 ("unloaded"), causing an
unhandled CalledProcessError.
Exit code 113 means the same thing practically -- the service definition needs
bootstrapping before it can be kicked. Add it to the same recovery path that
already handles exit 3, matching the existing pattern in launchd_stop().
Follow-up: add a unit test covering the 113 recovery path.
Replace the two-step stop/start restart with a single
launchctl kickstart -k call. When the gateway triggers a
restart from inside its own process tree, the old stop
command kills the shell before the start half is reached.
kickstart -k lets launchd handle the kill+restart atomically.
launchctl load/unload/start/stop are deprecated on macOS since 10.10
and fail silently on modern versions. This replaces them with the
current equivalents:
- load -> bootstrap gui/<uid> <plist>
- unload -> bootout gui/<uid>/<label>
- start -> kickstart gui/<uid>/<label>
- stop -> kill SIGTERM gui/<uid>/<label>
Adds _launchd_domain() helper returning the gui/<uid> target domain.
Updates test assertions to match the new command signatures.
Fixes#4820
gateway stop and restart previously called kill_gateway_processes() which
scans ps aux and kills ALL gateway processes across all profiles. Starting
a profile gateway would nuke the main one (and vice versa).
Now:
- hermes gateway stop → only kills the current profile's gateway (PID file)
- hermes -p work gateway stop → only kills the 'work' profile's gateway
- hermes gateway stop --all → kills every gateway process (old behavior)
- hermes gateway restart → profile-scoped for manual fallback path
- hermes update → discovers and restarts ALL profile gateways (systemctl
list-units hermes-gateway*) since the code update is shared
Added stop_profile_gateway() which uses the HERMES_HOME-scoped PID file
instead of global process scanning.
Previously, `hermes gateway install --system` hard-refused to create a
service running as root, even when explicitly requested via
`--run-as-user root`. This forced LXC/container users (where root is
the only user) to either create throwaway users or comment out the check
in source.
Changes:
- Auto-detected root (no explicit --run-as-user) still raises, but with
a message explaining how to override
- Explicit `--run-as-user root` now allowed with a warning about
security implications
- Interactive setup wizard prompt accepts 'root' as a valid username
(warning comes from _system_service_identity downstream)
- Added tests for all three paths: auto-detected root rejection,
explicit root allowance, and normal non-root passthrough
- stderr handler now uses RedactingFormatter to match file handlers
- restart path uses verbose=0 (int) instead of verbose=False (bool)
- test mock updated with new run_gateway(verbose, quiet, replace) signature
When `sudo hermes gateway install --system --run-as-user <user>` generates
the systemd unit, get_hermes_home() resolves to /root/.hermes because
Path.home() returns root's home under sudo. The unit correctly sets
HOME= and User= via _system_service_identity(), but HERMES_HOME was
computed independently and pointed to root's config directory.
Add _hermes_home_for_target_user() which remaps the current HERMES_HOME
to the equivalent path under the target user's home. This handles:
- Default ~/.hermes → target user's ~/.hermes
- Profiles (e.g. ~/.hermes/profiles/coder) → preserves relative structure
- Custom paths (e.g. /opt/hermes) → kept as-is
Supersedes #3861 which only handled the default case and left profiles
broken (also flagged by Copilot review).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: GPT tool-use steering + strip budget warnings from history
Two changes to improve tool reliability, especially for OpenAI GPT models:
1. GPT tool-use enforcement prompt: Adds GPT_TOOL_USE_GUIDANCE to the
system prompt when the model name contains 'gpt' and tools are loaded.
This addresses a known behavioral pattern where GPT models describe
intended actions ('I will run the tests') instead of actually making
tool calls. Inspired by similar steering in OpenCode (beast.txt) and
Cline (GPT-5.1 variant).
2. Budget warning history stripping: Budget pressure warnings injected by
_get_budget_warning() into tool results are now stripped when
conversation history is replayed via run_conversation(). Previously,
these turn-scoped signals persisted across turns, causing models to
avoid tool calls in all subsequent messages after any turn that hit
the 70-90% iteration threshold.
* fix: replace hardcoded ~/.hermes paths with get_hermes_home() for profile support
Prep for the upcoming profiles feature — each profile is a separate
HERMES_HOME directory, so all paths must respect the env var.
Fixes:
- gateway/platforms/matrix.py: Matrix E2EE store was hardcoded to
~/.hermes/matrix/store, ignoring HERMES_HOME. Now uses
get_hermes_home() so each profile gets its own Matrix state.
- gateway/platforms/telegram.py: Two locations reading config.yaml via
Path.home()/.hermes instead of get_hermes_home(). DM topic thread_id
persistence and hot-reload would read the wrong config in a profile.
- tools/file_tools.py: Security path for hub index blocking was
hardcoded to ~/.hermes, would miss the actual profile's hub cache.
- hermes_cli/gateway.py: Service naming now uses the profile name
(hermes-gateway-coder) instead of a cryptic hash suffix. Extracted
_profile_suffix() helper shared by systemd and launchd.
- hermes_cli/gateway.py: Launchd plist path and Label now scoped per
profile (ai.hermes.gateway-coder.plist). Previously all profiles
would collide on the same plist file on macOS.
- hermes_cli/gateway.py: Launchd plist now includes HERMES_HOME in
EnvironmentVariables — was missing entirely, making custom
HERMES_HOME broken on macOS launchd (pre-existing bug).
- All launchctl commands in gateway.py, main.py, status.py updated
to use get_launchd_label() instead of hardcoded string.
Test fixes: DM topic tests now set HERMES_HOME env var alongside
Path.home() mock. Launchd test uses get_launchd_label() for expected
commands.
Add ~/.local/bin, ~/.cargo/bin, ~/go/bin, ~/.npm-global/bin to the
systemd unit PATH so tools installed via uv/pipx/cargo/go are
discoverable by MCP servers and terminal commands.
Uses a _build_user_local_paths() helper that checks exists() before
adding, and correctly resolves home dir for both user and system
service types.
Co-authored-by: Kal Sze <ksze@users.noreply.github.com>
Fixes#2492.
`generate_systemd_unit()` and `get_python_path()` hardcoded `venv`
as the virtualenv directory name. When the virtualenv is `.venv`
(which `setup-hermes.sh` and `.gitignore` both reference), the
generated systemd unit had incorrect VIRTUAL_ENV and PATH variables.
Introduce `_detect_venv_dir()` which:
1. Checks `sys.prefix` vs `sys.base_prefix` to detect the active venv
2. Falls back to probing `.venv` then `venv` under PROJECT_ROOT
Both `get_python_path()` and `generate_systemd_unit()` now use
this detection instead of hardcoded paths.
Co-authored-by: Hermes <hermes@nousresearch.ai>
Python 3.12 changed PosixPath.__new__ to ignore the redirected path
argument, breaking the FakePath subclass pattern. Use monkeypatch on
Path.exists instead.
Based on PR #2261 by @dieutx, fixed NameError (bare Path not imported).
Repair stale launchd/systemd definitions during install and
teach launchd start to reload unloaded jobs before retrying.
Stop masking service restart failures by falling back to a
foreground gateway when a configured service manager is still
broken.
Refs: #1613
* fix: Anthropic OAuth compatibility — Claude Code identity fingerprinting
Anthropic routes OAuth/subscription requests based on Claude Code's
identity markers. Without them, requests get intermittent 500 errors
(~25% failure rate observed). This matches what pi-ai (clawdbot) and
OpenCode both implement for OAuth compatibility.
Changes (OAuth tokens only — API key users unaffected):
1. Headers: user-agent 'claude-cli/2.1.2 (external, cli)' + x-app 'cli'
2. System prompt: prepend 'You are Claude Code, Anthropic's official CLI'
3. System prompt sanitization: replace Hermes/Nous references
4. Tool names: prefix with 'mcp_' (Claude Code convention for non-native tools)
5. Tool name stripping: remove 'mcp_' prefix from response tool calls
Before: 9/12 OK, 1 hard fail, 4 needed retries (~25% error rate)
After: 16/16 OK, 0 failures, 0 retries (0% error rate)
* fix: auto-detect DBUS_SESSION_BUS_ADDRESS for systemctl --user on headless servers
On SSH sessions to headless servers, DBUS_SESSION_BUS_ADDRESS and
XDG_RUNTIME_DIR may not be set even when the user's systemd instance
is running via linger. This causes 'systemctl --user' to fail with
'Failed to connect to bus: No medium found', breaking gateway
restart/start/stop as a service and falling back to foreground mode.
Add _ensure_user_systemd_env() that detects the standard D-Bus socket
at /run/user/<UID>/bus and sets the env vars before any systemctl --user
call. Called from _systemctl_cmd() so all existing call sites benefit
automatically with zero changes.
Fixes: gateway restart falling back to foreground on headless servers
* fix: show linger guidance when gateway restart fails during update and gateway restart
When systemctl --user restart fails during 'hermes update' or
'hermes gateway restart', check linger status and tell the user
exactly what to run (sudo -S -p '' loginctl enable-linger) instead of
silently falling back to foreground mode.
Also applies _ensure_user_systemd_env() to the raw systemctl calls
in cmd_update so they work properly on SSH sessions where D-Bus
env vars are missing.
* fix(gateway): avoid recursive ExecStop in user systemd unit
* fix: extend ExecStop removal and TimeoutStopSec=60 to system unit
The cherry-picked PR #1448 fix only covered the user systemd unit.
The system unit had the same TimeoutStopSec=15 and could benefit
from the same 60s timeout for clean shutdown. Also adds a regression
test for the system unit.
---------
Co-authored-by: Ninja <ninja@local>
Multiple Hermes installations on the same machine now get unique
systemd service names:
- Default ~/.hermes → hermes-gateway (backward compatible)
- Custom HERMES_HOME → hermes-gateway-<8-char-hash>
Changes:
- Add get_service_name() in hermes_cli/gateway.py that derives a
deterministic service name from HERMES_HOME via SHA256
- Replace all hardcoded 'hermes-gateway' systemd references with
get_service_name() across gateway.py, main.py, status.py, uninstall.py
- Add HERMES_HOME env var to both user and system systemd unit templates
so the gateway process uses the correct installation
- Update tests to use get_service_name() in assertions
- store gateway PID metadata and validate the live process before trusting gateway.pid
- auto-refresh outdated systemd user units before start/restart so installs pick up --replace fixes
- sweep stray manual gateway processes after service stops
- add regression tests for PID validation and service drift recovery
