mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-06-19 10:02:16 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Teknium
|
fc086da8bd
|
fix(gateway,windows): reliability — JOB breakaway + status --deep probes + test-leak fix (#40909)
* fix(gateway,windows): reliability — supervisor task, JOB breakaway, status --deep
Three coordinated fixes for the Windows gateway reliability story:
1. CREATE_BREAKAWAY_FROM_JOB on every detached spawn
The 'hermes update' triggered from the Electron Desktop GUI ran inside
Electron's job object. Without breakaway, the post-update gateway
watcher spawned by update — already DETACHED_PROCESS — was still
reaped when Electron's job tore down, so the gateway never came back
after a GUI-initiated update. Adds CREATE_BREAKAWAY_FROM_JOB (0x01000000)
to:
- hermes_cli/_subprocess_compat.py::windows_detach_flags() — used by
every helper that calls windows_detach_popen_kwargs(), including
launch_detached_profile_gateway_restart()
- The watcher subprocess's own respawn snippet in
hermes_cli/gateway.py (inlined flags so the watcher's child
respawn also breaks away)
_spawn_detached() in gateway_windows.py already had the flag; this
change brings the rest of the codebase to parity.
2. Per-minute supervisor Scheduled Task — Windows equivalent of
systemd Restart=always
Introduces hermes_cli/gateway_supervisor.py and registers it as a
second Scheduled Task ('Hermes_Gateway_Supervisor', SC MINUTE /MO 1,
LIMITED rights) alongside the existing ONLOGON task. Every minute,
the supervisor uses the same gateway.status.get_running_pid() probe
as 'hermes gateway status' and, if no gateway is alive, calls
gateway_windows._spawn_detached() (which now includes BREAKAWAY) to
bring one back.
Covers every crash mode, not just 'machine rebooted': taskkill,
OOM, GUI update SIGTERM, parent job teardown. Cheap — one pythonw
startup per minute when down, one PID-existence check per minute
when up.
Wired into both the schtasks-success and Startup-folder-fallback
install paths via _install_supervisor_best_effort(), and removed in
uninstall(). Best-effort: a failing supervisor install logs a
warning but doesn't roll back the primary install.
3. 'hermes gateway status --deep' shows per-probe PASS/FAIL
Replaces the existing terse '--deep' output (which only printed
paths) with an actual diagnostic table:
[1] PID file present
[2] Lock file held by a live process
[3] get_running_pid() result
[4] _pid_exists(pid) — OS-level liveness
[5] gateway_state.json (state + age)
[6] Last lifecycle event from gateway-exit-diag.log
When the high-level summary disagrees with reality, the user can
see exactly which signal is lying.
Test-leak fix
-------------
tests/hermes_cli/test_gateway_wsl.py::TestGatewayCommandWSLMessages
monkey-patched is_linux/is_wsl/supports_systemd_services to simulate
WSL but did NOT stub is_windows(). On a Windows host, the dispatcher
in _gateway_command_inner takes the is_windows() branch BEFORE the
WSL guidance branch, so the test invoked gateway_windows.install()
for real. install() writes to %APPDATA%\...\Startup\Hermes_Gateway.cmd
— the REAL user Startup folder, never sandboxed by tmp_path — pointing
at the test's pytest-of-<user>/pytest-<N>/.../gateway-service/ wrapper.
When pytest tore down the tmp_path, every subsequent Windows login
flashed a cmd.exe window that failed to find the missing target.
Stubs is_windows=False on all four affected tests:
test_install_wsl_no_systemd
test_start_wsl_no_systemd
test_status_wsl_running_manual
test_status_wsl_not_running
Defense-in-depth: _build_startup_launcher() now prefixes the launcher
with 'if not exist <target> exit /b 0', so any future stale Startup
entry silently no-ops instead of flashing a console window.
Status enhancements
-------------------
- status() now reports supervisor task presence alongside the existing
schtasks/Startup info, and nudges the user to reinstall if the
supervisor isn't registered.
- Deep mode dumps both the supervisor task name + script path.
* fix(gateway,windows): drop the per-minute supervisor task — keep breakaway + deep probes
Earlier in this branch we added a per-minute schtasks-based supervisor to
respawn the gateway after crashes / GUI-update SIGTERMs. The implementation
flashed a brief console window on every firing, which stole window focus.
We tried several variants:
- cmd.exe wrapper invoking pythonw -> flashes (cmd.exe is console-subsystem)
- schtasks /TR pointing at pythonw -> flashes (uv venv launcher pythonw is
actually subsystem=Console, not GUI; it respawns the real pythonw)
- schtasks /TR pointing at base uv -> still flashes (Task Scheduler-side
conhost preallocation; documented Windows quirk)
- XML registration with <Hidden>true> -> still flashes (<Hidden> only hides
the task in the Task Scheduler UI, not the spawned window)
Researched what leading projects do:
- Ollama: GUI-subsystem tray exe + Startup-folder shortcut. No supervisor.
- Tailscale: real Windows Service via SCM. Session 0, no console possible.
- Syncthing: --no-console flag inside the binary + Startup folder.
- openclaw: VBS Run(..., 0, False) wrapper. Suppresses the *window* but
Super User Q971162 confirms focus-steal still occurs in some cases.
None of these use a per-minute polling scheduled task. The 'auto-restart on
crash' responsibility belongs INSIDE the daemon (Tailscale's in-process
recovery / Ollama's monitor+worker pair) OR is delegated to the Windows
Service Control Manager — not Task Scheduler.
So this commit drops the supervisor entirely. The CREATE_BREAKAWAY_FROM_JOB
fix in _subprocess_compat.py (from commit
|
||
|
kshitijk4poor
|
66827f8947 |
chore: prune unused imports and duplicate import redefinitions
Remove unused imports (F401) and duplicate/shadowed import redefinitions (F811) across the codebase using ruff's safe autofixes. No behavioral changes -- imports only. - ~1400 safe autofixes applied across 644 files (net -1072 lines) - __init__.py re-exports preserved (excluded from F401 removal so public re-export surfaces stay intact) - Re-exports that are imported or monkeypatched by tests but look unused in their defining module are kept with explicit # noqa: F401 (gateway/run.py load_dotenv; run_agent re-exports from agent.message_sanitization, agent.context_compressor, agent.retry_utils, agent.prompt_builder, agent.process_bootstrap, agent.codex_responses_adapter) - Unsafe F841 (unused-variable) fixes deliberately skipped -- those can change behavior when the RHS has side effects - ruff lints remain disabled in pyproject.toml (only PLW1514 is selected); this is a one-time cleanup, not a config change Verification: - python -m compileall: clean - pytest --collect-only: all 27161 tests collect (zero import errors) - core entry points import clean (run_agent, model_tools, cli, toolsets, hermes_state, batch_runner, gateway) - static scan: every name any test imports directly from an edited module still resolves |
||
|
Teknium
|
e2fd462ebe
|
ci(tests): add pytest-timeout 60s hard cap to break suite-teardown deadlock (#28861)
* ci(tests): add pytest-timeout 60s hard cap to break suite-teardown deadlock The full pytest suite reliably hangs at ~96% on origin/main, blowing through the 20-minute GHA job timeout on every CI push since yesterday. Individual tests complete in <30s — the deadlock builds up at session teardown after all tests run, when leaked threads and atexit handlers from thousands of tests interact and one of them lands in a futex-wait that never resolves. This PR is a stopgap that unblocks CI immediately + speeds up several slow tests we found while diagnosing. Changes - pyproject.toml: add pytest-timeout==2.4.0 to dev deps; bake --timeout=60 --timeout-method=thread into the default addopts. - scripts/run_tests.sh: re-add --timeout flags directly because the script wipes pyproject addopts with -o 'addopts='. - .github/workflows/tests.yml: explicit --timeout/--timeout-method on the CI pytest invocation for clarity. - gateway/run.py: in _run_agent, if the stream consumer was never created (e.g. non-streaming agent or test stub), cancel the stream_task immediately instead of waiting out the 5s wait_for timeout. ~5s saved per non-streaming gateway test run. - tests/run_agent/conftest.py: extend _fast_retry_backoff to patch agent.conversation_loop.jittered_backoff alongside run_agent.jittered_backoff. The retry loop was extracted into agent.conversation_loop which holds its own import — patching the run_agent reference alone left tests burning real wall-clock backoff seconds. - tests/run_agent/test_anthropic_error_handling.py tests/run_agent/test_run_agent.py (TestRetryExhaustion) tests/run_agent/test_fallback_model.py: same conversation_loop fix for per-test fixtures (defensive — the conftest covers them too). - tests/gateway/test_gateway_inactivity_timeout.py: trim run_duration 10.0 → 2.0 / 5.0 → 2.0 on three tests that wait the full SlowFakeAgent duration. Adjusted thresholds proportionally. - tests/gateway/test_api_server_runs.py: test_stop_interrupt_exception_does_not_crash trips the interrupted event in addition to raising, so the slow_run thread unblocks at teardown instead of waiting 10s. - tests/hermes_cli/test_update_gateway_restart.py: also patch time.monotonic in the autouse fixture. _wait_for_service_active loops on a wall-clock deadline; with sleep no-op'd the loop spun on real monotonic until 10s real-time per restart attempt (20s+ per test). - tests/tools/test_zombie_process_cleanup.py: cut runner._restart_drain_timeout 5.0 → 0.1 in test_gateway_stop_calls_close. Suite still hangs at 96% on full no-timeout runs; with these changes CI runs through to a real pass/fail signal. * chore(lock): regenerate uv.lock after adding pytest-timeout * ci: drop pytest-timeout 60 → 30s + bump GHA job 20 → 30 min Prior commit's timeout=60 was too generous — CI test job still hit the 20-min wall-clock cap with the suite hung at 96% (orphan agent-browser subprocesses blocking pytest session teardown). The local timeout=20 run completed in 6:17, so 30s is conservative enough to let real tests finish but aggressive enough to short-circuit deadlocks. Also bump GHA job timeout to 30 min as a safety margin. * test: delete 11 pre-existing failing tests + revert monotonic patch The previous PR commit landed pytest-timeout=30s and the suite now completes in 18:14 instead of hanging at 96%, but 11 pre-existing tests fail with real assertions. Per Teknium: nuke them. Deleted (no replacements): - tests/gateway/test_restart_resume_pending.py::test_clean_drain_does_not_mark_resume_pending - tests/gateway/test_restart_resume_pending.py::test_drain_timeout_only_marks_still_running_sessions - tests/hermes_cli/test_gateway_service.py::TestGatewaySystemServiceRouting::test_gateway_install_passes_system_flags - tests/hermes_cli/test_gateway_wsl.py::TestGatewayCommandWSLMessages::test_install_wsl_with_systemd_warns - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_detects_launchd_and_skips_manual_restart_message - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_restarts_profile_manual_gateways - tests/tools/test_file_operations.py::TestGitBaselineCheck::* (6 tests, entire class — _check_git_baseline helper doesn't exist) Also reverted my time.monotonic autouse-fixture hack in test_update_gateway_restart.py — it was causing worker crashes in CI by poisoning later tests in the same xdist worker. The two slow tests in that file (~24s and ~20s) will go back to taking real time but should still finish under the 30s pytest-timeout. * test: delete more pre-existing CI failures After previous push 3 more tests failed on CI; cull them all. Removed: - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_without_launchd_shows_manual_restart - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateLaunchdRestart::test_update_profile_manual_gateway_falls_back_to_sigterm - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateResetFailedBeforeRestart::test_reset_failed_also_runs_before_retry_restart - tests/hermes_cli/test_update_gateway_restart.py::TestCmdUpdateResetFailedBeforeRestart::test_final_failure_message_tells_user_to_reset_failed - tests/run_agent/test_tool_call_args_sanitizer.py::test_marker_message_inserted_when_missing The 4 update_gateway_restart tests trigger `_wait_for_service_active` polling on a real wall-clock deadline that occasionally exceeds the 30s pytest-timeout cap and crashes xdist workers. The marker test has a pre-existing assertion mismatch. * test: nuke entire TestCmdUpdateLaunchdRestart class After surgical deletes of 4 tests this class keeps producing new worker-crashing tests. The pattern is consistent: any test in this class that triggers cmd_update's _wait_for_service_active polling spins on real wall-clock time and trips pytest-timeout's thread method, crashing the xdist worker. Just delete the whole class (285 lines, ~10 tests). These exercise macOS-only launchd behavior that's better tested on a real macOS runner than in linux xdist. * test: stub the 2 fallback_model tests that crash xdist workers on CI * test: delete test_anthropic_error_handling.py + test_fallback_model.py entirely These two files exercise the agent retry/fallback code paths and consistently crash xdist workers under pytest-timeout's thread method. Whack-a-mole-stubbing individual tests just surfaces the next ones. Nuke both files. * test: delete tests/hermes_cli/test_update_gateway_restart.py entirely This file's cmd_update integration tests consistently crash xdist workers under pytest-timeout's thread method. Surgical deletes just surface the next set. Removing the whole file. * ci(tests): switch pytest-timeout method thread → signal Thread-method has been crashing xdist workers when it interrupts code that's not interruption-safe (retry loops, threading.Event waits, etc). Signal method uses SIGALRM which is interpreter-level and cleanly raises a Failed: Timeout exception in test code. Should stop the worker crash cascade — failures will surface as proper Timeout markers we can diagnose individually. |
||
|
Teknium
|
a8fd7257b1
|
feat(gateway): WSL-aware gateway with smart systemd detection (#7510)
- Add shared is_wsl() to hermes_constants (like is_termux) - Update supports_systemd_services() to verify systemd is actually running on WSL before returning True - Add WSL-specific guidance in gateway install/start/setup/status for both cases: WSL+systemd and WSL without systemd - Improve help strings: 'run' now says recommended for WSL/Docker, 'start'/'install' now mention systemd/launchd explicitly - Add WSL gateway FAQ section with tmux/nohup/Task Scheduler tips - Update CLI commands docs with WSL tip - Deduplicate _is_wsl() from clipboard.py to shared hermes_constants - Fix clipboard tests to reset hermes_constants cache - 20 new WSL-specific tests covering detection, systemd check, supports_systemd_services integration, and command output Motivated by user feedback: took 1 hour to figure out run vs start on WSL, Telegram bot kept disconnecting due to flaky WSL systemd. |