Review follow-up on the salvaged #47772 work. Three defects made the
filter a no-op or actively harmful in production, plus both Copilot
review items.
1. The blank-line burst filter never fired. pty_bridge.py spawns via
ptyprocess.PtyProcess.spawn() and never calls setraw(), so the PTY
line discipline runs with ONLCR: every LF the child writes reaches
xterm as CRLF. The /\n{50,}/ pattern requires consecutive LF, so a
real 1000-row burst matched nothing (verified against a live PTY:
b"A"+b"\n"*5+b"B" is read back as b"A\r\n\r\n\r\n\r\n\r\nB").
Now matches /(?:\r?\n){50,}/.
2. Bursts split across WebSocket frames were not collapsed. bridge.read()
does os.read(fd, 65536) per drain tick and each read is forwarded as
its own frame, so a burst spans frames and each fragment fell under
the 50 threshold (3000 rows survived in a 40-byte-read simulation).
The sanitizer now holds back a trailing newline run — including a lone
trailing CR, since a frame can split a CRLF pair — and resolves it on
the next frame or on flush.
3. Erase-code stripping was permanent, not resume-scoped. resumeParam is
the durable session identity and is never cleared after connect, so
every spinner/progress/status redraw in a resumed session lost its
ESC[K and left stale glyphs. Suppression is now bounded to
PTY_RESUME_SANITIZE_WINDOW_MS (30s) after connect; burst collapsing
still applies for the life of the socket.
Copilot review items:
- flush() no longer writes a buffered partial CSI into xterm. #pending
only ever holds an incomplete sequence, and emitting one leaves the
parser in an in-escape state that swallows output after reconnect. A
buffered newline run is still emitted (collapsed).
- Test expectations updated accordingly.
Tests: 29 cases (was 18), now using CRLF fixtures that match real PTY
output, plus cross-frame burst reassembly, CRLF-pair frame splits, and
post-window erase preservation. Full web suite 135 passing.
Addresses Copilot review:
- Reuse a single TextDecoder instance instead of allocating per message
- Only filter erase codes during session resume (resumeParam != null)
- Extract filter chain into named helper sanitizeResumeOutput()
Refs #47313
Ink two-pass virtual scrolling during session resume floods the
PTY output with \x1b[K (erase-line), \x1b[NX (erase-char), and
thousand-line \n bursts. In the Dashboard INLINE mode, xterm
main scrollback buffer absorbs these as blank rows.
Filter all three in ws.onmessage before they reach xterm.
Refs #47313.
The upstream cua-driver installer scripts on trycua/cua@main carry a
baked default version that Release Please bumps in the release PR
*before* the release assets are published. During that window an
unpinned installer run 404s on the asset download and the
`hermes update` cua-driver refresh fails with:
error: download failed: The remote server returned an error: (404) Not Found.
⚠ cua-driver refreshing did not complete. Re-run manually: ...
Observed live 2026-07-29: baked version 0.14.0 vs latest published
release 0.13.1 — every `hermes update` run with an out-of-date driver
hit the warning until upstream publishes the assets.
We already know the correct version: `cua-driver check-update --json`
returns `latest_version` straight from the GitHub Releases API, whose
entries by definition have published assets. When the check positively
confirms an update, export that version as CUA_DRIVER_RS_VERSION into
the installer child env (both install.sh and install.ps1 honour it over
their baked default), so the refresh downloads the release that
actually exists instead of racing the upstream release pipeline.
Malformed / missing latest_version values fall back to the previous
unpinned behaviour. The explicit `hermes computer-use install
--upgrade` force path and fresh installs are unchanged.
The import-agent subcommand (24c3c27ba8) shipped with its parser builder
and handler logic but build_import_agent_parser() was never called from
main.py, making the documented and unit-tested command uninvocable.
Register it alongside the other subcommand builders.
Streamed responses no longer insert real newlines at terminal width —
logical lines are emitted whole and the terminal soft-wraps them, so
highlight-copy rejoins the full line (emulators only keep linebreaks
the app actually printed). This is the CLI equivalent of the TUI's
selection copy, which reads logical source lines from its screen
buffer. TTFT perception is preserved by mirroring the unfinished
line's tail into the spinner status text instead of chunk-printing.
/copy now prefers OSC 52 when running over SSH (SSH_CONNECTION /
SSH_TTY / SSH_CLIENT) — native tools there write the REMOTE clipboard,
which is never what the user wants. The CLI's OSC 52 writer also gains
tmux/screen DCS passthrough wrapping, mirroring the TUI's
wrapForMultiplexer. Fixes#31528 for the CLI surface.
Sabotage-verified: restoring the old chunk emitter fails 3 of the new
tests (hard-wrap detection, spinner mirror, unbreakable-run split).
Premise check on live main: barge-in machinery EXISTS for the per-turn
STREAMING pipeline only — cli.py chat() arms _voice_barge_in_monitor and
tui_gateway _tts_stream_begin arms _tts_stream_barge_in_monitor. What was
actually broken for spoken interruptions:
1. CLI whole-file fallback (_voice_speak_response_async — used whenever
streaming TTS cannot start: sounddevice missing, requirement probe
fails): NO monitor was ever armed, so talking over the reply did
nothing. Now arms _voice_barge_in_monitor in continuous voice mode.
2. Gateway fallback speak (tts_queue None → speak_text thread) and the
voice.tts RPC (desktop-triggered speech): speak_text ran bare, and
its internal streaming dispatch created a PRIVATE stop event nothing
could reach — uninterruptible even by stop_playback(). New
_speak_text_with_barge() runs the same barge monitor beside the speak
thread; hermes_cli.voice.speak_text/_speak_text_streaming accept an
external stop_event so a barge cuts the streaming pipeline too.
Stop-phrase handling and voice.transcript submission are inherited
from the shared monitor (merged #73933 behavior preserved).
3. False barge during TTS (the reason interruption "worked" then
self-cancelled or fired randomly): salvaged PR #71083 by @beardedeagle
(previous commit, kept authorship) — rolling-window VAD floor, 8x
multiplier, 4000-RMS trigger ceiling, barge_in_grace_seconds (2s)
before the mic opens, min-floor clamp. barge_in_grace_seconds is now
documented in DEFAULT_CONFIG.
Desktop spoken barge (renderer mic via voice-barge-in.ts) already covers
both its live-stream and fallback speech paths — verified, no change.
Long thinking/tool stretches in a voice conversation are dead air — the
user cannot tell whether the agent is alive. New: quiet, repeating soft
bubble blips while the agent works and no speech audio is flowing.
- tools/voice_mode.py: numpy-synthesized blips (no binary assets) — two
alternating low pitches (G4/E4) with pitch glide + smooth attack/decay
envelopes, ~0.8-1.2s randomized spacing, volume = voice.beep_volume * 0.5.
start_thinking_sound(should_play=...) / stop_thinking_sound() daemon-loop
lifecycle; macOS-TCC-safe (sounddevice output gated there → silent skip,
no per-second afplay churn). New mark_audio_output_active()/
is_audio_output_active() ref-count wraps play_audio_file and the
streaming OutputStream sentence writes so "audio is flowing" is accurate.
- Config: voice.thinking_sound (default true) off-switch.
- cli.py: starts when a voice-mode turn begins, per-blip gate skips while
TTS speaks / mic records / barge capture owns the mic; stopped in the
chat() finally on every exit path.
- tui_gateway/server.py: same lifecycle around _run_prompt_submit turns
(voice mode on), gated on is_audio_output_active + continuous capture.
- Desktop: renderer owns voice-conversation audio, so a matching WebAudio
implementation (src/lib/thinking-sound.ts, same envelope/pitches) runs
while conversation status === "thinking"; honors voice.thinking_sound
(via config store) and the shared sound-mute toggle; stops instantly on
speaking/listening/end.
One owner for the wording: voice_stop_hint() in tools/voice_mode.py —
sources the phrase from voice.stop_phrases (first entry) so a custom
phrase renders correctly, and returns "" when the feature is disabled
(stop_phrases: []) so no surface shows a hint.
- CLI: printed in /voice on output (style-matched dim notice).
- TUI: voice.toggle action=on now carries stop_hint; the Ink client
renders it in the "Voice mode enabled" block (older gateways omit
the field — no hint, no crash).
- Desktop: the renderer voice loop never touches tools/voice_mode.py,
so the phrase is read from config (voice.stop_phrases → $voiceStopPhrase
store, seeded in use-hermes-config) and shown as an info toast when a
voice conversation starts. i18n: en/ja/zh/zh-hant/ar.
The continuous-voice no-speech counter (3 strikes -> voice off) counted
every silent capture cycle unconditionally. During a long agent turn
(thinking/tool-calling for minutes) or while TTS is speaking, the user
is CORRECTLY silent — those cycles ended the voice chat under them.
- hermes_cli/voice.py: new set_voice_busy_probe() seam + _voice_activity_held()
(TTS-playing via the existing _tts_playing Event, agent-busy via the
registered probe). Both the continuous-loop strike path and the
force-transcribe single-shot strike path skip counting while held.
Fail-open: a broken probe counts cycles as before.
- tui_gateway/server.py: registers _any_session_running() as the probe
on voice.record start (voice is process-global; any running session holds).
- cli.py: classic CLI strike path skips counting while _agent_running
or TTS playback is in flight.
Stop phrase and barge-in still work during the hold (own paths).
Includes a fixture fix for the #71083 cherry-pick: the fake tools.tts_tool
module needs _load_tts_config (main's tts_streaming imports it).
Replace one-shot VAD calibration with a rolling deque window that
continuously recalibrates the noise floor throughout TTS playback,
preventing false barge-in triggers from stale calibration. Add a
grace period before VAD activates so TTS playback establishes first.
Suppress duplicate text rendering when token streaming is enabled.
Mirror the barge-in and TTS stream stop logic to the TUI gateway path
so both CLI and gateway use the same VAD semantics.
Rolling-window VAD:
- 90th percentile of rolling window (~3s) for noise floor
- 8x multiplier (was 5x) for TTS volume variation headroom
- 4000 RMS trigger ceiling so genuine speech can still trip
- min_floor clamped to SILENCE_RMS_THRESHOLD * 2
- sustained_ms=1000, calibration_ms=800
Barge-in grace period (barge_in_grace_seconds, default 2.0s):
Delays VAD activation so TTS playback establishes before the mic opens.
Duplicate render suppression:
When streaming_enabled, pass display_callback=None to
stream_tts_to_speaker so the token stream is the sole display path.
TUI gateway mirror:
Mirror _tts_stream_stop and _tts_stream_barge_in_monitor changes to
tui_gateway/server.py so both code paths use the same VAD parameters,
grace period, and TTS CUT diagnostic logging.
Profile-scoped session DB and MoA progress events in tui_gateway/server.py
were necessitated by the TTS pipeline changes affecting session state
and event routing.
Normal-exit flag and TTS CUT diagnostic logging at all cut paths.
Regression tests:
- test_quiet_then_loud_playback_does_not_trip
- test_8x_multiplier_absorbs_tts_volume_spikes
- test_trigger_ceiling_lets_genuine_speech_trip
- test_silence_calibration_does_not_false_trip_on_tts
- test_tts_stream_stop_latches_interruption_for_next_turn
- test_tts_stream_stop_after_natural_finish_does_not_latch
- Profile-scoped session DB tests (10 tests)
Fixes#73973.
A finite one-shot whose dispatch was claimed (claim_dispatch increments
repeat.completed BEFORE execution) but whose run died before mark_job_run
was left permanently wedged: completed==times, last_run_at null, state
'scheduled'. The run-claim TTL blocked re-dispatch, and once it expired
the dispatch-limit guard silently removed the job with no output and no
error.
Two complementary fixes:
- cron/scheduler.py run_one_job: the outer handler now catches
BaseException, not just Exception. The inner run_job handler re-raises
CancelledError/KeyboardInterrupt/SystemExit after agent teardown, and
none of those are Exception subclasses, so the outer 'except Exception'
missed them and mark_job_run(False) was never called. Failures are now
recorded first (mark_job_run + finish_execution, each independently
guarded), then non-Exception BaseExceptions are re-raised to preserve
teardown semantics. Plain Exceptions keep the existing behavior
(recorded, return False, no re-raise). Empty str(e) (bare
CancelledError) falls back to the exception class name.
- cron/jobs.py: when either removal site (claim_dispatch or the
get_due_jobs dispatch-limit guard) drops a one-shot whose claimed run
never completed (last_run_at null), _write_wedged_oneshot_diagnostic
now writes an operator-visible .md into cron/output/<job_id>/ instead
of vanishing silently. Best-effort: diagnostics can never break the
removal. No diagnostic when last_run_at is set (normal completion
race).
Issue #65773: run_one_job installs a <home>/.env secret scope around every
job; before c758ded6d (#69057) an installed scope was authoritative even
with multiplexing off, so provider keys injected only via the process
environment (container env vars, systemd Environment=) resolved to empty
inside cron and every provider call went out with the no-key-required
placeholder -> HTTP 401, while interactive turns kept working.
The fix landed in agent/secret_scope.py (scope-miss fallthrough to
os.environ when multiplex is off) with unit tests at that layer only.
These two tests pin the end-to-end contract where the bug actually
surfaced - cron's run_one_job:
- env-injected key resolves during run_job with multiplex OFF (fails on
pre-fix code, verified by mutation against c758ded6d~1)
- .env value still wins when both sources define the key (precedence)
Implementation-agnostic: passes whether the fix is the get_secret
fallthrough (main today) or a multiplex guard at the installation site
(the approach in #65801/#65802/#73037).
The gateway's /reset cleanup path called shutdown_memory_provider without
first draining the memory manager's serialized background write worker.
shutdown_all only gives that worker a bounded (~5s) drain and abandons
whatever is still queued past it, so a /reset could silently drop writes
the session had already handed off -- the next session then loaded
stale MEMORY.md.
The CLI exit path already drains via MemoryManager.flush_pending before
shutdown; this PR pins the same contract on the gateway cleanup path.
Cleanup now calls agent._memory_manager.flush_pending(timeout=10) before
the existing shutdown_memory_provider step. The flush is best-effort:
a flush failure must never block teardown, so it is wrapped in
try/except and the existing shutdown path remains the fallback.
Closes#73297
- extract _commit_registry/_note_refresh_failure shared by the background
worker and foreground stage-4 (identical 4-step success + failure paths
were duplicated); worker now commits under _models_dev_fetch_lock so a
failing background refresh can never re-arm the backoff immediately
after a successful force_refresh committed (unsynchronized-write race)
- add should_clear_context_pin_async to hermes_cli/route_identity.py
(matching the get_model_context_length_async precedent) and use it at
the 4 async gateway sites instead of inline asyncio.to_thread wraps;
the sync _format_session_info site keeps the sync call (already
off-loop via its callers' to_thread)
- test the background-refresh success path (the PR's primary new
behavior): disk saved, mem cache swapped, backoff cleared, in_flight
reset — mutation-checked
- replace the race-prone spin-wait on _models_dev_refresh_in_flight with
a named-thread join in the backoff test
The branched call shape in get_provider_info/get_provider is deliberate:
~69 test sites across tests/hermes_cli and tests/gateway monkeypatch
fetch_models_dev (and get_provider_info) with zero/single-arg lambdas.
Passing allow_network= unconditionally broke 5 tests in CI slices 2/3/7.
Documented the constraint inline.
- _mark_stale_cache_grace only moves cache_time forward so a completed
background refresh is never rewound to a 5-minute grace window
- clear _models_dev_refresh_in_flight if Thread.start() raises so a
one-off thread-exhaustion failure doesn't disable refresh forever
- move empty-registry validation into _fetch_models_dev_from_network
(was duplicated in the background worker and the foreground fetch)
- pass allow_network through as a plain kwarg in get_provider_info and
hermes_cli.providers.get_provider instead of the branched call shape
- refresh the stale module docstring (no bundled snapshot exists; the
resolution order now describes stale-serve + background refresh)
Follow-ups on top of @DonutsDelivery's salvaged reaper commit (#64141):
- create_from_config() now parses lsp.idle_timeout (invalid values fall
back to DEFAULT_IDLE_TIMEOUT) — previously the constructor knob was
unreachable from config.yaml (config exposure adapted from #36892 by
@0xbWy and #68091 by @9miya20)
- canonical default declared in hermes_cli DEFAULT_CONFIG so config
discovery surfaces the knob (per sweeper review note on #64980)
- reaper loop survives transient sweep errors instead of dying and
silently re-opening the leak (gap flagged in #68091 review)
- eventlog.log_reaped(): one INFO line per sweep + clears the
log_active announce cache so respawns re-announce at INFO
- docs: replace the stale 'no idle-timeout reaper' paragraph with the
new lifecycle description + config reference
- tests: reuse-refresh protection (the regression teknium's sweeper
requested on #64141), reaper-survives-error, config propagation,
invalid-value fallback, DEFAULT_CONFIG/manager-constant sync
Two residual gaps from the #72707/#72632 probe-hardening series:
- backendSupportsServe never forwarded backend.shell to the serve
--help probe. A .cmd/.bat shim backend (which carries shell: true in
its step-4 descriptor) makes execFileSync throw EINVAL on modern
Node; the bare catch then caches supported=false for the process
lifetime, permanently routing that backend through the legacy
dashboard form. Forward the flag.
- The Windows python-discovery probes ran with no timeout at all:
reg query (registry read) and py.exe -c 'import sys;...' (bare
interpreter startup) are both synchronous execs on the boot path;
a wedged reg.exe or python.exe would hang the resolver forever.
Bound reg query at 5s and the py.exe probe at the shared
PROBE_TIMEOUT_MS budget.
Follow-up to #73907 (probe timeout 15s + env override + timeout-only
retry), widening the same fix to the two sibling sites it missed:
- backendSupportsServe's serve --help probe kept a bare execFileSync
with its own 15s literal: same cold-Windows Python-startup class
(#72632 measured ~10.5s for --version cold), no retry, and a false
negative is cached for the process lifetime - silently routing a
modern runtime through the legacy dashboard form. Route it through
execProbeSync with the shared PROBE_TIMEOUT_MS (honours
HERMES_PROBE_TIMEOUT_MS) and the timeout-only retry.
- resolveHermesBackend step 4 called unwrapWindowsVenvHermesCommand
twice; the second call re-ran the same un-memoized import probe,
costing up to another full probe timeout on a hung interpreter for
an answer the first call already gave. Drop the redundant re-probe.
Part of the #72707 bug class (transient disconnect must not strand a
healthy install).
Deduplicates the mkstemp→fsync→atomic_replace pattern that existed in
three places: agent_import.py (added by #72983), MemoryStore._write_file,
and skill_manager_tool._atomic_write_text. All three now call a single
utils.atomic_write_text helper.
Also wraps the atomic_write_text call in _merge_memory_entries with
try/except OSError so a write failure records a per-item error instead
of propagating uncaught and aborting the entire import with no record.
Follow-up to #72983.
memories/MEMORY.md is the "§"-delimited store written by MemoryStore, not a
markdown document. parse_existing_memory_entries() fell back to
extract_markdown_entries() -- the *source* parser for CLAUDE.md / AGENTS.md --
whenever the destination held no delimiter, which is exactly the case for a
single-entry store or one that was hand-edited or shell-appended. That
extractor skips fenced code blocks, skips table rows, splits a block into one
entry per bullet and reflows paragraphs. The shredded result was then written
straight back over the user's store and reported as "Imported", with no backup
to recover from.
Parse the destination the way MemoryStore._parse_entries does: split on
ENTRY_DELIMITER only, so a store with no delimiter is one intact entry.
extract_markdown_entries() is unchanged and still used on the sources, where
it is correct.
Also restore the safety net the port dropped. The openclaw migration script
this module was ported from calls maybe_backup(destination) before rewriting a
memory store; the port did not. Snapshot the store to <name>.bak.<unix_ts>
(same naming as MemoryStore._backup_drifted_file), refuse to rewrite when the
snapshot fails, and write via temp file + atomic rename so an interrupted
import cannot leave a truncated store and a symlinked MEMORY.md stays a
symlink.
The identical fallback lives in openclaw_to_hermes.py, where it is reached
from migrate_memory() (memories/MEMORY.md and memories/USER.md) and
migrate_daily_memory(); fixed there too.
Follow-up for salvaged PRs #73400 + #73372:
- Move _preserve_agent_history_on_shutdown logic into
gateway/shutdown_flush.py as flush_agent_history_to_file()
- Reuse the hardened _write_payload (atomic writes, fsync, private
permissions, UUID filenames) from #73372 instead of the plain
open()/write() from #73400
- Replace os.environ.get('HERMES_HOME') with get_hermes_home() for
profile-safe path resolution
- Update tests to test the real function directly
- Net: removes 37 lines from gateway/run.py, unifies both fix paths
in a single module with consistent atomic-write guarantees
The previous attempt (#73171) snapshotted GatewayRunner._pending_messages,
which on current main has no writers (commit f6736ced8 removed its write
path; interrupt delivery uses adapter._pending_messages instead). The live
container is the per-agent agent._session_messages, flushed via
_flush_messages_to_session_db. When that flush raises (FTS/SQLite corruption,
the disk=0/memory=N state from #72680), the in-memory transcript is lost when
the process exits.
Retarget the preservation to the real path: in _finalize_shutdown_agents, wrap
the _flush call; on exception, dump agent._session_messages to an external JSON
recovery snapshot under $HERMES_HOME/shutdown-recovery/ (tagged issue=#72680)
so an operator can salvage it after repairing state.db. The dump is fully
guarded (non-fatal) so shutdown never blocks on a best-effort backup.
This directly addresses the reviewer note on #73171: retarget to the actual
cached-agent history (agent._session_messages) and prove a stale DB + shutdown
leaves a recoverable transcript.
Regression tests: tests/gateway/test_session_messages_shutdown_preserve.py
- flush raises -> recovery file written with session_id + messages
- healthy flush -> no recovery file
- write error -> non-fatal, no raise
Fixes#72680
The old test asserted _warm_gateway_module was fire-and-forget (startup
completes in << SLOW_SECONDS). PR #73291 intentionally reversed this:
the import now runs synchronously before the lifespan yield because
run_in_executor didn't release the GIL on Windows + Python 3.11.
Updated the test to assert startup blocks for >= SLOW_SECONDS.
On Windows + Python 3.11 the gateway import triggers heavy .pyc
compilation and Defender real-time scans that do not release the GIL.
Running in run_in_executor still froze the event loop for 15-22 s,
causing the Desktop's 10-second WebSocket ready-probe to time out.
Move the call from the executor to a synchronous invocation before the
lifespan yield, so the GIL block is absorbed during backend
initialisation — before the server socket accepts probes.
Fixes#73083
