Fixes#73973.
A finite one-shot whose dispatch was claimed (claim_dispatch increments
repeat.completed BEFORE execution) but whose run died before mark_job_run
was left permanently wedged: completed==times, last_run_at null, state
'scheduled'. The run-claim TTL blocked re-dispatch, and once it expired
the dispatch-limit guard silently removed the job with no output and no
error.
Two complementary fixes:
- cron/scheduler.py run_one_job: the outer handler now catches
BaseException, not just Exception. The inner run_job handler re-raises
CancelledError/KeyboardInterrupt/SystemExit after agent teardown, and
none of those are Exception subclasses, so the outer 'except Exception'
missed them and mark_job_run(False) was never called. Failures are now
recorded first (mark_job_run + finish_execution, each independently
guarded), then non-Exception BaseExceptions are re-raised to preserve
teardown semantics. Plain Exceptions keep the existing behavior
(recorded, return False, no re-raise). Empty str(e) (bare
CancelledError) falls back to the exception class name.
- cron/jobs.py: when either removal site (claim_dispatch or the
get_due_jobs dispatch-limit guard) drops a one-shot whose claimed run
never completed (last_run_at null), _write_wedged_oneshot_diagnostic
now writes an operator-visible .md into cron/output/<job_id>/ instead
of vanishing silently. Best-effort: diagnostics can never break the
removal. No diagnostic when last_run_at is set (normal completion
race).
Issue #65773: run_one_job installs a <home>/.env secret scope around every
job; before c758ded6d (#69057) an installed scope was authoritative even
with multiplexing off, so provider keys injected only via the process
environment (container env vars, systemd Environment=) resolved to empty
inside cron and every provider call went out with the no-key-required
placeholder -> HTTP 401, while interactive turns kept working.
The fix landed in agent/secret_scope.py (scope-miss fallthrough to
os.environ when multiplex is off) with unit tests at that layer only.
These two tests pin the end-to-end contract where the bug actually
surfaced - cron's run_one_job:
- env-injected key resolves during run_job with multiplex OFF (fails on
pre-fix code, verified by mutation against c758ded6d~1)
- .env value still wins when both sources define the key (precedence)
Implementation-agnostic: passes whether the fix is the get_secret
fallthrough (main today) or a multiplex guard at the installation site
(the approach in #65801/#65802/#73037).
The gateway's /reset cleanup path called shutdown_memory_provider without
first draining the memory manager's serialized background write worker.
shutdown_all only gives that worker a bounded (~5s) drain and abandons
whatever is still queued past it, so a /reset could silently drop writes
the session had already handed off -- the next session then loaded
stale MEMORY.md.
The CLI exit path already drains via MemoryManager.flush_pending before
shutdown; this PR pins the same contract on the gateway cleanup path.
Cleanup now calls agent._memory_manager.flush_pending(timeout=10) before
the existing shutdown_memory_provider step. The flush is best-effort:
a flush failure must never block teardown, so it is wrapped in
try/except and the existing shutdown path remains the fallback.
Closes#73297
- extract _commit_registry/_note_refresh_failure shared by the background
worker and foreground stage-4 (identical 4-step success + failure paths
were duplicated); worker now commits under _models_dev_fetch_lock so a
failing background refresh can never re-arm the backoff immediately
after a successful force_refresh committed (unsynchronized-write race)
- add should_clear_context_pin_async to hermes_cli/route_identity.py
(matching the get_model_context_length_async precedent) and use it at
the 4 async gateway sites instead of inline asyncio.to_thread wraps;
the sync _format_session_info site keeps the sync call (already
off-loop via its callers' to_thread)
- test the background-refresh success path (the PR's primary new
behavior): disk saved, mem cache swapped, backoff cleared, in_flight
reset — mutation-checked
- replace the race-prone spin-wait on _models_dev_refresh_in_flight with
a named-thread join in the backoff test
The branched call shape in get_provider_info/get_provider is deliberate:
~69 test sites across tests/hermes_cli and tests/gateway monkeypatch
fetch_models_dev (and get_provider_info) with zero/single-arg lambdas.
Passing allow_network= unconditionally broke 5 tests in CI slices 2/3/7.
Documented the constraint inline.
- _mark_stale_cache_grace only moves cache_time forward so a completed
background refresh is never rewound to a 5-minute grace window
- clear _models_dev_refresh_in_flight if Thread.start() raises so a
one-off thread-exhaustion failure doesn't disable refresh forever
- move empty-registry validation into _fetch_models_dev_from_network
(was duplicated in the background worker and the foreground fetch)
- pass allow_network through as a plain kwarg in get_provider_info and
hermes_cli.providers.get_provider instead of the branched call shape
- refresh the stale module docstring (no bundled snapshot exists; the
resolution order now describes stale-serve + background refresh)
Follow-ups on top of @DonutsDelivery's salvaged reaper commit (#64141):
- create_from_config() now parses lsp.idle_timeout (invalid values fall
back to DEFAULT_IDLE_TIMEOUT) — previously the constructor knob was
unreachable from config.yaml (config exposure adapted from #36892 by
@0xbWy and #68091 by @9miya20)
- canonical default declared in hermes_cli DEFAULT_CONFIG so config
discovery surfaces the knob (per sweeper review note on #64980)
- reaper loop survives transient sweep errors instead of dying and
silently re-opening the leak (gap flagged in #68091 review)
- eventlog.log_reaped(): one INFO line per sweep + clears the
log_active announce cache so respawns re-announce at INFO
- docs: replace the stale 'no idle-timeout reaper' paragraph with the
new lifecycle description + config reference
- tests: reuse-refresh protection (the regression teknium's sweeper
requested on #64141), reaper-survives-error, config propagation,
invalid-value fallback, DEFAULT_CONFIG/manager-constant sync
memories/MEMORY.md is the "§"-delimited store written by MemoryStore, not a
markdown document. parse_existing_memory_entries() fell back to
extract_markdown_entries() -- the *source* parser for CLAUDE.md / AGENTS.md --
whenever the destination held no delimiter, which is exactly the case for a
single-entry store or one that was hand-edited or shell-appended. That
extractor skips fenced code blocks, skips table rows, splits a block into one
entry per bullet and reflows paragraphs. The shredded result was then written
straight back over the user's store and reported as "Imported", with no backup
to recover from.
Parse the destination the way MemoryStore._parse_entries does: split on
ENTRY_DELIMITER only, so a store with no delimiter is one intact entry.
extract_markdown_entries() is unchanged and still used on the sources, where
it is correct.
Also restore the safety net the port dropped. The openclaw migration script
this module was ported from calls maybe_backup(destination) before rewriting a
memory store; the port did not. Snapshot the store to <name>.bak.<unix_ts>
(same naming as MemoryStore._backup_drifted_file), refuse to rewrite when the
snapshot fails, and write via temp file + atomic rename so an interrupted
import cannot leave a truncated store and a symlinked MEMORY.md stays a
symlink.
The identical fallback lives in openclaw_to_hermes.py, where it is reached
from migrate_memory() (memories/MEMORY.md and memories/USER.md) and
migrate_daily_memory(); fixed there too.
Follow-up for salvaged PRs #73400 + #73372:
- Move _preserve_agent_history_on_shutdown logic into
gateway/shutdown_flush.py as flush_agent_history_to_file()
- Reuse the hardened _write_payload (atomic writes, fsync, private
permissions, UUID filenames) from #73372 instead of the plain
open()/write() from #73400
- Replace os.environ.get('HERMES_HOME') with get_hermes_home() for
profile-safe path resolution
- Update tests to test the real function directly
- Net: removes 37 lines from gateway/run.py, unifies both fix paths
in a single module with consistent atomic-write guarantees
The previous attempt (#73171) snapshotted GatewayRunner._pending_messages,
which on current main has no writers (commit f6736ced8 removed its write
path; interrupt delivery uses adapter._pending_messages instead). The live
container is the per-agent agent._session_messages, flushed via
_flush_messages_to_session_db. When that flush raises (FTS/SQLite corruption,
the disk=0/memory=N state from #72680), the in-memory transcript is lost when
the process exits.
Retarget the preservation to the real path: in _finalize_shutdown_agents, wrap
the _flush call; on exception, dump agent._session_messages to an external JSON
recovery snapshot under $HERMES_HOME/shutdown-recovery/ (tagged issue=#72680)
so an operator can salvage it after repairing state.db. The dump is fully
guarded (non-fatal) so shutdown never blocks on a best-effort backup.
This directly addresses the reviewer note on #73171: retarget to the actual
cached-agent history (agent._session_messages) and prove a stale DB + shutdown
leaves a recoverable transcript.
Regression tests: tests/gateway/test_session_messages_shutdown_preserve.py
- flush raises -> recovery file written with session_id + messages
- healthy flush -> no recovery file
- write error -> non-fatal, no raise
Fixes#72680
The old test asserted _warm_gateway_module was fire-and-forget (startup
completes in << SLOW_SECONDS). PR #73291 intentionally reversed this:
the import now runs synchronously before the lifespan yield because
run_in_executor didn't release the GIL on Windows + Python 3.11.
Updated the test to assert startup blocks for >= SLOW_SECONDS.
STT previously had no configuration surface outside hand-editing
config.yaml — no category in the hermes tools picker, no provider
matrix in the GUI capabilities tab, no status line in hermes setup.
- TOOL_CATEGORIES['stt']: 7 provider rows (Local Whisper, Nous
Subscription managed, OpenAI, Groq, xAI, ElevenLabs Scribe,
DeepInfra) with key prompts, badges, and post-setup hooks
- stt_provider marker wired through _write_provider_config,
_configure_provider, _reconfigure_provider, and
_is_provider_active — GUI and CLI share one write path
(apply_provider_selection)
- STT model picker (_configure_stt_model + STT_MODEL_CATALOG) runs
after provider pick: local sizes, Groq whisper family, OpenAI
whisper-1/gpt-4o-*/gpt-transcribe, ElevenLabs scribe (model_id key)
- faster_whisper post-setup hook auto-installs the local backend;
registered in _POST_SETUP_READY
- stt is CONFIG-ONLY (_CONFIG_ONLY_TOOLSETS): it ships no tool
schemas, so it is excluded from the per-platform enable checklist;
the GUI toolset toggle writes stt.enabled instead of
platform_toolsets
- hermes setup shows a Speech-to-Text status line per provider
- Mistral row omitted (mistralai PyPI quarantine), mirroring the
dashboard stt.provider options
Tests: tests/hermes_cli/test_stt_picker.py (20 cases) incl. invariant
checks against agent.transcription_registry builtins and the runtime
OPENAI_MODELS/GROQ_MODELS sets.
Follow-up on the #53205 salvage: replace bare is_file() probes of the
managed (~/.hermes/node[/bin]) and legacy (node_modules/.bin) locations
with shutil.which(..., path=dir) so Windows resolves the executable
.cmd shim instead of the extensionless POSIX script — the same miss
class fixed for _has_agent_browser() in #73932. Also covers the
Windows managed layout where the binary sits in node/ directly.
`hermes acp --setup-browser` installs agent-browser into the Hermes-managed
node prefix (~/.hermes/node/bin/agent-browser), which isn't necessarily on
PATH. doctor only checked PROJECT_ROOT/node_modules and PATH (shutil.which),
so it false-negatived with "agent-browser not installed" even though the
binary was present and runnable. Mirror dep_ensure._has_hermes_agent_browser()
by also checking HERMES_HOME/node/bin and the legacy
HERMES_HOME/node_modules/.bin path, each gated by agent_browser_runnable().
Tested with tests/hermes_cli/test_doctor.py (added positive + not-runnable
cases) and pytest tests/hermes_cli/test_doctor.py -q (66 passed).
The exact kwargs snapshot broke when VAD hardening added keys — the
test's real contract is 'null stt.local: must not crash or force
language/prompt'. Baseline kwargs are pinned by the dedicated suite.
Local faster-whisper called model.transcribe with bare {'beam_size': 5}:
no VAD, cross-window conditioning on, no confidence filtering. Pure
silence produced hallucinated tokens (E2E: 5s anullsrc WAV -> 'You',
no_speech_prob=0.705) and noisy clips could produce runs of junk, often
in other languages.
Three-layer class fix, one shared owner for every local-whisper call
site (build_local_transcribe_kwargs):
1. Silero VAD filter (bundled with faster-whisper) on by default —
silence never reaches the model. stt.local.vad: false restores the
raw behavior for music/ambient transcription.
stt.local.vad_min_silence_ms tunes chunk splitting (default 500).
2. condition_on_previous_text=False — one hallucinated token can no
longer seed a self-reinforcing run; negligible cost for
voice-note-length audio.
3. Segment confidence gate (_join_confident_segments): drop a segment
only when no_speech_prob > 0.6 AND avg_logprob < -1.0 (openai-whisper's
own heuristic shape; both must hit so quiet-but-real speech survives).
Config: stt.local.no_speech_prob_threshold / logprob_threshold.
The WHISPER_HALLUCINATIONS blocklist in voice_mode.py stays as
last-resort defense but should now almost never fire.
E2E (real faster-whisper 'base', CPU int8):
silence.wav before 'You' -> after ''
noise.wav before '' -> after ''
speech.wav before/after 'Hello World, this is a test of the
transcription system.' (unchanged)
Docs (EN + zh-Hans), DEFAULT_CONFIG, cli-config.yaml.example updated;
19 unit tests (kwargs contract, off-switch, confidence gate incl.
quiet-speech survival, _transcribe_local wiring), sabotage-verified.
Saying OR typing a configured stop phrase (voice.stop_phrases, default
"stop") now ends the voice chat everywhere, not just classic CLI PTT:
- hermes_cli/voice.py: new explicit on_stop_phrase callback through
start_continuous/stop_continuous. The force-transcribe path previously
DISCARDED the stop phrase silently — with auto_restart=False the client
re-arms the next capture, so the conversation never ended. Both halt
paths now fire on_stop_phrase (fallback: on_silent_limit for legacy
callers) as user intent, distinct from the no-speech timeout.
- tui_gateway/server.py: voice.record wires on_stop_phrase and emits
voice.transcript {stop_phrase: true} after flipping HERMES_VOICE(_TTS)
off and stopping streaming TTS — same teardown as /voice off. The TTS
barge-in monitor stop-checks its transcript too. prompt.submit consumes
a TYPED bare stop phrase at the server-side choke point when voice mode
is on (returns {voice_stopped: true}, no turn starts).
- ui-tui: voice.transcript {stop_phrase} ends voice mode with a clear
'voice chat ended' notice (distinct from the no-speech-limit message);
submitPrompt releases the busy latch on a consumed voice_stopped reply.
- cli.py: _typed_voice_stop in process_loop — typing a bare stop phrase
while voice mode/continuous is active ends voice mode instead of
sending 'stop' to the agent; typed 'stop' outside voice mode is
unchanged. Voice transcripts skip the check (already stop-checked).
- desktop: interceptsTypedVoiceStop — the composer's onSubmit ends the
live voice conversation (same path as clicking end on the pill) when a
bare stop command is typed with no attachments; renderer-owned loop, so
handled client-side like the existing spoken isVoiceStopCommand.
- tools/voice_mode.py: transcribe_recording never lets the Whisper
hallucination filter swallow a configured stop phrase (e.g. 'bye'
configured as a stop phrase is both a hallucination-blocklist entry and
a stop phrase — stop-phrase check now wins).
Tests: continuous-loop signal (sabotage-verified), force-transcribe stop
signal + legacy fallback, hallucination-filter ordering, typed-stop CLI
unit tests (voice on/off/longer text), prompt.submit typed-stop gateway
tests, TUI vitest for stop_phrase event handling, desktop vitest for the
typed-stop interceptor.
Streamed response text carried a 4-space _STREAM_PAD indent and the
final-response Rich Panel used padding=(1, 4), so every line selected
out of the terminal came with leading whitespace. Both now render
flush-left (pad empty, panel padding=(1, 0)); the table-realignment
width budgets were widened to match.
/copy now writes the ORIGINAL message text through native clipboard
tools (pbcopy / PowerShell Set-Clipboard via base64 / wl-copy / xclip /
xsel — same fallback chain as the TUI's writeClipboardText), falling
back to OSC 52 only when no native backend succeeds. This is the
TUI-equivalent answer to soft-wrap mangling: the clipboard gets the raw
text, not the rendered layout.
_has_agent_browser()'s new managed-Node rung calls
shutil.which('agent-browser', path=...); tests that monkeypatch
shutil.which globally with 1-arg lambdas raised TypeError when their
code path reached the browser readiness probe (test_post_setup_gating,
test_setup_model_provider).
Three GUI Capabilities-tab defects reported on Windows:
1. Browser rows stuck on 'Setup required' after a successful setup run.
Root causes, all in the readiness probe (not the installer):
- _has_agent_browser() never searched the Hermes-managed Node dir
(%LOCALAPPDATA%/hermes/node / $HERMES_HOME/node/bin) where the
Windows install lands, and probed node_modules/.bin/agent-browser
as the extensionless POSIX shim, which fails exec on Windows
(WinError 193) — now resolved via PATHEXT-aware shutil.which
against both rungs, mirroring _find_agent_browser().
- Cloud rows (Nous Subscription Browser Use, Browserbase, Browser
Use, Firecrawl) declared post_setup: agent_browser, whose
readiness gate requires a LOCAL Chromium build the cloud never
uses — switched to the cloud-scoped 'browserbase' hook (CLI-only).
- _agent_browser_installed() could read browser_tool's stale cached
'Chromium missing' result from before the install ran in the
spawned post-setup process — cache now dropped before probing so
the pill flips to Ready right after a successful run.
2. No way to tell which backend is active, and clicking a row to read
its details silently rewrote config. Row click now only
expands/collapses; activation is an explicit 'Use this backend'
button, the active row carries an 'Active' pill, and the expanded
active row says 'This is your active backend'.
3. OpenAI TTS showed one model and one voice. The options were always
defined but rendered through a native <datalist>, which filters by
the field's current value — a field already set to a valid option
suggested only itself. Replaced with a real combobox (Input +
dropdown) that lists every option, and voice suggestions now track
the selected model per the OpenAI TTS docs: tts-1/tts-1-hd = 9
voices, gpt-4o-mini-tts = 13 (adds ballad, verse, marin, cedar).
The Skills Hub 'Marketplace' tab showed a single useless entry: Anthropic
changed .claude-plugin/marketplace.json to bundle-shaped plugins whose
source is './', so all plugins collapsed to one identifier pointing at the
repo root, and the second marketplace repo (aiskillstore/marketplace) is
gone (404). Everything in anthropics/skills is already surfaced by the
GitHub tap as the Anthropic tab, making this source fully redundant.
Removes ClaudeMarketplaceSource and all wiring: source router, index
builder (crawl + floors + sort order + rate-limit messaging), extract
labels/install/URL mapping, hub UI tab, web server labels, CLI limits,
docs (en + zh), the legacy index-cache snapshot, and test fixtures.
Stale skills-index entries with source 'claude-marketplace' still install
fine: HermesIndexSource fetches via resolved GitHub paths generically.
Replace 20 lines of manual os.open/O_EXCL/fdopen/fsync/os.replace with the
existing atomic_json_write() from utils.py, which is already used by 6+
modules and handles temp-file creation, fsync, atomic replace, mode
control, and owner preservation. The only novel helper (_fsync_directory)
is retained — atomic_json_write does not do directory fsync.
Update test_flush_write_failure_leaves_no_recovery_file to monkeypatch
utils.os.replace (the new call path) instead of gateway.shutdown_flush.os.replace.
test_default_bind_serves_both_families asserts an IPv6 listening socket
exists, which is the environment's capability, not the adapter's — CI
runners with IPv6 disabled would flake. Probe an actual ::1 bind (not
just socket.has_ipv6, a compile-time constant) and skip cleanly when
the host has no usable IPv6 stack.
The LINE adapter's webhook server defaulted to host="0.0.0.0", which
binds IPv4 ONLY. On IPv6-only private networks — notably Fly.io 6PN,
where the hosted edge router reverse-proxies LINE ingest to
<app>.internal:8646 over an fdaa: IPv6 address — nothing is listening
on the dialed address: connection refused → customer-visible 502 when
LINE's console verifies the webhook (NS-603).
This is the same bug the generic webhook adapter fixed in d542894ad;
the LINE adapter was never updated to match. Fix mirrors that commit:
- DEFAULT_HOST = None → asyncio/aiohttp create_server binds one socket
per address family (v4 + v6), regardless of the bindv6only sysctl.
"::" is NOT a valid substitute — Fly machines set bindv6only=1, so
it would yield an IPv6-only socket and break IPv4 loopback probes.
- Empty-string host collapses to None; LINE_HOST/extra.host still pin
a specific bind address.
- reuse_address=False scoped to macOS only (BSD wildcard-socket
traffic-splitting footgun), mirroring 9420ad946.
- The three outbound-media guards compared webhook_host == "0.0.0.0"
by string equality; extracted to _missing_public_url() which treats
None/0.0.0.0/::/"" as "no fetchable hostname" so the LINE_PUBLIC_URL
requirement still fires under the new default. _media_url() falls
back to 127.0.0.1 instead of interpolating 'None' into URLs.
Tests: dual-stack default decision table (default None, empty→None,
pinned preserved, LINE_HOST override), behavioural both-families bind
proof via runner.addresses, and the media public-URL guard matrix.
88 passed, ruff clean.
Companion router fix (hermes-agent-router) routes /webhooks/line and
/line/webhook|/line/media to :8646 over 6PN; both are needed for
end-to-end hosted LINE delivery.
Fixes NS-603
show_status now reads get_nous_auth_status_local(); the test_status.py
mocks still patched the old live-resolve entry point, so the patched
dict was never consumed (CI slice 8/8 red on the salvage PR).
Follow-up widening of the /api/status fix: add get_nous_auth_status_local(),
a refresh-free auth-store snapshot (local invoke-JWT decode only), and use it
on the read-only display surfaces that previously called
get_nous_auth_status() -> resolve_nous_runtime_credentials() -> live OAuth
refresh POST:
- hermes_cli/status.py (hermes status auth-provider panel)
- hermes_cli/doctor.py (hermes doctor auth-provider checks)
- hermes_cli/portal_cli.py (hermes portal status display)
- hermes_cli/web_server.py /api/portal endpoint and the accounts-tab
provider card dispatcher (_resolve_provider_status nous branch)
Action paths (login flows, portal operations needing a live credential)
keep using get_nous_auth_status(). Part of NS-592.
resolve_sidecar_dir() probes the filesystem (touch/unlink) and can mirror
sidecar files to HERMES_HOME. Doing that as a module-import side effect
meant plugin discovery, `hermes --help`, and test collection all paid a
filesystem probe (and possibly a mirror copy) just for importing the
photon adapter or CLI.
Convert _SIDECAR_DIR/_NPM_ERROR_LOG in adapter.py and cli.py to lazy
cached accessors (_sidecar_dir()/_npm_error_log()); resolution now
happens on first actual use. Existing tests that monkeypatch the
_SIDECAR_DIR module global keep working — the accessors honor a
non-None value. Adds a regression test proving import performs no
resolution.
The Photon iMessage sidecar needs node_modules under
plugins/platforms/photon/sidecar/, but hosted/managed images keep the
whole install tree under an immutable /opt/hermes — every install and
self-heal path (setup CLI, stale-deps reinstall, cold install) died on
EROFS, and hosted users have no shell to work around it.
Three-layer fix, mirroring the WhatsApp bridge resolver pattern:
1. Bake the deps into the image. The Dockerfile now runs npm ci for the
sidecar in the layer-cached dependency stage (deterministic installs
from the committed lockfile; the postinstall spectrum-ts patch runs
at build time). Hosted happy path needs no runtime install at all.
2. New sidecar_paths.resolve_sidecar_dir() decides where the sidecar
runs from: PHOTON_SIDECAR_DIR override > writable source dir (dev
installs, unchanged) > read-only dir with baked fresh deps (managed
image) > mirror to $HERMES_HOME/photon/sidecar (writable data
volume) when deps are missing or stale in a read-only tree. The
mirror refreshes changed source files on image updates while
keeping node_modules, so the existing lockfile-staleness self-heal
works there.
3. connect() can now cold-install: _start_sidecar() runs the bounded
npm ci bootstrap when node_modules is missing instead of raising
immediately, and check_requirements() reports available when a
self-install is possible (npm present + writable resolved dir) so
the gateway actually creates the adapter on hosted instances. A
failed bootstrap still raises the actionable error, which connect()
surfaces as the retryable SIDECAR_FAILED fatal state on the
dashboard.
Tests: resolver decision table (env override, in-place, mirror,
refresh, fail-open), cold-install lifecycle paths, and a Dockerfile
contract test guarding the baked-deps + no-chown invariants.
Fixes NS-606.
The ownership guard let a sentinel with pid=None pass as self-owned, so
an exiting life could clobber evidence of unknown provenance with a
clean-exit claim. Tighten: only rewrite when the sentinel pid matches
os.getpid() exactly; pid=None (or malformed) is left untouched. Adds
tests for the pid=None no-op and the own-pid rewrite paths.
Hosted agents that die uncleanly (kernel OOM kill, SIGKILL, whole-VM
death) leave no trace: shutdown_forensics only covers graceful signals,
gateway-exit-diag.log only covers exit paths that actually run, and the
VM reboot wipes dmesg before anyone can capture it. NS-608 (BlueAtlas
hourly crash cycle, July 12-15) took days of manual log correlation to
classify because nothing recorded 'the previous life ended violently'.
Add gateway/lifecycle_ledger.py — a sentinel state machine persisted to
<HERMES_HOME>/state/gateway.lifecycle.json:
- start_gateway() claims the sentinel (phase=running) right after the
PID-file/runtime-lock claim, and reports any prior life that never
reached an exit path as gateway.previous_unclean_exit in
gateway-exit-diag.log + a WARNING log line.
- Every exit funnel marks the sentinel exited with a reason:
_exit_after_graceful_shutdown (graceful_shutdown), the shutdown
watchdog (shutdown_watchdog), and the loop-liveness watchdog
(loop_liveness_watchdog).
- Ownership-guarded for --replace takeovers: a live matching owner is
never reported dead, and the old life cannot clobber the
replacement's freshly claimed sentinel on its way out.
The 30s loop heartbeat now embeds a cheap /proc memory sample (own RSS,
MemAvailable, swap used) so every unclean-death report carries a
'memory N seconds before death' snapshot; the detector flags
suspected_oom when the last sample shows <64MiB or <5% available.
container-boot.log lines gain prior_exit=clean|unclean|unknown per
profile, stamping unclean container deaths into the volume-persisted
boot log where support can grep for them.
Tests: tests/gateway/test_lifecycle_ledger.py (16 cases) + 4 new
container-boot annotation cases. Existing watchdog/forensics/boot
suites all green; ruff clean.
Adds gpt-transcribe (OpenAI's new file-transcription model, $0.0045/min)
to the OpenAI STT provider:
- OPENAI_MODELS set: gpt-transcribe is recognized so provider
auto-correction keeps it on OpenAI and rejects it on Groq
- Language hint wiring: gpt-transcribe replaces the singular
'language' field with a 'languages' list; the API rejects the legacy
field, so the hint is sent via extra_body {languages: [..]}
- Config comment (DEFAULT_CONFIG), cli-config.yaml.example, desktop
settings enum, and docs (en + zh-Hans) updated
- Tests: model pass-through, languages-list hint shape, legacy singular
hint preserved for gpt-4o-transcribe, Groq auto-correction
gpt-live-transcribe (realtime WebSocket, $0.017/min) is NOT wired here:
the file-based STT pipeline has no realtime session path; it belongs in
a future realtime/voice-mode integration.
_install_dependencies now routes through tools.lazy_deps.install_specs
(NS-605); the force-reinstall test still stubbed
hermes_cli.tools_config._pip_install, so its spy list stayed empty
(CI slice 3/8 red on the salvage PR).
Widen NS-605 to the two remaining direct-install sites in the hindsight
plugin, which still shelled out to 'uv pip install --python
sys.executable' and therefore failed (EROFS/EACCES) on immutable hosted
images with sealed venvs, and lost packages on redeploy:
- post_setup dependency install (~L835): install_specs() with ok /
blocked-reason / stderr handling matching honcho/mem0.
- initialize()-time hindsight-client auto-upgrade (~L1240):
install_specs(); blocked installs log the gate reason with the manual
command instead of a raw subprocess error, and init proceeds.
Audited every other memory plugin (supermemory, byterover, holographic,
openviking, retaindb) for direct pip/uv install subprocess calls: none
remain — their deps flow through plugin.yaml pip_dependencies or
lazy_deps.ensure().
Tests: TestClientAutoUpgradeRoutesThroughLazyDeps — upgrade goes through
install_specs with the exact spec (regression guard asserts no
subprocess.run), blocked upgrade is non-fatal and surfaces the gate
reason. Updated TestPostSetupEnvEncoding stubs to the new install path.
Installing a memory provider (Honcho, mem0, hindsight, ...) from the
dashboard Plugins page failed on hosted deployments with a permission
error: the setup endpoint shelled out to
`uv pip install --python sys.executable`, which targets the sealed
read-only venv under /opt/hermes (immutable hosted image, NS-579/#49113).
The correct mechanism already exists: tools/lazy_deps.py redirects
installs to the writable durable target on the data volume
(HERMES_LAZY_INSTALL_TARGET=/opt/data/lazy-packages) when the venv is
sealed (HERMES_DISABLE_LAZY_INSTALLS=1), appends the target to the END
of sys.path (core venv always wins collisions), and constrains shared
deps to core-venv versions. The dashboard installer simply never used
it.
Fix:
- tools/lazy_deps.py: new public install_specs() — installs arbitrary
manifest-declared pip specs through the same environment routing as
ensure(): venv-scoped by default, durable-target on sealed images,
refused with an actionable reason when gated off (config kill switch
or sealed venv without a target — never surfaces raw EROFS/EACCES).
Specs are validated with _spec_is_safe(); post-install it invalidates
import/metadata caches so availability rechecks in the same process
see the new packages without a restart. Never raises.
- hermes_cli/web_server.py: _install_memory_provider_pip_dependencies
now calls install_specs() instead of building its own uv/pip
subprocess. Blocked installs surface the gate reason in the setup
results; the response's status block reflects post-install
availability (stale 'missing deps' state clears immediately).
- hermes_cli/memory_setup.py, plugins/memory/honcho/cli.py,
plugins/memory/mem0/_setup.py: CLI setup wizards routed through
install_specs() too — same sealed-venv failure mode, same fix.
No hosted setup path writes to /opt/hermes anymore; provider discovery
and installation now use the same environment (sys.path activation is
shared with the lazy-install bootstrap in hermes_bootstrap).
Tests:
- tests/tools/test_lazy_deps.py: TestInstallSpecs — gating matrix
(sealed+no-target blocked with immutable-deployment reason, config
kill switch, sealed+target proceeds), spec-safety rejection before
any subprocess, venv-scoped vs --target command display, failure
stderr passthrough, never-raises contract.
- tests/hermes_cli/test_web_server.py: setup endpoint routes pip
through lazy_deps (regression guard asserts no direct 'pip install'
subprocess), blocked-reason surfacing, same-response availability
recheck clears stale missing state.
Fixes NS-605 (Plain T-1111).
speak_text (hermes_cli/voice.py — the TUI/gateway one-shot TTS entry
point) now checks resolve_streaming_provider() first: when the
configured provider has a chunked streamer, the reply is spoken through
the same stream_tts_to_speaker pipeline CLI voice mode uses, so audio
starts on sentence one instead of after whole-file synthesis. No
streamer (edge/piper/etc.) or a streaming failure falls back to the
existing whole-file path unchanged — one dispatcher, zero parallel
streaming implementations.
Refs: #58930
Follow-up integration for the #47588 salvage, aligning the new streamers
with the post-campaign invariants:
- All streaming key lookups go through _resolve_key -> tts_tool.
_resolve_provider_key -> resolve_provider_secret (config > env/.env >
credential pool, profile-scoped) — never bare get_env_value. xAI
resolves via resolve_xai_http_credentials so OAuth users stream too.
- _capped(): every provider's chunk iterator is bounded at 16 MiB per
sentence, mirroring _read_tts_response_bytes' bounded-upstream-body
invariant on the sync paths.
- Tests updated for the resolver contract + new coverage for credential
routing and the cap.
Salvaged from PR #47588 and rebased onto the post-campaign streaming core:
the StreamingTTSProvider ABC/registry and the ElevenLabs/OpenAI streamers
already live on main (tools/tts_streaming.py), so this ports the pieces
main lacked:
- GeminiStreamer: streamGenerateContent?alt=sse -> base64 PCM chunks
(24 kHz mono int16), reusing main's DEFAULT_GEMINI_TTS_* constants.
- XAIStreamer: WebSocket wss://api.x.ai/v1/tts -> binary PCM frames,
async->sync bridged via the _collect_async test seam.
- tts.streaming.provider config knob: pin one streamer, or 'auto' to
walk the priority list elevenlabs -> gemini -> openai -> xai. Unset
keeps the never-swap-the-user's-voice default.
- docs/streaming-tts.md: architecture, capability matrix, how to add
a provider.
- Unit tests for the knob, SSE parsing, and the WS bridge; key-gated
E2E tests (skipped without credentials).
Refs: #47588
Add an opt-in streaming-audio adapter seam to BasePlatformAdapter so
voice-capable gateway platforms (LiveKit, Discord voice, future adapters)
can consume LLM output as streaming PCM audio before the full response
completes, dropping perceived voice latency from ~2-3.5s to ~500-800ms.
Adapter contract (gateway/platforms/base.py):
- AudioFormat dataclass: declared sample_rate, channels, sample_width
- StreamingTTSHandle: opaque handle with audible/aborted flags
- supports_streaming_tts / begin_streaming_tts / write_streaming_tts
/ finish_streaming_tts / abort_streaming_tts
- All default to unsupported/no-op so existing adapters are source-compatible
- Per-turn _streaming_tts_completed_chats set suppresses duplicate whole-file
auto-TTS when streaming succeeded; cleared after turn completion
Gateway consumer (gateway/streaming_tts_consumer.py):
- StreamingTTSConsumer: bridges sync agent deltas to async adapter audio sink
- Uses existing SentenceChunker (no competing parser)
- Thread-safe bounded queue; on_delta never blocks the agent worker thread
- Resolves configured streaming provider via resolve_streaming_provider()
- Serialises clause playback in order; flushes tail on completion
- Pre-audio failure: completed=False (falls back to whole-file TTS)
- Post-audio failure: completed=True, partial=True (no replay from start)
- Abort is idempotent; late chunks silently dropped
- Per-turn state isolated across concurrent chats
Gateway integration (gateway/run.py):
- message_type parameter threaded through _run_agent -> _run_agent_inner
- StreamingTTSConsumer created when voice input + auto-TTS + provider active
- Delta callback teed to both text stream consumer and TTS consumer
- TTS-only delta callback installed when text streaming is off
- finish() called from executor; wait_complete() in async context after
- Barge-in aborts the consumer at all three interrupt detection points
- Runner-level _send_voice_reply suppressed when streaming TTS completed
Tests (tests/gateway/test_streaming_tts_consumer.py):
- 15 focused tests: adapter defaults, lifecycle, ordered chunks,
unsupported/No-streamer fallback, abort idempotency, late-chunk drop,
pre/post-audio failure, concurrent-turn isolation, think-block suppression,
queue backpressure
Does not touch desktop/TUI code or add config flags. Plugin TTS provider
stream() metadata gap (#47896) is explicitly out of scope — built-in
ElevenLabs/OpenAI PCM streamers are the first consumers.
Refs: #60671, #47896
Sibling fix for #65977 — _model_flow_bedrock_api_key used only
get_env_value for AWS_BEARER_TOKEN_BEDROCK, missing pool-backed
keys. Now uses _resolve_api_key_provider_secret like the other
flows.
index.mjs now imports sibling .mjs helpers (send-format, stream-staleness);
the fixture copied only index.mjs so the sidecar died on module resolution
before reaching the health endpoint.
Maintainer rework of #45580 (issue #54036) on top of the contributor's
cherry-pick, which targeted spectrum-ts 3.1.0 while main pins 8.0.0:
Sidecar (primary detection, new):
- stream-staleness.mjs: pure decision rules, executable under node.
* classifyProbeRejection: only a not-found-shaped rejection of the
synthetic-id read counts as a completed round-trip (ALIVE); any other
rejection is INCONCLUSIVE — never alive. The original /probe treated
ANY rejection as alive, which was too loose.
* shouldProbe: probe only after 10+ min of stream silence (configurable
via PHOTON_STREAM_SILENCE_PROBE_MS; <=0 disables) with a cooldown.
* isZombieSuspect: zombie only on silence past threshold AND a
probe-proven live channel. Silence alone NEVER degrades (shared lines
can be quiet for hours); inconclusive probes NEVER degrade (network
may be down — the iterator will throw and the re-subscribe loop
recovers on its own).
- index.mjs: track last inbound-iterator yield (noteInboundYield), run a
30s watchdog tick, and on a confirmed zombie feed markStreamDegraded ->
the existing exit-75 restart path. /healthz gains a stream.staleness
block (silentForMs, threshold, lastProbeOutcome, zombieSuspected).
/probe reworked to strict semantics: 200 only on a proven round-trip,
503 with outcome hung|inconclusive otherwise.
Adapter (second layer, reworked):
- _probe_once returns tri-state alive|hung|inconclusive; only a hung
sidecar HTTP call counts toward the respawn counter — inconclusive
resets nothing and triggers nothing.
- default probe_interval_seconds 60 -> 600 (conservative; avoid restart
storms on quiet lines).
- _monitor_sidecar_health surfaces zombieSuspected from /healthz as a
warning; the fatal UPSTREAM_STREAM_DEGRADED path is unchanged and fires
when the sidecar escalates.
Tests: test_zombie_stream_watchdog.py executes the real node decision
module and drives the adapter against mocked /healthz responses;
test_presence_watchdog.py updated for the tri-state probe.
Also adds contributor mappings for nickkarhan (#53283) and vaibhavjnf
(#45580).
