The per-profile pairing isolation added self._dir and scoped every
per-file path helper (_pending_path, _approved_path) to it, but
_all_platforms still enumerated the module-global PAIRING_DIR. For a
profile-scoped PairingStore, list_approved/list_pending/clear_pending
therefore operated on the GLOBAL platform set while loading each
platform's file from the PROFILE dir — so list_approved() returned []
for a user that is_approved() confirmed as approved, a silent divergence
between the authz surface and the list/inspect/clear surface.
Route discovery through self._dir. Byte-identical for the global store
(self._dir == PAIRING_DIR when no profile is set); only the buggy
profile-scoped case changes. self._dir is guaranteed to exist (__init__
mkdirs it).
Follow-up hardening on the request-id grant path.
approve_request took the same lockout treatment as approve_code: gated by
it, and recording a miss toward it. But the two paths defend different
things. The lockout exists to stop guessing at the 8-char code space over a
messaging channel; a request id is only ever obtained by an admin already
authenticated to the store, so a miss means the row they clicked went stale.
Counting those let a handful of clicks on a stale list lock the operator out
of `hermes pairing approve` for an hour — the GUI DoSing the CLI.
Also drops the `code`/`code_hash_prefix` compat fields from list_pending.
The hash prefix is what admin surfaces mistook for an approvable code in the
first place, and re-exporting the request id under the old `code` key just
preserves the ambiguity; both consumers in the tree read `request_id` now.
The 16-hex sniffing that had been copy-pasted into the CLI and the endpoint
(where a chained conditional consulted it against the wrong field) moves to
one owner, PairingStore.looks_like_request_id.
The endpoint no longer reports a 429 on the request-id path, where lockout
can't apply — a stale id surfaced as a bogus "locked out" while the platform
sat locked for something else entirely.
approve_code()'s success path never cleared _failures:{platform}. The
counter is incremented on every non-matching code, persisted in
_rate_limits.json, and only ever reset to 0 when it reaches
MAX_FAILED_ATTEMPTS (firing the lockout). So it counts failures over the
gateway's entire lifetime, not consecutive ones.
An owner who mistypes a pairing code on a handful of separate occasions
— each time immediately retyping it correctly and successfully pairing —
accumulates those isolated typos. A later single fresh typo then hits
MAX_FAILED_ATTEMPTS and locks the whole platform out for an hour, at
which point _is_locked_out gates approve_code and even the *correct*
code is rejected.
Reset the counter on a successful approval, matching standard
brute-force-guard semantics (the counter tracks consecutive failures).
This does not weaken protection: an attacker cannot produce a success
without a valid code, and 5 consecutive wrong attempts still lock out.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves the PR's conflict with main (2252 commits). Two conflicts, both
"each side added an independent block in the same place" — kept both:
- gateway/run.py — the housekeeping loop. This branch adds the Skill Sync
pulls inside the CURATOR_EVERY branch (12-space indent); main adds a
stale-session auto-archive as a sibling `if` at loop level (8-space).
Different scopes, so the naive union would have mis-nested the archive
block into the curator branch; kept each at its own indent level.
- tools/skill_manager_tool.py — the _edit_skill result dict. This branch
appends the org auto-propose note; main appends
_add_description_prompt_preview(). Independent, order-insensitive.
No behaviour dropped from either side.
Verified: 3552 passed / 0 failed across 63 suites (scope regenerated to
include main's new maybe_auto_archive / _add_description_prompt_preview
consumers) via scripts/run_tests.sh. `hermes sync` and `hermes sync status`
still work against a live token, resolving the production plane default.
The Pyright Optional-parameter warnings in skill_manager_tool.py are
pre-existing on main (`content: str = None` etc.), not introduced here.
GatewayRunner carried ~19 separate Dict[str, ...] attributes keyed by
session_key, each with an ad-hoc lifecycle. They now live in one
`self._sessions: dict[str, SessionState]` (gateway/session_state.py) with
three lifecycle scopes and a `_session_state(key)` get-or-create accessor.
Mechanical refactor: same state, same semantics, new container.
Migration table (dict -> old decl line -> current clear path -> new home):
| legacy dict | decl | cleared by (before) | SessionState field |
|------------------------------------------|-------|--------------------------------------------------|----------------------------------------|
| _running_agents | 3505 | _release_running_agent_state; stop() .clear() | turn.agent |
| _running_agents_ts | 3506 | same | turn.started_ts |
| _active_session_leases | 3507 | same (+ lease.release()) | turn.lease |
| _busy_ack_ts | 3539 | same | turn.busy_ack_ts |
| _turn_lease_tokens ((key, gen)-keyed) | 3518 | _release_turn_lease (generation-guarded) | turn.lease_token + turn.lease_generation |
| _session_model_overrides | 3585 | _CONVERSATION_SCOPED_STATE funnel | conversation.model_override |
| _pending_one_turn_model_restores | 3586 | funnel; one-shot pop in turn finally | conversation.one_turn_restore |
| _session_reasoning_overrides | 3589 | funnel; lazy-init dict swap ~5692 (RACE) | conversation.reasoning_override |
| _session_service_tier_overrides | 3592 | funnel; lazy-init dict swap ~5738 (RACE) | conversation.service_tier_override |
| _last_resolved_model ("*" = process-wide)| 3527 | funnel | conversation.last_resolved_model |
| _queued_events | 3537 | funnel; lazy-init dict swap ~5204 (RACE) | conversation.queued_events |
| _pending_turn_sidecar_notes | 3597 | funnel; lazy-init dict swap ~19832 (RACE) | conversation.sidecar_notes |
| _session_ephemeral_pin | 3601 | agent-cache evict pop; lazy swap ~19885 (RACE) | conversation.ephemeral_pin |
| _session_vc_last | 3604 | agent-cache evict pop; lazy swap ~19864 (RACE) | conversation.vc_last |
| _pending_approvals | 3611 | boundary security funnel; stop() .clear() | persistent.approvals |
| _update_prompt_pending | 3623 | security funnel; update watcher pops | persistent.update_prompt_pending |
| _pending_native_image_paths_by_session | 3538 | one-shot consume; lazy swap ~12944 (RACE) | persistent.native_image_paths |
| _pending_messages (runner-level, str) | 3519 | _interrupt_and_clear_session pop; stop() flush | persistent.pending_command_text |
| _session_run_generation | 3540 | NEVER (monotonic, #28686) | persistent.run_generation (never reset)|
Races eliminated: every `self._X = {}` lazy-init/reset replaced the WHOLE
dict, so a writer on session A racing a lazy init triggered by session B
could lose its entry. All six such sites (_session_reasoning_overrides
~5692, _session_service_tier_overrides ~5738, _queued_events ~5204,
_pending_turn_sidecar_notes ~19832, _session_ephemeral_pin ~19885,
_session_vc_last ~19864, _pending_native_image_paths_by_session ~12944,
_turn_lease_tokens ~13644) are now per-session field writes on an existing
SessionState; a reset can no longer cross sessions structurally.
Registry successors:
- _release_running_agent_state -> state.turn.clear() (one structured reset
instead of the drifting pop-list; still pops the slot lease and calls
lease.release() first; still generation-guarded).
- _CONVERSATION_SCOPED_STATE funnel -> state.conversation.clear(); the
tuple is retained for legacy plain-dict stores not yet folded in
(_pending_model_notes) and for the public test contract.
- _turn_lease_tokens' (key, generation) tuple key -> lease_token +
lease_generation fields; release/rebind only match when the generation is
current, preserving the #28686/#64934 ownership check.
- _session_run_generation stays monotonic on persistent.run_generation and
is never cleared (conversation boundaries and turn releases don't touch it).
Compatibility adapters: tests (and a few mixin call sites) access the old
dict names directly (137 direct assignments to _running_agents alone), so
each legacy name is kept as a thin @property returning a live MutableMapping
view over the corresponding SessionState field (legacy_dict_property /
legacy_lease_token_property in session_state.py). Setter accepts a plain
dict (the `runner._X = {...}` test pattern); views support ==, in, len,
.get/.pop/.clear. The shutdown path in _stop_impl deliberately keeps
duck-typed legacy-attribute access because test fakes borrow it with plain
dicts.
Name collision noted (NOT touched, out of scope): gateway/platforms/base.py
has its own _pending_messages Dict[str, MessageEvent] (adapter-level slot);
the runner-level Dict[str, str] of the same name is what moved to
persistent.pending_command_text.
Entry leaks preserved (follow-up, no new eviction in this PR): SessionState
entries in self._sessions are never evicted, matching the old dicts — e.g.
_last_resolved_model, _session_run_generation, _session_vc_last entries for
dead sessions leaked before and their fields still occupy a SessionState now.
Verification: 123 tests/gateway files referencing the old names +
_release_running_agent_state + _CONVERSATION_SCOPED_STATE all pass (sole
failure test_feishu.py::test_websocket_sdk_accepts_channel_ua_tag is
pre-existing, stash-verified); 8 non-gateway test files touching the names
pass (266 tests); `import gateway.run` subprocess smoke OK; ruff clean;
post-migration grep shows zero non-comment `self._<oldname>` references in
run.py outside the property adapters and the duck-typed shutdown block.
Both relay Slack knobs read their value through bool(), while the native
adapter they mirror uses str(raw).strip().lower() in {"1","true","yes","on"}.
A YAML-quoted string diverges:
dm_top_level_threads_as_sessions: "false" → relay True, native False
Non-empty strings are truthy, so the escape hatch is silently ignored in
exactly the shape an operator writes to switch it OFF. reply_in_thread has the
same defect and gates reply placement, session keying and run.py's progress
resolver, so one quoted "false" misfires three ways.
Route both through a shared _coerce_flag mirroring native's predicate. Real
booleans pass through untouched; None falls back to the default. Contract §8
documents the accepted spellings.
Tests: both knobs parametrized over the true/false spellings native accepts,
plus the absent-key default.
The DM thread-anchor contract was resolved only in send(). _send_media() —
backing send_image, send_image_file, send_voice, send_video and send_document
— passed reply_to straight to the frame and never touched metadata, so
attachments egressing through the same connector-side Slack sender got both
failure shapes this branch set out to remove:
flat mode → reply_to survives, the image threads UNDER the user's DM
message (the original reported symptom)
thread mode → no metadata.thread_id, and threadTs() never reads reply_to,
so the image lands in the home channel instead of the
per-message thread
Both are reachable: gateway/run.py delivers agent artifacts through
send_voice/send_document.
Extract the three steps that must always happen together (mode gate, mirrored
reply_to_message_id strip, metadata promotion) into
_apply_slack_thread_anchor and route BOTH lanes through it, so text and media
cannot drift again. The media lane copies caller metadata rather than mutating
it — these helpers are called in loops with a shared mapping.
Also fold send_typing/stop_typing's duplicated status-anchor blocks into
_with_status_thread_anchor. They had already drifted (stop_typing omitted the
platform check) and the clear must target the thread the heartbeat set or the
status line sticks until Slack's own timeout.
Tests: media lane pinned in both modes plus the channel and
no-caller-mutation cases; verified as real by reverting the fix and watching
them fail.
Four hot-path consumers paid a full config deepcopy per read:
- telemetry gate relay_shared_metrics.enabled() — runs 2-3x per agent
turn (2x per API call from lifecycle hooks + 1x per tool call) and
called read_raw_config(), which deepcopies the whole raw config every
call. New read_raw_config_readonly() serves the cached dict directly:
248 us -> 4.6 us per call (54x) on Teknium's real 77-key config.
- interruptible_streaming_api_call local-endpoint stale-timeout branch
called load_config() once per API call for every local-model user.
- gateway get_inbound_media_max_bytes() + _get_ephemeral_system_ttl_default()
called load_config() on per-message paths. All three switched to
load_config_readonly() (345 us -> 12 us; PR #28866 lineage).
Together these account for ~90% of the ~1,900 deepcopy primitives per
turn measured in the 26-call stubbed-LLM profile.
read_raw_config_readonly() keeps the (mtime_ns, size) freshness key so
config edits are picked up next call, and preserves the identity
invariant (cache-miss returns the same object later hits serve) —
regression-tested with 'is', per the PR #28866 identity-bug lesson.
The mutable read_raw_config() is unchanged for save-path callers.
581 targeted tests green (config, relay metrics x2, ephemeral reply,
platform base, new readonly suite).
- base.on_processing_complete implements the opt-in remove-ack/add-outcome
flow driven by _OK_EMOJI/_FAIL_EMOJI class attrs and the
_add_reaction(chat_id, message_id, emoji)/_remove_reaction(chat_id,
message_id) primitive shape; default stays a no-op.
- photon drops its override (exact behavioral match).
- slack/discord/feishu/matrix/telegram/google_chat keep overrides: divergent
primitive signatures (team_id routing, raw message objects, reaction-id
handles, replace-semantics setMessageReaction) or extra state protocols.
The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
The gateway's /reset cleanup path called shutdown_memory_provider without
first draining the memory manager's serialized background write worker.
shutdown_all only gives that worker a bounded (~5s) drain and abandons
whatever is still queued past it, so a /reset could silently drop writes
the session had already handed off -- the next session then loaded
stale MEMORY.md.
The CLI exit path already drains via MemoryManager.flush_pending before
shutdown; this PR pins the same contract on the gateway cleanup path.
Cleanup now calls agent._memory_manager.flush_pending(timeout=10) before
the existing shutdown_memory_provider step. The flush is best-effort:
a flush failure must never block teardown, so it is wrapped in
try/except and the existing shutdown path remains the fallback.
Closes#73297
- extract _commit_registry/_note_refresh_failure shared by the background
worker and foreground stage-4 (identical 4-step success + failure paths
were duplicated); worker now commits under _models_dev_fetch_lock so a
failing background refresh can never re-arm the backoff immediately
after a successful force_refresh committed (unsynchronized-write race)
- add should_clear_context_pin_async to hermes_cli/route_identity.py
(matching the get_model_context_length_async precedent) and use it at
the 4 async gateway sites instead of inline asyncio.to_thread wraps;
the sync _format_session_info site keeps the sync call (already
off-loop via its callers' to_thread)
- test the background-refresh success path (the PR's primary new
behavior): disk saved, mem cache swapped, backoff cleared, in_flight
reset — mutation-checked
- replace the race-prone spin-wait on _models_dev_refresh_in_flight with
a named-thread join in the backoff test
Follow-up for salvaged PRs #73400 + #73372:
- Move _preserve_agent_history_on_shutdown logic into
gateway/shutdown_flush.py as flush_agent_history_to_file()
- Reuse the hardened _write_payload (atomic writes, fsync, private
permissions, UUID filenames) from #73372 instead of the plain
open()/write() from #73400
- Replace os.environ.get('HERMES_HOME') with get_hermes_home() for
profile-safe path resolution
- Update tests to test the real function directly
- Net: removes 37 lines from gateway/run.py, unifies both fix paths
in a single module with consistent atomic-write guarantees
The previous attempt (#73171) snapshotted GatewayRunner._pending_messages,
which on current main has no writers (commit f6736ced8 removed its write
path; interrupt delivery uses adapter._pending_messages instead). The live
container is the per-agent agent._session_messages, flushed via
_flush_messages_to_session_db. When that flush raises (FTS/SQLite corruption,
the disk=0/memory=N state from #72680), the in-memory transcript is lost when
the process exits.
Retarget the preservation to the real path: in _finalize_shutdown_agents, wrap
the _flush call; on exception, dump agent._session_messages to an external JSON
recovery snapshot under $HERMES_HOME/shutdown-recovery/ (tagged issue=#72680)
so an operator can salvage it after repairing state.db. The dump is fully
guarded (non-fatal) so shutdown never blocks on a best-effort backup.
This directly addresses the reviewer note on #73171: retarget to the actual
cached-agent history (agent._session_messages) and prove a stale DB + shutdown
leaves a recoverable transcript.
Regression tests: tests/gateway/test_session_messages_shutdown_preserve.py
- flush raises -> recovery file written with session_id + messages
- healthy flush -> no recovery file
- write error -> non-fatal, no raise
Fixes#72680
Replace 20 lines of manual os.open/O_EXCL/fdopen/fsync/os.replace with the
existing atomic_json_write() from utils.py, which is already used by 6+
modules and handles temp-file creation, fsync, atomic replace, mode
control, and owner preservation. The only novel helper (_fsync_directory)
is retained — atomic_json_write does not do directory fsync.
Update test_flush_write_failure_leaves_no_recovery_file to monkeypatch
utils.os.replace (the new call path) instead of gateway.shutdown_flush.os.replace.
Same class of bug as the LINE adapter (NS-603): defaulting the webhook
bind to "0.0.0.0" (or hardcoding it) binds IPv4 ONLY, so the listener
is unreachable over IPv6-only private networks such as Fly.io 6PN.
- wecom callback_adapter: DEFAULT_HOST None; config.py env seed no
longer forces 0.0.0.0 when WECOM_CALLBACK_HOST is unset.
- msgraph_webhook: DEFAULT_HOST None; the allowed_source_cidrs
requirement still fires for the all-interfaces default (host=None is
treated as network-accessible).
- whatsapp_cloud: DEFAULT_WEBHOOK_HOST None.
- teams: hardcoded 0.0.0.0 TCPSite bind → _DEFAULT_HOST=None with new
TEAMS_HOST / extra.host override (mirrors LINE_HOST pattern).
- telegram: hardcoded listen="0.0.0.0" → default "" (tornado
bind_sockets opens one socket per address family; verified against
PTB 22.6/tornado) with new TELEGRAM_WEBHOOK_HOST / extra.webhook_host
override.
Explicit host overrides everywhere are preserved; empty/unset collapses
to the dual-stack default. "::" remains a bad substitute on
bindv6only=1 hosts (see LINE adapter comment).
The ownership guard let a sentinel with pid=None pass as self-owned, so
an exiting life could clobber evidence of unknown provenance with a
clean-exit claim. Tighten: only rewrite when the sentinel pid matches
os.getpid() exactly; pid=None (or malformed) is left untouched. Adds
tests for the pid=None no-op and the own-pid rewrite paths.
scripts/check-windows-footguns.py (blocking CI lint) rightly flagged the
os.kill(pid, 0) liveness probe: on Windows sig=0 collides with
CTRL_C_EVENT and GenerateConsoleCtrlEvent hard-kills the target's whole
console group (bpo-14484) — a forensics module must never be able to
kill the process it's checking on. Route through gateway.status._pid_exists,
the repo's canonical psutil-backed no-kill probe.
Hosted agents that die uncleanly (kernel OOM kill, SIGKILL, whole-VM
death) leave no trace: shutdown_forensics only covers graceful signals,
gateway-exit-diag.log only covers exit paths that actually run, and the
VM reboot wipes dmesg before anyone can capture it. NS-608 (BlueAtlas
hourly crash cycle, July 12-15) took days of manual log correlation to
classify because nothing recorded 'the previous life ended violently'.
Add gateway/lifecycle_ledger.py — a sentinel state machine persisted to
<HERMES_HOME>/state/gateway.lifecycle.json:
- start_gateway() claims the sentinel (phase=running) right after the
PID-file/runtime-lock claim, and reports any prior life that never
reached an exit path as gateway.previous_unclean_exit in
gateway-exit-diag.log + a WARNING log line.
- Every exit funnel marks the sentinel exited with a reason:
_exit_after_graceful_shutdown (graceful_shutdown), the shutdown
watchdog (shutdown_watchdog), and the loop-liveness watchdog
(loop_liveness_watchdog).
- Ownership-guarded for --replace takeovers: a live matching owner is
never reported dead, and the old life cannot clobber the
replacement's freshly claimed sentinel on its way out.
The 30s loop heartbeat now embeds a cheap /proc memory sample (own RSS,
MemAvailable, swap used) so every unclean-death report carries a
'memory N seconds before death' snapshot; the detector flags
suspected_oom when the last sample shows <64MiB or <5% available.
container-boot.log lines gain prior_exit=clean|unclean|unknown per
profile, stamping unclean container deaths into the volume-persisted
boot log where support can grep for them.
Tests: tests/gateway/test_lifecycle_ledger.py (16 cases) + 4 new
container-boot annotation cases. Existing watchdog/forensics/boot
suites all green; ruff clean.
Add an opt-in streaming-audio adapter seam to BasePlatformAdapter so
voice-capable gateway platforms (LiveKit, Discord voice, future adapters)
can consume LLM output as streaming PCM audio before the full response
completes, dropping perceived voice latency from ~2-3.5s to ~500-800ms.
Adapter contract (gateway/platforms/base.py):
- AudioFormat dataclass: declared sample_rate, channels, sample_width
- StreamingTTSHandle: opaque handle with audible/aborted flags
- supports_streaming_tts / begin_streaming_tts / write_streaming_tts
/ finish_streaming_tts / abort_streaming_tts
- All default to unsupported/no-op so existing adapters are source-compatible
- Per-turn _streaming_tts_completed_chats set suppresses duplicate whole-file
auto-TTS when streaming succeeded; cleared after turn completion
Gateway consumer (gateway/streaming_tts_consumer.py):
- StreamingTTSConsumer: bridges sync agent deltas to async adapter audio sink
- Uses existing SentenceChunker (no competing parser)
- Thread-safe bounded queue; on_delta never blocks the agent worker thread
- Resolves configured streaming provider via resolve_streaming_provider()
- Serialises clause playback in order; flushes tail on completion
- Pre-audio failure: completed=False (falls back to whole-file TTS)
- Post-audio failure: completed=True, partial=True (no replay from start)
- Abort is idempotent; late chunks silently dropped
- Per-turn state isolated across concurrent chats
Gateway integration (gateway/run.py):
- message_type parameter threaded through _run_agent -> _run_agent_inner
- StreamingTTSConsumer created when voice input + auto-TTS + provider active
- Delta callback teed to both text stream consumer and TTS consumer
- TTS-only delta callback installed when text streaming is off
- finish() called from executor; wait_complete() in async context after
- Barge-in aborts the consumer at all three interrupt detection points
- Runner-level _send_voice_reply suppressed when streaming TTS completed
Tests (tests/gateway/test_streaming_tts_consumer.py):
- 15 focused tests: adapter defaults, lifecycle, ordered chunks,
unsupported/No-streamer fallback, abort idempotency, late-chunk drop,
pre/post-audio failure, concurrent-turn isolation, think-block suppression,
queue backpressure
Does not touch desktop/TUI code or add config flags. Plugin TTS provider
stream() metadata gap (#47896) is explicitly out of scope — built-in
ElevenLabs/OpenAI PCM streamers are the first consumers.
Refs: #60671, #47896
Sibling of #64686: _run_agent's empty-final_response early return dropped
failure_reason (and #64686 only fixed the non-empty path), so downstream
consumers (TUI billing surface, transient-failure persistence) lost the
structured reason exactly when a failed run produced no text.
Also hardens the two BasePlatformAdapter identity checks (edit_message /
delete_message) with getattr so duck-typed adapters without the attribute
mean 'capability absent', not AttributeError — this was crashing the
send_progress_messages path for minimal adapters and test fakes.
Encapsulates the feature behind one command for launch, and adopts the
official product name.
One command:
- `propose` moves from `hermes skills propose` to `hermes sync propose`, so
the whole feature is one command to learn and one to document. Its handler
moves from cmd_skills to cmd_sync accordingly.
- The `hermes sync` parser now documents both halves plainly: personal sync
across your devices, and sharing with your organisation. Added an examples
epilog; rewrote the verb help in user language ("Include a skill in your
sync" rather than "Opt a skill into sync").
- Every user-facing string that pointed at `hermes skills propose` now points
at `hermes sync propose` (8 sites, including the agent-visible guidance
returned by skill_manage and the org provenance header).
This also clears the way for #39343, which adds its own top-level `sync` for
git-repo profile backup — that feature nests under `skills`, this one owns
`sync`.
Naming:
- HSP / "Hermes Sync Protocol" is gone from prose, docstrings, and comments.
The feature is "Skill Sync".
- Public identifiers renamed: HSPClient -> SyncClient, HSPError -> SyncError,
HSPConflict -> SyncConflict, hsp_address -> wire_address, HSP_VERSION ->
WIRE_VERSION.
- The WIRE names are deliberately NOT renamed: the `hsp_version` capability
field and the `x-hsp-object-type` response header are set by the deployed
gateway-gateway sync plane (verified in src/sync/syncRouter.ts), so
renaming them client-side would break sync against a live server. A comment
at the version constant records why they differ from the product name.
- The version-mismatch error is now actionable ("this server speaks sync
version X, but this Hermes speaks Y — update Hermes to sync with it")
instead of leaking the protocol acronym.
Also fixes a wiring gap found on the way: the gateway housekeeping tick
pulled personal skills but never org skills — the same defect already fixed
for the CLI. Org pull now runs there too, gated on real org membership.
Tests: the jargon guard now also fails on a bare "HSP". The two tests that
asserted the old cross-command structure are replaced by three asserting the
new one (propose IS under sync, propose is NOT under skills, sync usage
lists it). 2294 passed / 0 failed across all 51 suites that import the
changed modules, via scripts/run_tests.sh.
Verified by running the real CLI: `hermes sync --help` lists all eight verbs,
`hermes skills --help` no longer mentions propose, `hermes sync propose
--help` parses, and `hermes sync status` still reports live org state.
Follow-up to salvaged PR #56473: dedupe the five cleanup_*_cache bodies
into a shared _cleanup_cache_dir() helper, add cleanup_video_cache() and
cleanup_screenshot_cache() (with get_screenshot_cache_dir()), and drive
all five from a single (name, fn) loop in _start_gateway_housekeeping()
instead of one copy-pasted try/except per cache. Covers the video/
screenshot half of #56427.
