- Cut the leftover telemetry.* DEFAULT_CONFIG block (nothing reads it) and
the legacy telemetry-key fallback in policy.py.
- install_id: persist the minted UUID back to config.yaml on first use so
service.instance.id survives gateway restarts (fail-open when the write
is not possible); regression test covers the restart path.
- Remove the no-op gateway_health_export.redaction config keys. Redaction
is always-on by design and deliberately not configurable; status output
now says so.
- Collapse redaction to one unconditional secrets+PII scrub: drop the
none/pii content modes (they served the dropped trajectories plane) and
fold gateway_health.py's duplicate bearer/token/email/phone regex layer
into agent/monitoring/redaction.py.
Salvages the event-spine foundation from feat/telemetry-observability
(emitter, typed events, OTLP streaming, redaction — authorship preserved in
the preceding commits) and scopes it to the plane enterprise operators need
today: gateway Service Health Monitoring plus redacted Operational
Diagnostics, exported over OTLP.
Dropped from the salvaged branch, deliberately:
- run/model/tool trajectory capture (plugins/telemetry hooks, tel_spans)
- the local JSONL + state.db tel_* store (monitoring is egress, not storage)
- usage rollups/metrics, /insights integration, bulk export
- hermes telemetry CLI (replaced by hermes monitoring status)
Those planes — shared client usage metrics and enterprise trace telemetry —
are being designed on the NeMo Relay integration with distinct consent,
policy, and export boundaries; this keeps the monitoring plane content-free
and independently enableable.
Renames agent/telemetry -> agent/monitoring, config telemetry.* ->
monitoring.*, and pins the otlp extra at OpenTelemetry 1.39.1 (matching
uv.lock; 1.30.0 conflicts with mistralai>=2.4 on opentelemetry-api).
Self-review after the #51714 feedback found the reviewer's dead-table finding
was not isolated — the schema advertised far more than the code populates, and
our own tests hid it by hand-feeding fields production never sends. Make the
surface honest by subtraction.
Schema (10 tel_* tables -> 5):
- Delete tel_gateway_events, tel_cron_events, tel_skill_events,
tel_memory_events, tel_feedback_events — declared, never written, never read.
- Drop columns nothing populates: tel_runs.{profile_id,estimated_cost_usd,
cost_status}; tel_model_calls.{ttft_ms,estimated_cost_usd,cost_status,
cost_source,end_reason,retry_count}; tel_tool_calls.{backend,retry_count,
approval}; tel_spans.attrs_json. Cost duplicated the existing sessions
billing columns and was always NULL here.
- events.py / emitter _TABLE_COLUMNS / OTLP _span_attrs / rollup / preview
display all trimmed to match.
Correctness:
- end_reason no longer hardcodes "completed". Production finalize callers pass
`reason` (shutdown/session_expired/session_reset); _coarse_end_reason now
reads it and maps accordingly.
- Fix a latent bug the trim exposed: the model_call hook passed end_reason= to
ModelCallEvent, which the @_safe wrapper was silently swallowing — so
tel_model_calls dropped every row in real runs. Now writes correctly.
Tests:
- Stop hand-feeding estimated_cost_usd / turn_exit_reason that no production
call site sends. Finalize is now driven with the real `reason` kwarg, and
assertions cover only fields that are actually populated. This is what let
the model_call drop hide — the suite graded on a fictional contract.
Net: a smaller system that does what it says. Verified end-to-end over the real
dispatch path (runs + connected span tree + model/tool rows populate; dead
tables gone). 160 telemetry/state/insights tests green.
Addresses the review on #51714: the trace/span layer was declared but unwired —
tel_spans was never written, call rows had no timestamp, and nothing set parent
lineage, so the store was metrics-only and couldn't reconstruct a trace.
Wire the span layer (keeping the praised star-schema shape):
- New SpanEvent (span_id/trace_id/run_id/parent_span_id/name/kind/start_ns/end_ns)
mapped into tel_spans via the emitter's _TABLE_COLUMNS.
- The plugin mints a root span per run and, on each model/tool call, emits a
SpanEvent (timing + parent = the run's root) keyed by the SAME span_id as the
detail row, so tel_model_calls / tel_tool_calls JOIN to their span.
- Call hooks fire on completion, so end_ns = now and start_ns is reconstructed
from the measured latency/duration. The run's root span is emitted at finalize
with the true run start/end.
Result: tel_spans is a connected, single-trace_id, run -> calls tree a desktop
waterfall (or any reader) can render directly, ordered by start_ns. Existing
metrics rows (tel_runs/model_calls/tool_calls) are unchanged.
OTLP: spans now flow to the exporter with their trace/parent/timing attributes.
The exporter still emits one OTel span per event rather than reconstructing OTel
SpanContexts into a connected trace tree; that projection is left for a follow-up
and the module docstring now says so plainly instead of over-claiming.
Adds test_spans_trace.py (connected-tree + detail-row JOIN) over the real dispatch
path. Accurate (pre-hook) start times, real OTLP SpanContexts, and subagent
cross-run lineage remain follow-ups.
Aggregate metrics are derived from the local tel_* tables — they're a coarsened
view of local data, not an independent capture path. With telemetry.local=false
nothing is written, so an aggregate opt-in had nothing to aggregate, yet
may_upload_aggregate() returned True and `status` showed "Aggregate metrics: on".
The config could claim a state it couldn't fulfill.
Gate aggregate on local being enabled:
- may_upload_aggregate() now requires local_enabled AND allow_aggregate AND
consent_state == aggregate.
- `telemetry status` computes aggregate_enabled the same way and, when consent is
aggregate but local is off, prints "inert: local telemetry is off — nothing to
aggregate" instead of the opt-in hint.
Happy path is unchanged (local on + consent aggregate -> on). Adds policy and CLI
tests for the inert combo.
The existing hook tests call the plugin's _on_* callbacks directly, which passes
even if the bundled plugin stops auto-loading or a hook name drifts from what core
fires — real runs would go dark while the suite stays green.
Add test_plugin_e2e.py, which drives the real dispatch chain through public entry
points only (discover_plugins -> invoke_hook -> registered callback -> emitter ->
tel_* tables), exactly as core does:
- one completed turn produces tel_runs / tel_model_calls / tel_tool_calls rows
with real provider/model/tool values and correct counts;
- telemetry.local=false means the plugin does not load and nothing is written.
Verified robust against test ordering (singleton resets for the plugin manager and
the emitter in the fixture).
Rename the telemetry tiers away from the borrowed control-plane/data-plane
jargon to plain language, across code, CLI output, config, and docs:
- "local plane" -> "local telemetry"
- "aggregate plane" -> "aggregate metrics"
- "trajectories plane" -> "trajectories" / "telemetry.trajectories"
- "three planes with a hard wall" -> "three settings, isolated from each other"
User-facing `hermes telemetry status` now reads "Local telemetry: on" /
"Aggregate metrics: off" / "Content export: off (trajectories disabled)".
The OTLP resource attribute key telemetry.plane is renamed to telemetry.scope
(wire-level identifier; nothing consumes it yet).
No behavior change — wording only. Status renders identically apart from the
labels; tests updated to match the new strings.
policy.resolve() / TelemetryDecision was a read-only projection used only by
`hermes telemetry status` for display. The actual behavior gates already read
telemetry.* straight from config: the emitter (whether to write) and the plugin
loader (whether to auto-load) each call .get("local", True) on the loaded config,
never through policy.
Make config the single chokepoint the status command reads too: it now resolves
local/allow_aggregate/consent_state inline from the loaded config, the same way
the other gates do. policy.py keeps only what config can't express on its own —
the consent constants, ensure_install_id(), and may_upload_aggregate(config) as a
pure function (the gate a future uploader must consult). resolve() and the
TelemetryDecision dataclass are removed; policy.py drops 107 -> 70 lines.
No behavior change: status renders identically, and the default-on local plane is
still defaulted in DEFAULT_CONFIG plus a fail-safe .get(..., True) at each gate.
Add a built-in telemetry system that records what the agent does — workflows,
model calls, tool calls, errors — to the local machine, powers `/insights`, and
can export to an operator-chosen destination. Default-on locally; nothing leaves
the machine unless the user exports it or opts into the aggregate plane.
Three planes with a hard wall between them:
- local: full-fidelity observability (real model/provider/tool names), on by
default, never leaves the machine.
- aggregate: opt-in metadata, default off. No uploader ships — consent is
recorded via telemetry.consent_state, and `preview` shows what would be
produced, computed locally.
- trajectories: full message content, opt-in, exported only to the operator's
own destination.
Mechanism:
- Bundled `telemetry` plugin registers observational lifecycle hooks
(on_session_start / post_api_request / post_tool_call / on_session_finalize).
No core call sites are edited; hooks already carry the data.
- Fire-and-forget emitter: emit() returns in microseconds, never blocks or
raises into a model/tool call. A daemon thread writes events to an
append-only JSONL log and the tel_* tables in state.db (its own sqlite
connection, separate from SessionDB).
- tel_runs / tel_model_calls / tel_tool_calls live in the declarative
SCHEMA_SQL and are reconciled automatically; SCHEMA_VERSION 16 -> 17.
- metrics derives rollups for /usage and /insights; rollup builds per-run
summaries for `hermes telemetry preview`.
Consent is config, not a parallel command surface. The config file is the root
of trust: set telemetry.consent_state with `hermes config set`, or pin any
telemetry.* key (including allow_aggregate) via managed scope, which overrides
the user's value per key. `hermes telemetry` exposes only what config cannot:
status (report), preview (query), and export.
Export:
- exporter_bulk writes telemetry (and, when the trajectories plane is enabled,
session content) to ndjson/json.
- otlp_exporter streams spans to a configured OpenTelemetry Collector over
OTLP/HTTP. The SDK is an optional extra (hermes-agent[otlp]), lazily
installed via tools.lazy_deps on first use.
- Secrets are always redacted on every export path
(redact_sensitive_text(force=True)); content export is gated by the
trajectories plane, and PII scrubbing follows telemetry.content_redaction.
OTLP auth headers reference environment variable names, never inline values.
No outbound emission to Nous. The aggregate uploader is intentionally not built.
Three parity fixes between the API server and the native gateway's
agent-runtime resolution, integrated with the provider-aware request
routing that landed in #70853:
- Session-persisted model is honored: POST /api/sessions {"model": ...}
stores a model that the chat handlers previously fetched and threw
away. A stored value that matches a model_routes alias goes through
the route path (route provider/credentials apply); a raw model string
threads through as session_model, pinning the session's turns ahead
of per-request body values but below an explicit session /model
override.
- Empty-model recovery: provider-catalog default when config has no
model.default but a provider resolved, plus last-known-good model
recovery (#35314) keyed on gateway_session_key only (never ephemeral
session_id — no unbounded growth from one-off requests).
- Provider auth failures surface as controlled responses: RuntimeError
from _resolve_runtime_agent_kwargs() is re-raised as a dedicated
_ProviderAuthResolutionError at the call site, caught narrowly in
_run_agent() and the /v1/runs executor to return run.py's response
shape instead of an undifferentiated 500 (session-chat endpoints
previously returned a raw aiohttp 500 with no JSON body).
Salvaged from PR #57947 by @FvanW; session-model route-alias resolution
from PR #59941 by @kaishi00.
Co-authored-by: kaishi00 <kaishi00@users.noreply.github.com>
cron/scheduler.py deliberately applies utf-8/replace only on Windows via
popen_kwargs (non-Windows keeps locale default per its test contract) —
drop the sweep's unconditional inline kwargs there. Update the gateway
force-kill kwarg snapshot for the new guard.
- whatsapp taskkill + webhook gh-comment assert_called_with: add the two
new kwargs
- test_status fake_run: accept **kwargs so signature-strict stub doesn't
TypeError on encoding/errors
- Strip the salvaged commit's inline encoding kwargs where main had since
gained its own (process_registry, local env, cua doctor, gateway,
commands, gateway_windows — the latter keeps its locale-aware
_schtasks_encoding() from #38186)
- Revert encoding kwargs mistakenly applied to non-subprocess APIs
(exa get_contents, tempfile.mkstemp in webhook.py)
- Guard the ddgs worker Popen (new on main since #55339)
- Update two kwarg-snapshot test assertions for the new kwargs
Adds a new rule to scripts/check-windows-footguns.py that flags
subprocess.run/Popen/call/check_output/check_call(..., text=True, ...) calls
missing an explicit encoding= kwarg.
On Chinese Windows (cp936/GBK) and other non-UTF-8 default codepages,
text=True without encoding= decodes child output with
locale.getpreferredencoding(False), crashing _readerthread with
UnicodeDecodeError on non-default-codepage bytes (issues #47939, #53428,
rule prevents future regressions.
Rule design:
- Pattern matches 'text=True' / 'text = True'
- post_filter skips lines that:
- already pass encoding= on the same line
- are method definitions (def text)
- contain text=True inside string literals
- are not subprocess-shaped calls (heuristic via _is_likely_subprocess_call)
- Two helper functions: _is_likely_subprocess_call, _looks_like_string_literal
- Multi-line calls where subprocess.X( and text=True are on different lines
are not flagged (acceptable false negative for a line-based scanner)
Also fixes the linter's own footgun: get_staged_files() and get_diff_files()
used subprocess.check_output(text=True) without encoding= — now fixed.
Suppresses 4 false positives on non-Windows platform-exclusive calls:
- tools/voice_mode.py (Termux/Android)
- tools/environments/singularity.py (Linux HPC)
- plugins/google_meet/cli.py (macOS system_profiler)
Test plan:
- 21 unit tests in tests/scripts/test_footgun_subprocess_encoding.py
- TestDetection: 6 cases verifying the rule flags real subprocess calls
- TestSuppression: 7 cases verifying false-positive avoidance
- TestHelpers: 7 cases for the two helper functions
- TestFullRepoScan: scans the whole tree and asserts the new rule finds
only the 7 call sites that PR #60741 fixes (or zero, once #60741 merges)
Verified: full-repo scan reports 7 matches on main (the #60741 sites),
4 platform-exclusive calls correctly suppressed, zero false positives.
Hermes-sweeper review on #60741 flagged that _op_version (the paired
op probe used by the same setup/status CLI flow at lines 127 and 205)
still ran text=True without explicit encoding/errors.
Add encoding='utf-8', errors='replace' to match _op_whoami and the
production op read path at agent/secret_sources/onepassword.py:271-278.
Also extend the regression test to cover _op_version alongside
_op_whoami, and update the module docstring to reflect the widened
scope. Test sensitivity verified: reverting the source change makes
test_op_version_passes_utf8_encoding fail with encoding=None.
PR #55339 adds encoding='utf-8', errors='replace' to 26 subprocess.run(text=True)
call sites across the codebase. The triage review (thanks @alt-glitch) diffed
this PR against #55339 and found that 5 of the 6 originally-touched call sites
are already covered there byte-identically:
- hermes_cli/main.py::_probe_container
- hermes_cli/setup.py SSH probe
- tools/tts_tool.py::_generate_neutts
- tools/transcription_tools.py::_prepare_local_audio
- tools/transcription_tools.py::_transcribe_local_command (both branches)
The one genuinely net-new site — hermes_cli/onepassword_secrets_cli.py::_op_whoami
(the 1Password op CLI whoami probe) — is NOT in #55339 and is fixed here.
Without explicit encoding=, text=True decodes child output with
locale.getpreferredencoding(False) — cp936 on Chinese Windows — which crashes
_readerthread on non-GBK bytes, cascading into pipe buffer fills, event loop
stalls, and TUI freezes (issues #47939, #53428, #57238).
Scope narrowed per triage feedback: the other 5 sites should land via #55339.
Refs #53428 (together with #55339).
Add authenticated GET /api/model/options to the gateway API server,
sharing the dashboard/TUI picker payload builder so external clients
can sync to the user's configured Hermes provider catalog instead of
scraping the single OpenAI-compatible /v1/models alias.
- new shared hermes_cli.inventory.build_model_options_payload() wraps
build_models_payload with the stable picker shape and safe
custom-provider probe policy (probe current only on normal open,
probe all + cache bust on explicit refresh)
- dashboard web_server and TUI gateway model.options refactored onto
the shared builder; dashboard build moved off the event loop via
run_in_threadpool
- capabilities endpoint advertises model_options
- docs for both API server and programmatic integration
Salvaged from PR #54689 by @abundantbeing.
Follow-up to salvaged PR #70549:
- Replace fragile 'or' assertions with single precise checks that catch
partial-rewrite regressions (would have masked a missing closing brace)
- Add test_pty_path_uses_rewritten_command covering the PTY spawn path
that was modified but previously untested
A session pinned to a named custom provider could silently reroute to the
user's default provider on resume/rebuild. Session rows persist the RESOLVED
provider — bare "custom" for every named providers:/custom_providers: entry —
and when no base_url survived in model_config, the existing heal
(canonical_custom_identity) had only the config.model.provider fallback left.
For users whose global default is a BUILT-IN provider (e.g. nous) that tier
cannot fire, so the bare provider was dropped, resume fell back to the default
provider with the session's custom model name, and the default endpoint 404'd
with "Model '<x>' not found. The requested model does not exist in our
configuration or OpenRouter catalog." Re-selecting via /model fixed it until
the next resume — the reported symptom.
Add a model-name recovery tier between the base_url reverse-lookup and the
config fallback: find_custom_provider_identity_by_model() maps the stored
model back to the entry that serves it (model/default_model/models catalog,
dict and legacy list shapes). The session row always stores the model, so the
entry identity survives even when the row has no base_url AND the global
default points elsewhere.
All five bare-custom heal sites in tui_gateway/server.py now pass the model:
_ensure_session_db_row, _stored_session_runtime_overrides,
_runtime_model_config, _make_agent, and _model_picker_context.
Issue #68915: when the agent runs a compound command with trailing & (e.g.
`cd /app && node server.js &`), bash parses it as `(A && B) &` — a subshell
that holds the stdout pipe open forever when B is a long-running server.
The existing _rewrite_compound_background in terminal_tool.py correctly
rewrites this to `A && { B & }` to avoid the subshell fork, but it was only
applied in the foreground execute() path (tools/environments/base.py).
The background spawn_local() path bypasses base.py entirely and passed the
raw command directly to Popen/PTY, leaving the deadlock unmitigated.
Fix: apply _rewrite_compound_background in spawn_local() before the command
is passed to Popen or PTY spawn. Uses a lazy import to avoid circular
dependency (terminal_tool imports process_registry).
- PTY spawn path: now uses safe_command (rewritten)
- Popen spawn path: now uses safe_command (rewritten)
- Session.command still stores the original (unrewritten) command for display
- Simple `cmd &` is left unchanged (no subshell bug)
Tests: 4 regression tests verifying (1) compound is rewritten, (2) simple bg
is preserved, (3) multi-line compounds are rewritten, (4) session.command
stores original.
Follow-up on the salvaged #68469 commit:
- Literal-IP hostnames never take the proxy DNS-delegation path (a
getaddrinfo failure on a literal IP is not a proxy-environment
symptom, and IPs need no DNS) — keeps the private-IP/metadata floor
intact under proxy env vars.
- Adds TestProxyEnvironmentDnsDelegation: delegation fires only for
hostnames, metadata hostname/IP floor holds, DNS-success path
unchanged, empty proxy var ignored.
- Guards the three pre-existing DNS-failure tests against ambient
proxy env vars so they don't flake on developer machines.
Track the selected credential by stable pool entry ID so token refreshes and shared cursor movement cannot detach failures from the entry that issued them. Stop unmatched single-entry pools from reporting a no-op rotation as successful recovery.
Co-authored-by: Maxim Esipov <maksesipov@gmail.com>
Follow-ups on the salvaged #54426 routing contract:
- Bare `model` without `provider` on the OpenAI-compatible endpoints
(/v1/chat/completions, /v1/responses) is now opt-in via
gateway.platforms.api_server.direct_model_requests (default off) —
generic OpenAI clients hardcode model names ('gpt-4o', ...) and
existing deployments rely on those falling back to the gateway
default. Explicit `provider` requests and the Hermes-native
session-chat + /v1/runs surfaces are always honored.
Idea credit: PR #22825 by @mssteuer.
- A model_routes alias with no `model` key can no longer leak the
alias string as the executing model name (defensive; parse-time
validation already drops such routes).
- Fix mis-indented _run_agent call args in _handle_session_chat_stream.
- Docs: document the opt-in flag.
Carry model, provider, and model_options through the API server's
execution surfaces (session chat, Chat Completions, Responses, /v1/runs)
without mutating global configuration. Precedence: session /model
override -> model_routes alias -> direct request selection -> global
defaults. Conflicting route/provider mixes fail closed with 400.
model_options stays request-scoped regardless of which selection wins.
Salvaged from PR #54426 by @abundantbeing.
test_docker_network_config.py landed on main after the #58489 revert and
stubbed docker ps with the 2-field ID\tState format. The re-landed
egress-aware reuse probe requests ID\tState\tEgressLabel when egress is
off, so the fake line failed to parse and the reuse path never fired.
Fixture-only change; production behavior is unchanged.
The gateway startup maintenance block gained a maybe_auto_archive call in
the same provably-off-loop __init__ site as maybe_auto_prune_and_vacuum;
bump the reviewed sync-escape count from 3 to 4.
New sessions.auto_archive / auto_archive_days config: soft-hide (never
delete) sessions with no activity for N days, aging on last activity
rather than creation so an old-but-active chat is spared. Sweeps are
throttled through state_meta and fire from CLI startup, gateway startup
+ hourly housekeeping, and the serve/dashboard backend (opportunistic
on session list + an hourly lifespan ticker), so every surface honours
one setting.
A new pinned column (declaratively migrated) exempts sessions from the
sweep; PATCH /api/sessions/{id} accepts pinned and flips the whole
compression lineage as a unit, mirroring set_session_archived.
session.branch wrote the child ROW into the parent's profile db but
built the live agent with the launch defaults: _make_agent fell back to
_get_db() and no HERMES_HOME override was active. The branched agent's
own message flushes — and any later compression rotation it performed —
therefore landed back on the launch profile, splitting the lineage one
turn after the branch. Mirror session.create/resume: open the parent
profile's SessionDB for the agent and hold the home override across the
build, so config/skills/memory resolve to the profile too.
Spotted in #70605's sibling implementation of the same fix.
Co-authored-by: HexLab98 <liruixinch@outlook.com>
The blanket MAX_DESCRIPTION_LENGTH=1024->60 change is narrowed:
create-time validation now rejects new skills whose description
exceeds SKILL_PROMPT_DESC_LIMIT (60) with actionable guidance, while
edit/patch paths stay permissive (warning via system_prompt_preview)
so existing over-limit skills remain maintainable. Runtime display
truncation in skills_tool is left at 1024 (display behavior is a
separate concern from authoring validation).
Boundary tests: 60 accepted, 61 rejected at create; edit/patch on
over-budget skills still succeed.
RFC 8628 §3.2 makes the device-authorization `interval` optional with a
client-side default of 5 seconds. request_device_code required it, so a
compliant AS that omitted it hit the malformed-response path and the flow
could never complete. Fall back to 5s and cover it with a regression test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a device authorization grant flow alongside the existing loopback
OAuth flow, so `hermes setup` can connect to Honcho cloud from SSH and
other no-browser environments.
- oauth.py: new HTTP seams — _http_post_form_status (non-raising, since
RFC 8628 polling reads the OAuth error off a 400) and _http_get_json
for the RFC 8414 metadata probe
- oauth_flow.py: DeviceCode, request_device_code, poll_for_token with
slow_down backoff (+5s, capped at 60s) bounded by expires_in, typed
errors (AccessDenied, DeviceCodeExpired, AuthorizationTimeout), and
supports_device_login (fail-closed metadata gate); device flow ends in
the same install_grant tail as loopback so refresh/status work
unchanged
- oauth_flow.py: loopback callback now serves a "sign-in was not
completed" page on consent cancel instead of the success page
- cli.py: cloud menu offers oauth / device / apikey; the device option
only appears when the host advertises the grant, and becomes the
default when no browser is detected
- 18 new tests covering the full flow against a local fake AS, backoff
schedule, error mapping, deadline bound, metadata gate, and wizard
branches
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under multiplex_profiles, the gateway starts a single InProcessCronScheduler
bound to the process-global HERMES_HOME (the default profile's home), so
only that profile's cron/jobs.json is ticked. A job registered from a
secondary-profile session lands in <profile>/cron/jobs.json, reports a valid
next_run_at — and never fires.
Changes:
1. cron/scheduler_provider.py — InProcessCronScheduler.start() now accepts
an optional profile_homes kwarg (list of (name, Path) tuples). When set,
_start_multiplex() iterates tick() over each profile home using
use_cron_store(), so every served profile's cron store is ticked on
every tick cycle. Heartbeats and interrupted-execution recovery are also
scoped per profile via use_cron_store().
2. gateway/run.py — start_gateway() now resolves profiles_to_serve(multiplex=True)
when multiplex_profiles is on and passes them to the cron scheduler as
profile_homes. Only applies to InProcessCronScheduler (the built-in);
external providers are unchanged.
3. cron/jobs.py — record_ticker_heartbeat(), get_ticker_heartbeat_age(), and
get_ticker_success_age() now resolve paths via _current_cron_store()
instead of module-level TICKER_HEARTBEAT_FILE / TICKER_SUCCESS_FILE
constants. This makes heartbeats correctly scoped per profile, so
'hermes cron status' reflects liveness for every profile independently
under multiplex_profiles.
4. tests/cron/test_scheduler_provider.py — two new tests:
- test_multiplex_ticker_ticks_each_profile_once: verifies tick() is called
once per profile per tick cycle.
- test_multiplex_heartbeat_scoped_per_profile: verifies heartbeat files
are written to each profile's cron store.
Widens the salvaged .env injection fix (#50315) to the sibling site it
missed: hermes_cli/memory_setup.py::_write_env_vars is the near-identical
core writer the openviking plugin's copy was forked from, is fed directly
by interactive _prompt() (pasted API keys), and is reused by other memory
plugins (e.g. supermemory imports it). A pasted secret with an embedded
CR/LF injected an arbitrary extra KEY=VALUE line on the next read.
Same _env_line_safe() treatment as the plugin writer (strip every
str.splitlines() separator + NUL), matching config.save_env_value's
existing newline strip. Mutation-checked: reverting the sanitizer makes
the new regression tests fail.
Follow-up hardening on the salvaged _ensure_client() (#21130 fix):
- Failed-config cooldown: after a refresh attempt fails for a given
resolved config, skip re-probing for 30s. Previously every provider
access against a down endpoint paid a 3s health probe under
_client_refresh_lock and emitted a warning (2+ per turn, some on
user-facing threads: prefetch, tool calls, session end). Retries
still happen after the cooldown or immediately when config changes,
and the log message now says so instead of the false 'disabled until
config changes'.
- Atomic connection snapshot: _conn_snapshot (5-tuple, single
assignment) is published only after a health check passes.
_new_client() and on_memory_write's writer read it as one load, so
background writers can no longer observe a torn mix of old/new
identity fields mid-refresh or target an endpoint that never passed
health. Field writes in _ensure_client_locked keep tracking the
attempted config for the unchanged-config dedupe.
- _env_refresh_enabled moves to the top of initialize(): an exception
mid-initialize (swallowed by MemoryManager) can no longer leave the
provider silently stuck in never-refresh mode.
- _search_prefetch_context reuses _new_client() and degrades to ''
on construction failure instead of propagating.
Mutation-checked: neutering the cooldown or publishing the snapshot on
failed health makes the new regression tests fail.
