After context compression, the parent session holds pre-compression messages
and a child (or deeper descendant) holds the continuation.
resolve_resume_session_id() short-circuited when the input session already
had messages (row is not None -> return session_id), causing REST API
endpoints, gateway resume, and CLI resume to serve stale parent messages.
Remove the early-return. Walk the full descendant chain, record the
deepest node that has messages (best), and return best if not None
else the original session_id (preserving the empty-chain fallback).
Callers (api_server.py, web_server.py, cli_agent_setup_mixin.py,
cli_commands_mixin.py) all use the resolved != input -> redirect pattern
and are transparent to this change.
The pre-update HERMES_HOME zip shipped on by default (DEFAULT_CONFIG +
runtime fallback both True), so every `hermes update` zipped the entire
~/.hermes — sessions DB, caches, skills — adding minutes to each update.
The shipped cli-config.yaml.example, the --backup help, and the example
config all already said "off by default," so the live default
contradicted its own documentation.
Flip the default to off everywhere: DEFAULT_CONFIG, the runtime
`.get(..., False)` fallback in _run_pre_update_backup, and the stale
--backup help string. Users who want the #48200 safety net opt in via
updates.pre_update_backup: true or --backup for a single run.
Updated test_default_enabled_creates_backup -> test_default_disabled_is_silent
to assert the new default (silent no-op, no zip).
* fix(cron): add default retention to per-run job output to bound disk usage (#52383)
Per-run cron output (cron/output/<job>/<timestamp>.md) is written once
per execution and was never pruned, so a frequently-scheduled job on
a long-running deploy accumulates one file per run indefinitely and
can fill the volume ('no space left on device').
save_job_output() now keeps the most recent N output files per job and
removes older ones. N defaults to 50 and is configurable via
cron.output_retention; a non-positive value disables pruning for
operators who manage cleanup externally.
Salvaged from #52402 by @0xDevNinja.
Closes#52383
* fix(config): add cron.output_retention to DEFAULT_CONFIG
Follow-up to #52383: the retention config key was functional via
get()-with-default but missing from DEFAULT_CONFIG, so the deep-merge
wouldn't auto-populate it for new installs. Add it explicitly.
---------
Co-authored-by: 0xDevNinja <manmit0x@gmail.com>
Regression for the salvaged #48254 fix: billing route is first-writer-wins
via update_token_counts (COALESCE), so a mid-session provider switch left
the dashboard attributing cost to the original provider. Asserts the new
update_session_billing_route() overwrites unconditionally, nulls system_prompt
so the next turn rebuilds Model:/Provider:, and preserves billing_mode when
omitted (COALESCE on None).
The session database records billing_provider and billing_base_url using
COALESCE(column, ?) in update_token_counts(), making them write-once.
When a user switches models mid-session via /model, the runtime (agent.provider,
agent.base_url) updates correctly, but the session row never reflects the new
provider. This causes the dashboard Models page to display a stale provider
badge and misattributes token usage / cost analytics.
Fix: add update_session_billing_route() that unconditionally sets
billing_provider, billing_base_url, and billing_mode (no COALESCE), and call
it from switch_model() in agent_runtime_helpers.py after the swap succeeds.
This follows the same pattern as update_session_model() which already
unconditionally updates the model column (added for the identical COALESCE
problem on the model field).
Closes#48248
A stale-index render race in assistant-ui (a just-shrunk thread rendered
at an old message index during a session switch / teardown) throws
errors like "tapClientLookup: Index N out of bounds", "Cannot read
properties of undefined (reading 'type')", or "Tried to unmount a fiber
that is already unmounted". These bubble to the root ErrorBoundary and
latch the WHOLE desktop app on the "Reload window" fallback even though
the next render against fresh state would be fine.
Teach the root boundary to treat that small set of known-transient
renderer errors as recoverable: log them and schedule a next-tick
reset() so React re-renders against current state instead of stranding
the user on the fallback.
Auto-recovery is BOUNDED -- at most MAX_RECOVERIES (3) attempts within a
5s window -- so a genuinely persistent error can't spin the boundary in
a reset -> throw -> reset loop; after the budget is spent the fallback
is left up for the user. Manual retry (the button) resets the budget.
Only the root boundary auto-recovers; scoped boundaries keep their own
fallbacks, and unrecognized errors are never swallowed.
Tests: transient race recovers (fallback never sticks), a persistent
recoverable error stops at the cap and surfaces the fallback (proving
the loop is bounded), and neither a non-root boundary nor an
unrecognized root error auto-recovers.
Closes#41693. Supersedes #41787 by @izumi0uu, reimplemented with a
bounded recovery budget so a non-transient error can't loop forever.
Co-authored-by: izumi0uu <izumi0uu@gmail.com>
resumeSession's warm-cache fast-path trusted the
storedSessionId -> runtimeId -> ClientSessionState mapping without
checking the cached state still BELONGS to the session being resumed.
A pooled profile backend that gets idle-reaped and respawned
(pruneSecondaryGateways) re-mints runtime ids, so a recycled id can
resolve to a live-but-DIFFERENT session's cache entry. The only
existing guard was a session.usage 404 -- that catches a fully-dead
runtime id, but a recycled id still 200s, so the fast-path happily
painted the wrong transcript under the current route (open chat A,
chat B loads).
Fold the belongs-to check into a single takeWarmCache() helper used at
BOTH cache reads -- the early transcript-keep decision and the fast-path
itself -- so a cross-wired entry can't even briefly flash a stale
transcript before the full resume repaints. On a mismatch the helper
purges both stale map entries and reports a miss, falling through to a
full resume that rebinds a correct runtime id. The full-resume path
already guards its final paint with isCurrentResume(), so only the
cached fast-path was missing the belongs-to check.
Pre-existing bug from the initial desktop app (#20059); not introduced
by the session-switch perf work (#49807), which left these lines
untouched.
Tests: two cases in use-session-actions.test.tsx driven through a
harness that owns the two cache maps -- a cross-wired mapping is
rejected + purged (the bug), and a correctly-wired cache still serves
from memory with no needless refetch (no perf regression).
Supersedes #50464 by @professorpalmer, reimplemented to also guard the
early transcript-keep read (whole-class fix, not just the fast-path).
Co-authored-by: professorpalmer <professorpalmer@users.noreply.github.com>
* feat(moa): expose MoA presets as selectable virtual models
Reconstructed onto current main (PR #46081's base had diverged with no common
ancestor, marking the PR dirty so CI never dispatched). MoA is now a virtual
provider: each named preset is a selectable model under provider 'moa', and the
preset's aggregator is the acting model that answers and calls tools.
Reference models fan out in parallel via a bounded ThreadPoolExecutor (the same
batch pattern delegate_task uses) — all references dispatched at once, collected
when every one finishes, then handed to the aggregator. Output order is
preserved, failures and the MoA-recursion guard stay isolated per reference.
- Removed the old mixture_of_agents model tool and moa toolset.
- Added moa as a virtual provider in the provider/model inventory.
- /moa is shortcut behavior over model selection (default preset / named preset
/ one-shot prompt).
- Dashboard + Desktop manage named presets; presets appear in model pickers.
- Parallel reference fan-out in agent/moa_loop.py with regression test.
* fix(moa): thread moa_config through _run_agent to _run_agent_inner
The reconstructed gateway MoA wiring declared moa_config on _run_agent (the
profile-scoping wrapper) and used it inside _run_agent_inner, but the wrapper
never forwarded it — _run_agent_inner had no such parameter, so the runtime hit
NameError: name 'moa_config' is not defined on the compression-failure session
sync path. Add moa_config to _run_agent_inner's signature and forward it from
both wrapper call sites (multiplex and non-multiplex). Caught by
tests/gateway/test_compression_failure_session_sync.py on CI shard test(4).
* fix(moa): classify moa as a virtual provider in the catalog
The moa virtual provider has no PROVIDER_REGISTRY/ProviderProfile entry, so
provider_catalog() fell through to the default auth_type="api_key" with no
env vars — tripping two catalog invariants:
- test_provider_catalog: api_key providers must expose a credential env var
- test_provider_parity: every hermes-model provider must be desktop-configurable
moa already declares auth_type="virtual" in HERMES_OVERLAYS; consult that
overlay as an auth_type fallback so the catalog reports moa as virtual (no real
credential, no network endpoint). Exempt virtual providers from the desktop
parity union check the same way 'custom' is exempt — derived from the catalog,
not a hardcoded slug, so future virtual providers are covered too.
Scheduled jobs delivering to Telegram/etc. started posting a literal
'⚠️ No reply: the model returned empty content…' message instead of
staying silent. Two interacting causes:
1. The turn-completion explainer (#34452) replaces an empty model turn
with a user-facing '⚠️ No reply…' string. In a cron context that is
not a silence marker, so the scheduler delivered it — a regression
from the previously-silent empty turn. run_job now detects the
explainer text deterministically (via the same formatter that
produced it) for abnormal-empty turn_exit_reasons and strips it to
empty, so the existing empty-response suppression + soft-fail guard
apply. The explainer is unchanged on CLI/gateway.
2. The cron suppression used a loose 'SILENT_MARKER in ...upper()'
substring check. It leaked bracketless near-markers the model emits
('SILENT', 'NO_REPLY', 'NO REPLY' — #51438, #46917) and wrongly
swallowed a real report that merely quoted '[SILENT]' mid-sentence.
Replaced with _is_cron_silence_response(): suppresses a canonical
token as the whole response, its own first/last line, or the
documented bracketed '[SILENT] <note>' prefix — while a token buried
mid-sentence in a genuine report is delivered. Preserves the
intentional cron trailing/prefix tolerance (existing tests unchanged).
Tests: bracketless-variant suppression, mid-sentence-quote delivery,
direct matcher contract, and explainer-strip + defensive real-report
delivery.
Assert bare tables upgrade to sendRichMessage under default/opt-out config,
DM-topic resumed sends without reply anchors, and rich finalize edits carry
forum topic routing metadata.
Pipe-only markdown tables now use sendRichMessage even when rich_messages
is off, and resumed DM-topic sends route via direct_messages_topic_id
without requiring a reply anchor. Rich finalize edits forward topic kwargs.
The salvaged context-window screen (#52392) skips fallback candidates that
are too small, and the rate-limit/403 fixes skip candidates that are at
capacity. A third hard failure remained uncovered: a fallback that builds a
client fine but returns a 400 because it structurally cannot run the model.
The canonical case is a configured openai-codex / ChatGPT-account fallback
asked to compress a glm-5.2 conversation:
400 - {'detail': "The 'glm-5.2' model is not supported when using
Codex with a ChatGPT account."}
This is a request-validation error, so should_fallback was False and the
explicit-provider gate blocked it — the auxiliary task (compression) aborted
every turn, dropping middle turns without a summary and churning the session,
which is exactly what destroys the prompt cache.
Adds _is_model_incompatible_error() (400 + capability phrasing, excluding
not-found and billing 400s which the sibling predicates own) and treats it as
a fallback-worthy capacity error in both sync and async call_llm, so the chain
skips the incapable route and continues to the next viable candidate.
The runtime auxiliary fallback chain (_try_configured_fallback_chain and
_try_main_fallback_chain) returned the first reachable candidate without
checking whether the candidate's context window was large enough for the
task. For task='compression' this meant a reachable but undersized
fallback (e.g. 32K) could be selected and then fail, even when a later
larger-context fallback was available.
This adds two small helpers:
_task_minimum_context_length(task)
Returns MINIMUM_CONTEXT_LENGTH (64K) for compression, None for
other tasks (vision, web_extract, etc.).
_candidate_context_window(provider, model, ...)
Thin wrapper around get_model_context_length that returns None on
probe failure so unknown/custom endpoints pass through unchanged
(preserves the existing fallback surface).
Both fallback loops now skip reachable candidates whose resolved context
is below the task minimum and continue iterating. The success path
(first viable candidate wins) is unchanged. Return shape and ordering
for healthy candidates are preserved.
Six regression tests cover:
L2 configured chain skips too-small candidate
L2 chain continues after skipping, returns last viable
L3 main chain skips too-small candidate
L4 unknown-context candidate passes through
L5 non-compression task is not filtered
L6 minimum constant matches MINIMUM_CONTEXT_LENGTH (64K)
3/6 fail on upstream/main without the production change (verified); all
6 pass with the fix. Full test_auxiliary_client.py suite (231 tests)
and related compression tests (130 tests) remain green.
When an explicit aux provider cannot build a client before any request is
sent (missing raw env key, exhausted/unavailable OAuth or credential-pool
auth, resolver returning (None, None)), call_llm raised a misleading
"no API key was found" error and bypassed the configured fallback_chain
entirely. A provider authenticated through Hermes auth / the credential
pool (e.g. ollama-cloud) whose pool entry is exhausted hit this path, so
compression failed instead of routing to the configured fallback.
Adds _try_configured_fallback_for_unavailable_client() and wires it into
both sync and async call_llm before the raise, and into the startup
compression feasibility check.
Salvaged from #51835 by @herbalizer404.
Rate-limit (429) errors on explicit-provider auxiliary tasks were
silently failing instead of triggering the fallback chain. The
is_capacity_error gate only checked payment and connection errors,
excluding rate limits — so when a configured provider like
openai-codex hit its rate limit, auxiliary tasks (kanban_decomposer,
vision, web_extract, approval, etc.) had zero resilience.
Add _is_rate_limit_error() to is_capacity_error at both call sites
(sync and async paths) so rate limits trigger fallback regardless
of whether the provider was auto-detected or explicitly configured.
Fixes#52228
Ollama Cloud (and similar) return 403 with bodies like "this model requires
a subscription, upgrade for access" or "you have reached your session usage
limit, upgrade for higher limits". These are capacity/billing conditions
semantically identical to credit exhaustion, but _is_payment_error() did not
recognize them (403 missing from the status set; keywords missing), so the
configured fallback_chain was never tried and compression failed outright.
Adds 403 to the status set and the subscription/session-usage keywords.
Salvaged from #49076 by @herbalizer404.
These 7 test sites assert rotation behavior (fork, child sessions, lock
contention, logging session-context follows id rotation, boundary hooks fire
on rotation). Pin each builder to in_place=False explicitly so they keep
exercising the retained rotation fallback regardless of the global default
(flipped to True in #38763). Rotation stays a working opt-out fallback and
deserves continued coverage — these are NOT deleted.
Pinned sites:
- test_compression_concurrent_fork._build_agent_with_db
- test_compression_logging_session_context._build_agent_with_db
- test_compression_rotation_state._build_agent_with_db
- test_compression_boundary_hook._make_agent (2 helpers: CompressionBoundaryHook + SessionCompressEvent)
- test_compression_concurrent_sessions._build_agent_with_db
In-place compaction (single durable session id, non-destructive soft-archive)
becomes the default. Rotation is now the opt-out fallback via
compression.in_place: false.
Prerequisite: #50098 (hygiene guard reads result flag not config flag) merged
first — without it, flipping the default causes permanent transcript loss on
gateway hygiene-compress and /compress when no session_db is available.
Blast radius (empirically measured on current main): 7 rotation-asserting
tests broke and are pinned to in_place=False in the companion test commit:
- tests/agent/test_compression_concurrent_fork.py (2)
- tests/agent/test_compression_logging_session_context.py (1)
- tests/agent/test_compression_rotation_state.py (1)
- tests/run_agent/test_compression_boundary_hook.py (2 _make_agent helpers)
- tests/gateway/test_compression_concurrent_sessions.py (2)
Rotation stays as a working fallback and deserves continued coverage.
Plan: .hermes/plans/in-place-compaction-38763.md
Salvage of #50098 by @srojk34, cherry-picked onto current main.
The hygiene auto-compress guard and the /compress slash command both read
compression_in_place (config flag — is in-place mode enabled?) instead of
_last_compaction_in_place (result flag — did in-place compaction actually
succeed?). Both agents are built without a session_db, so archive_and_compact
always fails silently and _last_compaction_in_place stays False. Reading the
config flag makes the guard think in-place succeeded, triggering
rewrite_transcript() which replaces the original messages with only the
compressed summary — permanent data loss.
Co-authored-by: srojk34 <srojk34@users.noreply.github.com>
build_turn_context() created the DB session row via _ensure_db_session()
before the system prompt was restored/built, so a fresh API/gateway agent
carrying client-managed history inserted a row with system_prompt=NULL. That
tripped the misleading 'stored system prompt is null; rebuilding from scratch
... investigate the previous turn's write path' warning and a guaranteed
first-turn prefix cache miss. Move row creation to after _cached_system_prompt
is populated.
Verified live (OpenRouter + claude-sonnet-4.5): persistent-agent turns show
cache_read jumping to the full prefix on turn 2+ (write 24411 -> read 24411),
and the persisted system_prompt is non-NULL so fresh-agent restore keeps the
prefix cache warm.
Tests: turn-context ordering regression asserting _ensure_db_session runs
after _cached_system_prompt is populated.
/learn told the agent to fill the skill `author` field, and the system
prompt environment probe surfaces the OS login name (user=$(whoami) in
prompt_builder.py), so the model wrote the host username into published
SKILL.md frontmatter — a privacy leak the user never opted into, and
inconsistent run to run as the most-salient identity changed.
The /learn authoring prompt now sets `author` to the literal value
`Hermes` and explicitly forbids deriving it from the host environment
(OS/login user, git config, or any probeable identity). The skill names
itself as the tool that wrote it.
Closes#52368.
