Widen the #59395 fix to the sibling site: update_job's schedule-change path
(cron/jobs.py) had the SAME unguarded compute_next_run -> next_run_at pattern,
so updating a job's schedule to a one-shot >ONESHOT_GRACE_SECONDS in the past
would re-create the ghost job (next_run_at=None, state='scheduled', never fires)
that create_job now rejects. Apply the identical guard on update (raise before
any disk write, so the original job is left intact), with regression tests for
the reject + future-accept cases.
Also surface ONESHOT_GRACE_SECONDS in the raised ValueError (not just the
warning log) so a caller knows how far in the past is too far. Message from the
competing PR #59410 by @isheng-eqi.
Co-authored-by: isheng-eqi <265044697+isheng-eqi@users.noreply.github.com>
Bare 'hermes sessions prune' keeps the historical 90-day default, but any
filter — now including --source — suppresses the implicit cutoff, so
'prune --source cron' targets ALL cron sessions instead of silently only
those older than 90 days (the surprise a user hit live: 'No sessions
match ... source cron' despite plenty of recent cron runs).
- CLI preview + confirmation now show the match count plus the oldest
and newest matching session start times before deleting.
- Dashboard /api/sessions/prune mirrors the semantics: attribute filters
without an explicit older_than_days match all ages (model_fields_set
distinguishes an explicit 90 from the Pydantic default); dry_run
responses gain oldest_started_at/newest_started_at.
- Docs + argparse help updated; tests for both surfaces.
The Skills Hub 'Browse Hub' landing page and index-backed search render
empty on fresh deployments (e.g. Fly.io VPS agents) with no stale cache.
Root cause: the centralized index at /docs/api/skills-index.json is a
large body (~34MB, tens of MB compressed) served with Content-Encoding:
br. httpx's streaming Brotli decoder — backed by brotlicffi 1.2.0.1,
which is pinned so aiohttp can decode Discord attachments — trips over
its own output_buffer_limit on a payload this size and raises:
DecodingError("brotli: decoder process called with data when
'can_accept_more_data()' is False")
_load_hermes_index() catches that (DecodingError is an httpx.HTTPError
subclass) and silently falls back to the on-disk cache. On a fresh box
that cache never existed, so HermesIndexSource.is_available is False,
the index contributes 0 skills, and the hub landing page — which is
built solely from an empty-query index search — is blank. Existing
installs only appear to work because they serve a (possibly weeks-)stale
cached index instead.
Fix: request 'gzip, deflate' on the index fetch so httpx never
negotiates the broken Brotli path, and retry once with 'identity' if a
DecodingError still occurs (defends against a proxy that ignores the
header). Falls through to the stale cache only when both attempts fail.
Verified on a live staging VPS agent: index_available flips False->True
and the featured landing list repopulates from 0 to 12.
Also un-freezes already-deployed images: skills added after an image was
built (e.g. the 'unbroker' optional skill) become reachable again via
the index, which is the whole point of the centralized catalog.
load_gateway_config() only surfaced the top-level `multiplex_profiles`
key into gw_data before calling GatewayConfig.from_dict(). A config.yaml
that pinned the flag under the nested `gateway:` section -- the form
written by `hermes config set gateway.multiplex_profiles true` -- was
silently ignored, so the gateway loaded with multiplex_profiles=False.
from_dict() already honors the nested fallback, but load_gateway_config()
builds gw_data from top-level keys first, so the nested value never
reached it.
Read gateway.multiplex_profiles into gw_data when the top-level key is
absent, mirroring the existing nested fallback for max_concurrent_sessions.
Adds a load_gateway_config() regression test that writes a config.yaml
with `gateway.multiplex_profiles: true` and asserts the loaded config has
multiplex_profiles=True (fails without the fix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(sessions): full filter surface for prune + new bulk archive subcommand
hermes sessions prune previously only supported --older-than N (integer
days) and --source — no way to target a window like 'the last 5 hours'
(e.g. a batch of CI smoke-test sessions), and no non-destructive option.
- SessionDB.prune_sessions gains keyword filters that AND together:
started_before/started_after epoch bounds, title_like, end_reason,
cwd_prefix, min/max_messages, archived tri-state. Default call is
byte-for-byte compatible (90-day cutoff, ended-only, source).
- New SessionDB.list_prune_candidates (backs --dry-run + confirmation
previews) and SessionDB.archive_sessions (bulk soft-hide via the
existing set_session_archived lineage-aware path; nothing deleted).
- CLI: prune gains --newer-than/--before/--after (durations like 5h/2d/1w,
bare days, or ISO timestamps), --title, --end-reason, --cwd,
--min/--max-messages, --include-archived, --dry-run. New
'hermes sessions archive' takes the same filters, requires at least one,
and is idempotent. Both show a preview before confirming.
- Dashboard /api/sessions/prune accepts the same filters + dry_run.
- Docs: sessions.md + cli-commands.md updated.
Filter parsing lives in hermes_cli/session_filters.py with unit tests;
DB filters covered in tests/test_hermes_state.py.
* feat(sessions): prune/archive filters for model, provider, user, chat, branch, tokens, cost, tool calls
Extends the prune/archive filter surface to everything identifiable in
the sessions table:
- --model (substring on model slug), --provider (exact on
billing_provider, case-insensitive), --user, --chat-id, --chat-type
(exact), --branch (substring on git_branch), --min/--max-tokens
(input+output), --min/--max-cost (USD, actual_cost_usd falling back to
estimated_cost_usd), --min/--max-tool-calls.
- SessionDB prune/archive/list_prune_candidates now share the filter
kwargs via **filters into _prune_filter_where (unknown names raise
TypeError); candidates listing + CLI preview now include the model.
- Any attribute filter (except legacy --source) suppresses the implicit
90-day default so 'prune --model X' matches all ages.
- Dashboard /api/sessions/prune passes the new fields through.
- Docs + tests updated (7 new DB tests, 3 new parser tests).
Fixes#50051 by preserving nested gateway.multiplex_profiles and routing gateway config env reads through the active profile secret scope when present.
This keeps secondary profile adapter startup from inheriting default-profile platform tokens or port-binding enables while preserving legacy single-profile behavior outside a scope.
Constraint: latest upstream main f57ff7aef1 still reproduced both nested-config loss and cross-profile env leakage
Rejected: special-casing API_SERVER_* only | left other profile-scoped tokens vulnerable to the same leak
Confidence: high
Scope-risk: moderate
Directive: keep future gateway/config env reads on the scoped helper path unless a variable is explicitly process-global
Tested: pytest -q tests/gateway/test_multiplex_phase0.py tests/gateway/test_multiplex_credential_isolation.py tests/gateway/test_config.py -k 'multiplex or scope or getenv or api_server or relay'
Not-tested: full gateway startup across live platform adapters
test_group_new_keeps_existing_reset_semantics_when_dm_topic_mode_enabled
asserts 'parallel work' not in the /new reply — but /new appends a
random tip from hermes_cli.tips (380 entries), and one tip's text
contains exactly that phrase (the delegate_task concurrency tip). CI
failed on PR #59331 slice 2 when the dice landed on it. Pin
get_random_tip in the test.
The routing sweep sends these paths through _adapter_for_source, which
reads source.profile. A bare MagicMock auto-attribute is truthy, so the
fixtures looked like stamped secondary profiles and hit the new
fail-closed branch. Real SessionSource.profile is None or str
(AGENTS.md pitfall #17).
Follow-up to the routing sweep: when a stamped secondary profile has no
_profile_adapters entry (adapter failed to connect / was refused), return
None instead of falling back to the default profile's adapter — the
fallback sends replies out the wrong bot, which is the exact leak class
this cluster fixes. Also restores main's deliberate fail-fast on
port-binding platforms in secondary profiles (the cherry-picked commit
had softened it to silent force-disable).
Co-authored-by: ManniBr <m888.braun@hotmail.com>
Replace 53 instances of self.adapters.get(source.platform) with
self._adapter_for_source(source) in gateway/run.py.
self.adapters is the default profile's adapter map. In multiplex mode,
secondary profiles (lars, kira, jonas, caro) have their adapters in
_profile_adapters[profile]. _adapter_for_source() (from authz_mixin.py)
correctly resolves through _profile_adapters when source.profile is set.
Without this fix, ALL response paths for secondary profiles — streaming,
sending, media delivery, voice, typing indicators, queue operations,
startup restore, and platform notices — route through the default
profile's bot token instead of the profile's own token.
Fixes: Multiplex profiles responding with wrong bot token on Telegram,
Discord, and all other platforms.
register_mcp_servers now nudges cached entries whose session is None
via _signal_reconnect, so a new agent session recovers a parked server
immediately instead of waiting up to _PARKED_RETRY_INTERVAL for the
next self-probe (#50170). Gate-check idea credit: @izumi0uu (#50184),
@LeonSGP43 (#37772), @Tranquil-Flow (#37899).
The dead-session half-open test drives _signal_reconnect with
session=None; the salvaged _ReconnectAdapter assumed a live old
session. Also count set() calls explicitly instead of relying on
MagicMock introspection.
_wait_for_server_session_ready used a time.monotonic deadline; the
circuit-breaker tests freeze monotonic, turning the loop into an
infinite spin (300s SIGKILL in CI-parity runs). Bound by iteration
count instead.
Four independent pre-request stalls sat on the critical path between
prompt submission and the first streamed token, measured with cProfile
against a live process:
1. Discord capability detection (~2.0s, worst 5s): get_tool_definitions
-> _get_dynamic_schema made a BLOCKING https call to discord.com
inside AIAgent.__init__ for any user with DISCORD_BOT_TOKEN set, on
every platform, every cold process. Now non-blocking: memory cache ->
24h disk cache -> permissive default + one background detection that
seeds the disk cache for the next process. The permissive default is
pinned per-process so tool schemas never flip mid-conversation
(prompt-cache safety); it mirrors the existing detection-failure
fallback (all actions exposed, 403s enriched at call time).
2. Ollama /api/show probe (~0.3s): get_model_context_length step 5e
POSTed to <base_url>/api/show for KNOWN providers (openrouter etc.),
got a 404, and never cached the miss - so every fresh process paid a
full HTTP round-trip. Known non-Ollama providers now skip the probe;
local/custom/unknown endpoints keep the exact previous behavior.
3. env_probe subprocess sweep (~0.5s): the Python-toolchain probe ran
4-8 subprocess calls inside the FIRST system prompt build. Now warmed
off-thread during agent init; the prompt build hits the cache (same
lock, so a mid-flight warm just joins instead of recomputing).
4. tools.mcp_tool import (~0.4s): the between-turns MCP refresh in
build_turn_context imported the whole mcp package even with zero MCP
servers configured. MCP tools can only exist if tools.mcp_tool was
already imported (discovery/reload paths), so gate the import on
sys.modules membership - no behavior change for MCP users.
CLI additionally pre-imports run_agent + openai off-thread during the
idle banner window (same pattern as the /model picker prewarm), hiding
the remaining ~1.5s of module imports while the user types. Fixes 1-4
apply to every interaction layer (CLI, gateway, TUI, desktop, cron).
Measured cold first turn (submit -> request dispatched, openrouter,
discord token set): 4.3s before -> 0.9s after CLI prewarm (~80%); the
agent-side non-import cost drops 2.9s -> 0.36s (init) + 0.27s (turn
prologue).
DeepSeek v4 (and other strict OpenAI-compatible providers) reject an
assistant message carrying ``tool_calls: []`` with HTTP 400 "Invalid
'messages[N].tool_calls': empty array. Expected an array with minimum
length 1, but got an empty array instead." Once it hits, every retry on
the session returns the same 400 and the conversation is stuck.
An empty array is semantically identical to "no tool calls", but it
reaches the wire from session resume, host-fed histories, and the
consecutive-assistant merge in repair_message_sequence (which preserves
a pre-existing []). None of the existing sanitizer passes touch it —
they all short-circuit on ``if not tcs`` / ``if msg.get("tool_calls")``.
Fix it at sanitize_api_messages, the final pre-API chokepoint, per the
#56980 review guidance: normalize on the per-call copy (shallow-copy the
message, drop the key) rather than in repair_message_sequence, which
would destructively rewrite the persisted trajectory and prompt cache.
The salvaged commit rewrote update-marker.cjs and its test with CRLF
line endings (Windows editor artifact); restore LF so the diff shows
only the substantive change.
Some MCP servers (e.g. Spring Boot apps with a React SPA) serve their
frontend on any unmatched GET route. The MCP endpoint works perfectly
via POST (JSON-RPC), but a GET to /mcp falls through to the SPA
controller and returns text/html. Hermes's preflight content-type probe
sees HTML instead of application/json or text/event-stream and refuses
to connect.
This adds a per-server config option that
bypasses the content-type probe, letting the SDK connect directly via
POST where it works fine.
```yaml
mcp_servers:
stirling-pdf:
url: http://localhost:8090/mcp
headers:
X-API-KEY: <key>
skip_preflight: true
```
Related: #52460 (OAuth redirect preflight), #51600 (skip probe on mcp add),
#40366 (skip probe on reconnect — already merged).
Some MCP servers (e.g. DocuSeal) serve their web UI on HEAD/GET but
speak Streamable HTTP only via POST. The preflight probe now tries a
lightweight JSON-RPC `initialize` POST before rejecting endpoints
whose HEAD/GET returns a non-MCP content type (e.g. `text/html`).
If the POST returns `application/json` or `text/event-stream` with a
2xx status, the endpoint is accepted. Otherwise the original rejection
behaviour is preserved.
Adds 5 new test cases covering the POST probe path:
- POST rescues HTML HEAD with JSON response
- POST rescues HTML HEAD with event-stream response
- POST still rejects when it also returns HTML
- POST still rejects on non-2xx status
- POST not attempted when HEAD already returns valid MCP content type
Follow-up to #9006/#58899. The gateway routing index (session_key ->
SessionEntry) now lives in a new gateway_routing table in state.db as the
primary store; sessions.json is demoted to an optional legacy mirror.
- hermes_state.py: schema v19 — gateway_routing table (scope + session_key
PK; scope = resolved sessions_dir so multiple stores sharing one state.db
never cross-contaminate) with save/replace/load/delete methods
- gateway/session.py: _save() writes the whole index atomically to the DB
(mirrors the old full-file JSON rewrite semantics) and only falls back to
JSON when the DB write fails; _ensure_loaded reads the DB first and folds
in legacy sessions.json entries for keys the DB lacks (pre-migration
import; DB entries win over stale JSON)
- gateway/config.py + hermes_cli/config.py: new write_sessions_json flag
(default true for compat/downgrade safety); gateway.write_sessions_json:
false stops producing the file entirely
- sessions.json _README updated to say it's a legacy mirror + how to
disable it
Rehydration is now lossless across restarts even with sessions.json deleted:
suspended/resume_pending/model_override/token state all round-trip through
the DB (the old sessions-table recovery only rebuilt the bare key mapping).
Follow-up to the #45545 salvage: the cherry-picked fix duplicated the
~35-line header-shaping block (kimi UA, copilot headers, nvidia NIM,
provider-profile defaults, query params) from the explicit_base_url
branch. Route the main_runtime case through the same block instead —
one client-build path, no drift risk. Also uses _create_openai_client
like the sibling branch instead of constructing OpenAI directly.
When the main agent uses a named custom provider (custom:<name>),
resolve_runtime_provider correctly resolves the base_url and api_key.
But the auxiliary client re-resolves from the bare 'custom' provider
name, losing the provider identity. The bare 'custom' falls back to
OpenRouter, which _resolve_custom_runtime() then rejects — leaving all
auxiliary tasks (title gen, compression, vision, session search, etc.)
with no credentials.
Fix: when resolve_provider_client receives a main_runtime dict
containing concrete base_url + api_key, use it directly instead of
re-resolving. The main agent already solved provider resolution;
the auxiliary client just needs to reuse its answer.
Closes#45472
A Windows venv broken mid-update (e.g. python-dotenv missing after a partial
pip install) still has python.exe + Scripts\hermes.exe on disk.
unwrapWindowsVenvHermesCommand() returned that interpreter with no probe --
bypassing even the caller's --version smoke test -- so every recovery action
(Retry, Repair install, Use local gateway) re-resolved the same dead backend:
ModuleNotFoundError: No module named 'dotenv', same overlay, forever.
- unwrapWindowsVenvHermesCommand now runs canImportHermesCli() on the venv
python (checkout on PYTHONPATH, mirroring isActiveRuntimeUsable) and
returns null on failure so the resolver falls through to the bootstrap
installer, which actually repairs the venv.
- hermesRuntimeImportProbe() adds 'import dotenv' -- the first third-party
import on the CLI boot path (hermes_cli/env_loader.py) -- so a venv missing
python-dotenv fails the probe everywhere it's used (isActiveRuntimeUsable,
system-python rung, and the new unwrap gate).
- Regression tests: probe content + source assertion that the unwrap path
probes and falls through.
Regression tests from PR #51586: the inspection agent must receive the
platform-resolved enabled_toolsets and agent.disabled_toolsets, and a
Blank Slate profile's prompt-size must count exactly the 6 file/terminal
tool schemas.
The `hermes prompt-size` command now uses `_get_platform_tools()` to resolve
platform-specific toolsets the same way the gateway does, and also honors
`agent.disabled_toolsets` from config. This fixes the discrepancy where
`prompt-size` reported more tools than actually available in real sessions
for a given platform.
Fixes#41445.
Two pre-existing tests awaited run() to return after initial-connect
retry exhaustion; with #57477's parking that await hangs (CI: 300s
SIGKILL on slices 4 and 6). Assert the new contract instead: the task
stays alive (parked) and exits on shutdown.
Parking deregisters the server's tools, which removes the only paths
that could ever set _reconnect_event (circuit-breaker half-open probe
and _signal_reconnect both live inside registered tool handlers). A
parked server was therefore unrevivable short of a manual /mcp reload —
the park comment's promised breaker wake could never fire.
Make the parked wait a timed wait: every _PARKED_RETRY_INTERVAL (300s)
the run task wakes and attempts one revival probe, re-parking on
failure instead of burning the full 5-retry budget each cycle. Explicit
reconnect requests still wake it immediately. Idea credit: @Hellbayne
(PR #38881, earliest never-abandon proposal), reconciled with the
park design from #53599.
_discover_tools only filled self._tools; registry registration happened
only in _discover_and_register_server (initial start) and _refresh_tools.
After parking deregistered a server's tools, a revival rebuilt the
transport but published zero tools — a phantom recovery.
Register freshly discovered tools whenever _ready is set and the
registry entry list is empty. Extracted from PR #54139 by @nicha16
(the remainder of that PR reverses the park design and is not taken).
The local retries variable in MCPServerTask.run() accumulated across
transient disconnections — each transport exception incremented it, but
only clean transport returns (auth recovery / manual refresh) or
park-wake reset it. Five transient blips over a long-uptime gateway
would permanently park the MCP server.
Promote retries to instance attribute _reconnect_retries and reset it
at all 4 session-establishment sites in _run_stdio / _run_http, so only
consecutive failures without successful reconnection count toward the
parking budget.
Fixes#57604
Raise the CLI login floor from 180s to 315s (OAuth callback window 300s
+ headroom, matching web_server's existing constant), and let the GUI
re-auth path honor a configured connect_timeout larger than 315s.
Persists as the server's connect_timeout in config, which the probe
now honors. CLI-flag portion of PR #54494; the probe-wrapper portion
was superseded by resolving connect_timeout inside _probe_single_server.
_reauth_oauth_server (hermes mcp login / reauth) called
_probe_single_server without a timeout, so it always used the 30s
probe default — far too short for a human browser OAuth round-trip
(open → sign in → consent → loopback redirect). The server-level
connect_timeout in config.yaml was silently ignored, so login timed
out at ~40s no matter what the user configured.
Pass the server's configured connect_timeout through, with a 180s
floor for the interactive login path. Update the two TestMcpLogin
probe mocks for the new kwarg and assert the login path propagates a
>=180s timeout.
Follow-up to the salvaged #54938: the bounded reader gives a proper 413 +
anomaly telemetry for oversized chunked bodies; client_max_size makes
aiohttp enforce the same 1 MiB cap on every other read path
(#58536/#58902/#59180 pattern). Test fixture's fake Application now
accepts kwargs.
Follow-up to the salvaged #54944: before this, aiohttp's implicit 1 MiB
default client_max_size tripped BEFORE the intended 3 MB Meta cap could
apply on read() paths — the explicit value makes the documented limit
real while the bounded reader keeps chunked bodies from buffering past
3 MB (#58536/#58902/#59180 pattern).
