Review finding: callers mutate the returned dicts in place —
hermes_cli/web_server.py annotates s['enabled']/s['usage'] on the skills
list — so handing out the cached objects poisons the cache for every
subsequent caller (and is a cross-thread shared-mutable hazard in the
gateway). Return [dict(s) for s in cached] on both hit and miss paths;
warm-path cost is negligible (241x speedup retained on a 300-skill
fixture). Regression test mutates a returned list/dict and asserts the
next cached call is clean.
Review findings on the cherry-picked cache (follow-up to #58985):
- The cache key was the max mtime of only the TOP-LEVEL scan dirs.
Adding/removing a skill inside a category subdir bumps the category
dir's mtime, NOT the root's, so the cache served a stale list
indefinitely. Replace with a per-dir signature covering roots +
immediate children (one scandir per dir; mirrors
hermes_cli/profiles.py::_count_skills from d5eee133e).
- The disabled-set is config-driven and changes with no filesystem
mtime bump; fold it into the signature so /skills disable takes
effect without a restart.
- Platform is part of the signature (gateway processes serve multiple
platform scopes; scan results are platform-filtered).
- Add a 30s TTL to bound staleness from in-place SKILL.md edits (file
mtime is invisible to any directory signature).
- The original also keyed dirs off the module-level SKILLS_DIR constant;
the scan itself uses _skills_dir() (live profile HERMES_HOME) — use
the same resolution for the signature.
Mutation-verified: nested-add, disabled-set, and TTL tests fail against
the pre-fix cache and pass with it.
_find_all_skills() re-reads every SKILL.md on every call, which is
wasteful when nothing changed between turns. Cache results keyed by
the max mtime across all scanned skill directories — a skill write
touches the directory, bumping mtime past the cached value and
triggering an automatic re-scan.
skip_disabled True/False are cached separately.
This commit is unstacked from #58984; it carries only the skill
discovery cache change.
(cherry picked from commit cd65673a8f)
The salvaged PR added a standalone test_reasoning_timeouts.py that duplicated
the structure of the existing parametrized test_reasoning_stale_timeout_floor.py.
Fold the v4-flash/v4-pro/-free positive cases and deepseek-chat negative cases
into the canonical parametrized tables and remove the redundant file.
DeepSeek V4 models (deepseek-v4-flash, deepseek-v4-pro) emit
reasoning_content in a separate delta field before final content,
requiring the same 600s stale timeout floor as R1. Without this,
streams hang for 30–50s with APITimeoutError on providers like
opencode-go while direct calls succeed in ~3s.
Fixes#60338.
Phase-2 review finding: the validation branch expanduser()s the path but
web_server.py reads os.environ['HERMES_WEB_DIST'] raw at import — a
'~/dist' value would validate here and still 404 there. Write the
expanded path back before the web_server import. Adds a regression test
asserting the env var holds the expanded path after cmd_dashboard.
Review finding: PR #40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)
Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by #57229 which
already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
behavior change reverting deliberate e8c3ac2f5; belongs in its own
PR with its own justification if pursued.
Conflict resolution: preserved main's newer timestamp-stripping
(#47868) inside the copy-on-write path.
Mock _preflight_user_systemd and _select_systemd_scope in
test_systemd_start_refreshes_outdated_unit and
test_systemd_restart_refreshes_outdated_unit. These tests target
unit-file refresh logic, not D-Bus reachability, so the preflight
check was causing spurious UserSystemdUnavailableError on macOS,
WSL, and Docker where systemd is unavailable.
(cherry picked from commit 34113300a1)
A custom HERMES_WEB_DIST without --skip-build skipped BOTH the web UI
build and any validation: cmd_dashboard fell through the build gate and
started the server against a dist that may not exist, serving 404s with
no obvious cause. This is the same failure mode issue #23817 fixed for
the --skip-build branch — the env-var branch was left unvalidated.
Add the missing else-branch: fail fast with actionable guidance when
HERMES_WEB_DIST has no index.html, proceed (still without building) when
it does.
Credit: @Caelier (#17845) originally proposed dist validation for the
dashboard startup path; the --skip-build half of that PR's scope has
since landed via the #23817 fix, this covers the remaining env-var path
on the rewritten cmd_dashboard surface.
Two async handlers still called the cron profile-walk helpers directly on
the FastAPI event loop after the 49fa04a23/346e5673d threadpool migration:
- POST /api/cron/fire called _find_cron_job_profile() inline — it walks
every profile and lists its jobs (file I/O per profile), stalling the
loop before the 202 is returned.
- POST /api/cron/blueprints/instantiate called _call_cron_for_profile()
inline for create_job.
Route both through the existing _run_cron_dashboard_io threadpool wrapper
like every other cron dashboard endpoint.
Credit: @riceharvest (#50948) originally identified the sync-I/O-in-async-
handlers bug class for the desktop boot endpoints; 49fa04a23, 346e5673d,
7d0ddbb2f, d5eee133e and 24d5bda1e have since fixed most of that PR's scope
via the managed threadpool + PID cache + alias-map surfaces. This covers
the two cron handlers those merges missed.
Review finding: the substring check ('kimi' or 'moonshot' in model)
under-matches bare release slugs like k2-thinking that the repo's
canonical _model_name_is_kimi_family matcher (anthropic_adapter.py)
already covers. Reuse it instead of a second ad-hoc matcher; add a
regression test for the bare-slug case.
Adapted from PR #26014's test file to the canonical seam
(tests/run_agent/test_anthropic_prompt_cache_policy.py _make_agent
helper) instead of a new top-level file with sys.path manipulation.
Covers: kimi-k2.6 + moonshot-v1 on OpenRouter (envelope layout),
kimi via Nous Portal, and the non-OpenRouter negative case.
Kimi/Moonshot models on OpenRouter honour the same envelope-layout
cache_control markers as Claude on OpenRouter, but the policy fell
through to (False, False) — serving ~1% cache hits on 64K-token prompts
and re-billing the full prompt every turn. Observed within-turn
progression with cache enabled: 1% -> 67% -> 84% -> 97%.
(cherry picked from commit 3b857a35e, agent/agent_runtime_helpers.py hunk only;
the original PR #26014 branch also carried unrelated .dev-workflow artifacts
which are intentionally not included)
Structured review (2a/2b/2c) findings, all fixed:
- MAJOR: detect_local_server_type memo was process-lifetime with no
invalidation, permanently pinning a URL's server type. Now a bounded
1h TTL ((type, monotonic) tuples) so a backend swap on the same port
is re-detected. Test covers ollama->lm-studio swap after expiry.
- MAJOR: legacy disk-row compat was one-way. get_cached_context_length
and _invalidate_cached_context_length now consult the same key-shape
set {canonical, literal, canonical+slash} in both directions, so an
old slashed row is found (and cleared) when the runtime passes the
normalized URL. Tests pin both migration directions.
- MINOR: _localhost_to_ipv4 did whole-string replacement, which could
corrupt a proxy URL embedding http://localhost in its query. Now a
scheme-anchored host-only regex; localhost.example.com and embedded
substrings pass through. Tests added.
- MINOR: _invalidate_cached_context_length now also drops the
in-memory TTL probe rows for the pair, so a resolution inside the
TTL window can't re-persist the value just declared stale.
- Test gaps closed: detect-type cache hit + TTL-expiry re-detection,
ollama-show TTL expiry re-probe, reverse legacy-row lookups.
- Attribution gate: added zhchl@hermes-agent.local -> 8294 (PR #50572
author) to AUTHOR_MAP; the strict CI grep needs bare non-plus emails
literal in release.py.
Gates: ruff clean; targeted suites 202 passed / 0 failed; full
tests/agent 5426 passed with 17 failures identical on clean
upstream/main (pre-existing env-dependent anthropic/bedrock/credpool
tests); mypy delta vs base: 0 new errors; live smoke 6/6 PASS.
#37595 fixed the Windows dual-stack IPv6 timeout only inside
detect_local_server_type. The same 2s-per-probe penalty existed at every
other helper that builds a probe URL from base_url. Extract the rewrite
into _localhost_to_ipv4() and apply it at:
- query_ollama_num_ctx
- query_ollama_supports_vision
- _query_ollama_api_show (server_url derivation)
- _query_local_context_length (server root + LM Studio native URL)
Tests cover the helper's URL forms, non-localhost passthrough, and that
the ollama probes actually POST to 127.0.0.1.
Follow-up hunks completing the probe-cache cluster:
1. _query_ollama_api_show now goes through the existing
_LOCAL_CTX_PROBE_CACHE (30s TTL, positive-only, namespaced key) —
it was the one remaining per-resolution POST not covered by the
#56431-era wrapper. Failures are never memoized so a server that
comes up mid-startup is re-probed. Idea credit: #42081 (@Morad37),
reworked to comply with the positive-only rule.
2. Persistent context-cache keys are normalized through
_context_cache_key (trailing-slash strip) so http://host/v1 and
http://host/v1/ share one entry; reads and invalidation honor
legacy un-normalized rows. Idea credit: #37905 (@stevenau21).
Tests: TTL hit collapses to one POST, failure-not-memoized
(mutation-verified: unconditional caching makes it fail), namespace
no-collision vs the sibling probe, slash-variant dedup, legacy-row
read, dual-shape invalidation.
_cli's show_banner() calls _resolve_active_context_length() at every
startup. For non-OpenRouter providers (e.g. minimax-cn, kimi-coding,
custom endpoints) the resolver falls through to step 6 (OpenRouter
live /models fetch), which blocks ~2-3s per CLI launch and adds up
to 7+ minutes when openrouter.ai is unreachable through a proxy that
403s CONNECT (#46620).
Two complementary changes:
1. model_tools.py: read model.context_length from config.yaml and pass
it as config_context_length to get_model_context_length. The
step-0 config override short-circuits the entire resolution chain
including the OpenRouter fetch. No network call is made when the
user has set the value explicitly.
2. agent/model_metadata.py: replace flat timeout=10 with (5, 10)
tuple at all five sites (fetch_model_metadata + four endpoint
probes). urllib3 can otherwise block for 10s per retry stage
through proxies that 403 CONNECT. The tuple bounds connect at 5s
while still allowing slow reads.
Complements the in-flight PR #46685 (which adds HERMES_DISABLE_MODEL_METADATA
env var + same timeout tuple change for fetch_model_metadata). This PR
extends the timeout fix to the other four endpoint probes and adds the
config-override path that addresses the slow-but-reachable scenario
where env-var disable is too heavy-handed.
Refs #46620, PR #46685.
(cherry picked from commit e7faa34199)
Remaining hunk of the PR-branch fixup commit: the LM Studio first-probe
assertion now expects the IPv4-resolved URL (the code half — applying
the rewrite to `normalized` before deriving lmstudio_url — was folded
into the previous cherry-pick's conflict resolution).
(cherry picked from commit 7d324b0e47)
On Windows, `localhost` resolves to both ::1 (IPv6) and 127.0.0.1 (IPv4).
httpx tries IPv6 first, hanging 2 sec per probe when the server binds IPv4
only. detect_local_server_type() is called 3+ times during init, each with
a new httpx.Client, compounding to ~14s of dead time.
Replace localhost with 127.0.0.1 inside the function before connecting.
The function is only called for local endpoints (callers guard with
is_local_endpoint()), so IPv6 loopback adds no diagnostic value.
Measured: 19.9s → 4.0s on Windows with a local proxy on 127.0.0.1:8317.
(cherry picked from commit a075d3194b)
Every 5 minutes fetch_endpoint_model_metadata() re-runs the full server-type
waterfall (LM Studio -> Ollama -> llama.cpp -> vLLM), spraying 404s at
endpoints the server never exposes (e.g. /api/v1/models and /api/tags on a
vllm backend).
Add _endpoint_probe_path_cache (base_url -> server type) so the first
successful probe's result is reused for the lifetime of the process.
Subsequent refreshes skip straight to the known-good path.
Fixes#29971.
(cherry picked from commit f3d7a8960a)
Follow-up hardening on the salvaged NS-591 mobile-chat reconnect fix, from
review findings:
- Guard the page-resume reconnect against the async socket-open window:
a connectInFlightRef is set synchronously before the ticket-URL await so
a visibilitychange/focus fired during that gap (wsRef still null) can't
spawn a redundant second socket. Threaded through
shouldReconnectPtyOnPageResume as connectInFlight.
- Recover a socket wedged in WS_CONNECTING (half-open mobile socket after a
radio handoff — the NS-591 scenario) via a PTY_CONNECTING_TIMEOUT_MS
force-close so onclose routes into scheduleReconnect. Cleared on
open/close/effect-cleanup.
- Avoid collapsing legitimate single-letter reduplication ("a a") in the
mobile duplicate-final-word heuristic (>=2-char guard).
- Extract the 350ms replacement window and 1000ms resume throttle to named
exported consts; drop the dead WS_CONNECTING term from the resume
predicate's final expression.
Adds tests for the in-flight guard and the single-letter reduplication case.
Two review fixes on the mobile input normalization path:
- updatePtyInputLine appended the printable payload of escape sequences
(the '[D' of a left-arrow) to the tracked line, and after any cursor
movement the flat tracker no longer matched the visual line — the
DELETE-repeat replacement could then be computed against a stale
snapshot. Any chunk containing ESC now resets the tracker, disarming
replacement normalization until a cleanly-tracked line starts.
- Move the SGR mouse-report filter ahead of the blocked-input check so
scrolling a disconnected terminal doesn't print the reconnect notice.
Follow-up to HexLab98's fix. The sync-before-regenerate invariant was
enforced by convention across 6 callsites (~3 idempotent unit-file reads
per command). Consolidate it into the one function every compare/regenerate
path funnels through — systemd_unit_is_current — and drop the now-redundant
callsite pre-syncs in refresh_systemd_unit_if_needed / systemd_start /
systemd_restart / systemd_status.
Kept the systemd_install pre-sync: the --force path bypasses the
is_current gate and calls generate_systemd_unit() directly, so it needs
its own sync to avoid baking /root/.hermes under sudo.
Reworked the two callsite-ordering tests into a chokepoint-invariant guard
(test_is_current_syncs_before_reading_unit) + a delegation test proving
start/restart no longer pre-sync. Both fail if the chokepoint sync is
removed; the pre-existing behavior test still passes.
Under sudo, start/restart refreshed the unit from /root/.hermes before
adopting the unit's pinned home, so TimeoutStopSec and env drifted and
status stayed stuck on "service definition is outdated".
Follow-up to the salvaged str(tools) fix. The id()-keyed
_TOOLS_TOKENS_CACHE had no eviction, so a long-lived gateway/desktop
backend could accumulate an unbounded number of stale entries as it
builds transient tool lists. Cap it at 256 with oldest-first eviction
(insertion-ordered dict) and add a regression test asserting the cache
never exceeds the cap.
Estimate tool-schema size without repeatedly stringifying full tool lists, and cache the result per tool snapshot to reduce GIL-heavy work during preflight and compaction.
PR #61281 removed the client-side X-Hermes-Session-Token requirement from the
dashboard OAuth mutation calls so cookie-authenticated hosted/mobile sessions
can start provider logins. That change is safe only because the server still
gates those endpoints (gated_auth_middleware cookie check + _require_token).
The PR's api.test.ts suite mocks fetch and only asserts client behavior, so a
re-break of the gated-mode cookie gate would pass CI unnoticed.
Add gated-mode TestClient tests asserting POST /api/env/reveal and the OAuth
mutation endpoints (disconnect/start/submit/cancel) return 401 with no session
cookie. Mutation-verified: neutering both the middleware gate and _require_token
flips all five to 200.
The new oauth.copyCode/copyFailed keys existed only in en.ts, with
optional types and English literal fallbacks in OAuthLoginModal — so
non-English users got English strings on the device-code copy button.
Backfill translations in all 16 non-English locales, refresh the
updated oauth.description/notConnected copy (dashboard Login flow
mention) to match en.ts, make the two keys required in the
Translations interface, and drop the English fallbacks from the modal.
Verified with web tsc --noEmit (required keys enforce locale
completeness), vitest, and a web build.
Electron 40 ships Node 24.15, where tsx's ESM load hook returns null and
crashes with ERR_INVALID_RETURN_PROPERTY_VALUE. Bundle main+preload via
esbuild for `npm run dev` and always load the JS preload from dist/.
grok-4.5 is GA and is now the single curated Grok entry on the
aggregator lists. grok-4.3 is NOT retired upstream — it remains fully
usable by typing the model name (validated against the live catalogs);
this only removes it from the short curated picker snapshots. The
xAI-direct list is models.dev-cache-driven and unaffected.
The model often emits a follow-up batch of tool calls as its own
assistant message with no prose or reasoning. On screen those rows look
like one continuous run, but assistant-ui only groups tool calls within a
single message, so the auto-scrolling tool window never triggered on them
(e.g. two batches of two searches read as 2 + 2, never reaching the
threshold).
Coalesce each settled tool-only assistant message into the preceding
assistant message in the render pipeline so its calls join that message's
tool group. Render-only (never touches the $messages store) and
settle-only (pending messages are skipped) so a live turn is never
merged/un-merged mid-stream; merged results are cached by source identity
so a stable turn yields stable objects with no re-render churn.
- Return the boundary snapshot from
_launch_session_boundary_memory_flush as a local value instead of
staging it on self._session_boundary_snapshot. The instance-attr
handoff could leak (no memory manager configured) or mis-fire a
stale snapshot on a later /new if an exception hit between staging
and consumption. A local variable eliminates the class; the helper
also returns None when no memory manager is configured so
new_session takes the inline-switch path.
- Drop the now-dead session_id kwarg from commit_memory_session:
after the redesign no production caller passes it (gateway, TUI,
compression all use the default), and speculative params are
rejected per AGENTS.md. The explicit-old-session need is served by
cli.py's direct engine call + commit_session_boundary_async.
- Drop the dead providers snapshot in commit_session_boundary_async
(only the emptiness check used it).
- Tests updated accordingly (dead-kwarg test removed, snapshot
assertion now covered by return-value contract).
Phase-2 gates: 2a tests/cli 1048 passed + 6 memory files 137 passed;
2b programmatic live smoke 0.38ms non-blocking caller, end→switch→sync
ordering verified; 2c structured 4-angle review — no Criticals, these
warnings fixed.
Deep review of the cherry-picked #16454 found the ad-hoc flush thread
raced new_session()'s inline on_session_switch(reset=True): memory
providers key off internal _session_id state (MemoryManager.on_session_end
takes no session id), so a late off-thread extraction ran against
post-rotation bindings — misattributing the old transcript to the new
session id, double-ingesting the old turn buffer (supermemory), or
double-committing (openviking already async-finalizes in
on_session_switch).
Redesign: new MemoryManager.commit_session_boundary_async queues
on_session_end + on_session_switch as ONE task on the manager's existing
single-worker background executor (the same worker sync_all already
uses). This preserves the strict end→switch ordering providers depend on,
serializes against per-turn syncs FIFO, keeps /new non-blocking, and
degrades to inline (pre-#16454 behavior) when the executor is
unavailable. No ad-hoc threads; no per-provider changes needed.
The context-engine on_session_end half stays synchronous in
_launch_session_boundary_memory_flush (cheap, must land before
reset_session_state rebinds the engine).
Exit durability: _run_cleanup calls the manager's existing
flush_pending(timeout=10) barrier before shutdown, so '/new then quit'
doesn't drop the queued extraction (shutdown_all's own drain is ~5s and
cancels queued tasks). Bounded well inside the 30s exit watchdog.
Tests: ordering invariant with slow (LLM-like) extraction, FIFO
serialization vs sync_all, switch-fires-even-if-end-raises, no-provider
no-op, CLI snapshot handoff + inline-switch fallback, sync engine
boundary, cleanup flush_pending.
