Phase 2 review findings on the salvage branch:
C1 (critical): batch and micro summary markers share
COMPRESSED_SUMMARY_METADATA_KEY, and compress() never reset micro state.
After micro absorbed exchanges 1..k, a batch compaction summarizing
1..m (m>k) could fire; the next micro pass's supersede then dropped the
batch marker (whose content the stale rolling summary does NOT contain)
and archive_and_compact immediately made the loss durable. Defrag had
the same hazard: it rewrote "the newest marker" even if that was a
batch marker. Empirically confirmed with a probe (batch marker content
destroyed in one pass).
Fix, three parts:
- Micro-created markers now carry MICRO_COMPACT_MARKER_KEY; supersede
and defrag only ever touch micro-tagged markers. Rehydration in
_resolve_compact_cursor tags the marker it absorbs (containment
proof), which safely covers adopting a batch marker as the new
rolling base after a reset.
- compress() success path resets micro rolling summary/cursor state so
a stale summary can never claim cumulativeness over a batch marker.
- Regression tests for both directions plus the reset.
W4: _splice_micro_compact_result no longer strips _db_persisted stamps
from surviving messages. Micro archives in place under the SAME session
id (unlike batch's child-session rotation, #57491), so surviving stamps
are accurate; stripping them meant an archive_and_compact failure left
every previously-persisted message unstamped and the next append-only
flush re-inserted them all as duplicate active rows.
W5: finalize_turn micro gate now checks agent._persist_disabled —
persistence-isolated fork agents (background review) must not burn an
aux call per review turn, and must never archive_and_compact the
canonical session rows if their compressor ever gains a DB binding.
W1: _serialize_one_exchange now delegates to _serialize_for_summary
(was a ~70-line near-verbatim copy; one serializer, one place to fix).
S4: _find_one_exchange boundary guard rejects only assistant/tool
boundaries (the actual alternation hazard) instead of requiring user —
a stray mid-list system/injected message can no longer wedge the
cursor forever.
5 new regression tests; 38 micro/prune tests, 400 compression-suite
tests, 61 finalize/persist tests pass; ruff clean.
Two integration bugs found during review of #74522, both confirmed with
empirical probes against the production message-repair path:
1. Alternation: the summary marker was role="user" and an exchange was a
single assistant+tools group, so splicing between two user turns produced
user -> marker(user) -> user. The pre-request repair_message_sequence pass
(conversation_loop.py, runs before EVERY API call) then merged the marker
into the neighbouring real user message: metadata gone, cursor
unrecoverable on resume, and the summary text duplicated into the
transcript on every later pass (the transcript GREW every turn).
Fix: an exchange is now a full agent turn (assistant + tools + follow-up
assistant iterations, bounded by user messages), the marker is
assistant-role, and superseding an old marker deliberately merges the two
adjacent real user turns (plain-text \n\n-join, identical to repair
pass 2) so the returned transcript is alternation-valid by construction.
Probe result: repairs 0 (was 2), marker survives, no summary leakage.
2. Defrag destroyed user messages: _defrag_rolling_summary serialized the
whole remaining middle (user turns included) and spliced it away —
8 of 10 user prompts destroyed in one pass, contradicting the feature's
"your messages are never compacted" invariant. Fix: defrag now
re-summarizes only the rolling summary TEXT and rewrites the marker
content in place; transcript shape, cursor, and user turns untouched.
Probe result: 10 of 10 user prompts survive.
Also: marker provenance is now COMPRESSED_SUMMARY_HAS_USER_TURN_KEY=False —
micro markers absorb only assistant/tool content (#64650 invariant), and
real user turns remain in the transcript for provenance detection.
Adds 5 regression tests (repair-pass integration, alternation on
multi-iteration tool turns, defrag user survival, defrag input scope,
marker provenance); updates the two existing tests and the design doc to
the corrected semantics. 28 tests pass.
The on/off switch was the only knob. A pass fired after every completed turn,
absorbed exactly one exchange, and there was no way to ask for less. Since a
pass is also what breaks the prompt-cache prefix, "how often does it run" and
"how often do I pay a cache break" are the same question, and it had no answer.
Add `compression.micro_compact_every_n_turns` (default 1, clamped to >= 1). At 1
the behaviour is what it was; at 5 you get a fifth of the breaks and a fifth of
the reclaim rate. The counter advances per invocation rather than per committed
pass, so a turn that finds nothing to absorb still moves the cadence along and
cannot wedge it, and a bogus 0 or negative degrades to "every turn" instead of
silently disabling compaction.
Also expose `micro_compact_defrag_threshold_tokens`, which has been a hardcoded
attribute on the compressor with no path from config since it was added.
This does not give micro-compaction the prune's reclaim-size gate -- a pass
still commits whatever the single absorbed exchange saved. It makes the break
frequency tunable, which reaches the same end by absorbing less rather than by
waiting for a bigger win. The docs now say that plainly, including that a
reclaim threshold is the obvious follow-up and does not exist yet.
Tests cover the skip-until-due window, the cursor and prefix staying untouched
on skipped turns, the clamp, and that the feature is off unless enabled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rolling summary lives only in memory. A resumed session starts with an
empty one while the marker carrying every previously absorbed exchange is
still in the transcript. The first pass after a resume therefore built a
marker from a single exchange and superseded the marker holding the entire
history -- silently discarding everything micro-compaction had accumulated.
This was introduced by the supersede fix. Before it, markers piled up
wastefully, but nothing was ever lost.
Two changes, so a single failure cannot lose data:
Rehydrate. When the cursor is recovered by scanning the transcript -- the
resume path -- also recover the rolling summary from that marker, so the
next pass merges into the existing history instead of replacing it.
Extraction uses rfind for the heading because SUMMARY_PREFIX references the
heading text itself, so the first occurrence is inside the preamble.
Gate superseding. Earlier markers are dropped only when this pass's summary
is demonstrably cumulative, i.e. the rolling summary was non-empty going in.
If rehydration ever fails, the pass keeps both markers: wasteful, but the
history survives.
Tests cover the resume path, the failed-rehydration fallback, and the
round trip of a summary through a marker.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cursor was set to the pre-splice `exchange_end`. A splice collapses the
absorbed span -- an assistant plus its tool results, often four or more
messages -- into a single marker, and may also drop a superseded marker
further back, so every index after it shifts.
The stale cursor therefore overshot, landing inside a *later* exchange's
tool group. The next pass's `_find_one_exchange` walked forward from there
to the following assistant, so the exchange it had landed inside was never
absorbed at all. On tool-bearing conversations micro-compaction was
silently doing roughly half the work it should.
Traced on a 3-tool-per-exchange transcript: the cursor sat at index 6 when
the marker was at 2, and the message count stalled at 32 instead of
continuing to 28.
Derive the cursor from the marker's actual position in the spliced result
instead, which is self-correcting regardless of how much the splice moved.
Apply it on the defrag path too, which had the same staleness.
Found by a randomized long-horizon harness (480 conversation shapes x 25
passes, varying tool-group sizes and summarizer failure modes) asserting
structural and progress invariants after every pass. Existing tests missed
it because their fixtures have no tool results, so the absorbed span is one
message and nothing shifts. The regression test uses tool groups.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tokens saved is the wrong headline for this feature. Micro-compaction is
not an efficiency optimisation — the same summarization work happens either
way. What it buys is (a) that work amortized across turns instead of one
stall, and (b) a window kept low enough that a session runs much further
before needing a hard compaction at all.
Neither shows up in "net tokens saved". A session can save nothing on paper
and still be a clear win on both counts.
So the telemetry now carries occupancy: tokens_after as a share of the
compaction threshold, plus the threshold and resolved window it was
computed from. That is the number that says whether a session has headroom
left. The report leads with it, and cross-references the batch
`compression_attempt` lines already in the log so it can show how often the
long pause actually fired — ideally never.
Occupancy is read from the cached threshold only. The public
`threshold_tokens` property resolves lazily and can issue a synchronous
/models probe (#32221); telemetry must never be the thing that blocks a
turn, so an unresolved window reports null. In practice a pass has already
resolved it via the tail calculation, so the field is populated. A test
pins the no-forcing behaviour directly against the emitter.
The report is pure ASCII: `scripts/check_subprocess_stdin.py` currently
dies on a cp1252 console before printing its results, and a diagnostic tool
that crashes on the platform it is diagnosing is worse than no tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The existing log line reports message counts, which is the least
informative number available here: absorbing one tool-heavy exchange can
drop hundreds of tokens while moving the count by one. There was no way to
answer "is this actually helping?" from a real session.
Emit one content-free JSON line per pass, in the same shape as the batch
compaction telemetry: before/after tokens, the delta, the size of the
absorbed exchange, the rolling summary size, duration, and running
per-session totals so a whole run can be read off the last line. No
transcript content rides along.
Add scripts/micro_compaction_report.py to aggregate those lines into
passes, outcome mix, net tokens saved, mean exchange size and durations,
with an optional per-session breakdown.
Measuring it immediately surfaced something worth documenting: the first
pass in a session normally *costs* tokens. The summary marker carries a
fixed ~400 tokens of scaffolding, paid on pass one against a single
absorbed exchange. From pass two the marker is replaced rather than added,
so the overhead is already paid and each exchange is close to pure saving.
Break-even is typically the second or third pass. Tests cover the
telemetry contract, the cumulative totals, and that first-pass/later-pass
shape so nobody reads a single turn and concludes it made things worse.
The estimator costs ~5 ms at 600 messages and ~20 ms at 1200, taken twice
per pass, post-turn — and only once an exchange is actually in hand, so
turns that no-op early pay nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`_find_one_exchange`'s docstring described an exchange as "(optional) user
message + assistant message + its tool results", but the walk skips past
user messages and starts at the assistant, so user turns are never absorbed
into the rolling summary.
The code is right and the docstring was wrong. Assistant output is largely
an account of what was done and survives summarising with little loss. The
user's messages are the intent everything else is derived from and cannot be
reconstructed from the work that followed — paraphrasing "use the existing
helper, don't add a new one" into a summary is how an agent ends up doing
the opposite six turns later. They are also cheap: a prompt is normally a
tiny fraction of what one tool result costs.
Correct the docstring, document the property (and its cost — a floor on how
small the middle can get, since user turns accumulate), and add a test so it
stays deliberate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Batch compaction pauses a session for one large summarization once the
window fills. Micro-compaction spreads that cost out: after each completed
turn, `finalize_turn` folds the single oldest un-absorbed exchange
(assistant message plus its tool results) into a rolling summary, so the
work happens in small increments during post-turn idle time instead of one
long stall.
Mechanics:
- a cursor tracks the first message not yet absorbed, recovered from the
transcript's last summary marker when in-memory state is unavailable;
- protected head and tail windows are never touched, so the system prompt
and recent turns stay verbatim;
- the absorbed span is replaced by a marker carrying the usual
`_compressed_summary` metadata, so resume, handoff and `/compress`
treat it exactly like a batch summary;
- `archive_and_compact` keeps the session DB in step, otherwise the
append-only flush would leave the original rows active and a resume
would double-load both summary and originals;
- when the rolling summary itself passes a token threshold it is
defragged: re-summarized in one shot and the cursor jumps to the tail;
- an exchange the summarizer can't handle is retried a bounded number of
times, then skipped, so one poison exchange can't stall every turn.
Keep only the newest summary marker. The rolling summary is cumulative, so
each marker already contains everything the previous ones held; leaving them
stacked near-duplicate copies of the same text, each with its own heading and
end-marker scaffolding, and the transcript grew on every turn instead of
shrinking. Measured over six turns on a 12-exchange conversation with tool
output: 4104 -> 4797 tokens before, 4104 -> 2572 after.
Off switch: `compression.micro_compact: false` (default on).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The compaction summary's role was selected against the LITERAL
neighbouring messages (compressed[-1] / tail_messages[0]). Mistral-family
chat templates (Devstral, Mistral Small 3.x, Magistral) enforce
user/assistant alternation but exempt the tool flow (tool results and
assistant messages carrying tool_calls) from the check, so a protected
head ending [user, assistant(tool_calls), tool] pinned the summary to
role="user" while the last role the template counts is "user": the
backend rejects the whole request with a Jinja alternation error
(HTTP 500). The summary persists in the stored conversation, every
retry replays the identical poisoned history, and the session is
permanently unrecoverable. Fires on EVERY compaction against a
Mistral-strict backend, captured byte-exact via a tee-proxy in front of
a llama.cpp/llama-swap Devstral deployment.
Fix: compute both neighbour roles through _template_visible_role(),
which skips template-exempt messages. The #52160 (Anthropic user-first)
and #58753 (zero-user-turn) forced-user guards are preserved; their
forced shapes (summary-user followed only by exempt messages) are
alternation-safe. When the visible head ends "assistant" and the
visible tail opens "user", no standalone role can alternate and the
existing merge-into-tail fallback now correctly fires (the literal
logic emitted a standalone user summary there: a second poisoning
shape).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGN45sMMbwM8cW9T9ga4ou
Replace OS-scheduler-dependent elapsed-time ceilings with deterministic
concurrency proofs in three tests:
- test_context_refs_concurrent: asyncio.Barrier rendezvous — all 3 URL
fetches must be in flight simultaneously before any returns.
- test_memory_boundary_commit: positive non-blocking witness — the
provider call list must still be empty when the async commit returns.
- test_mem0_v3 slow-prefetch: threading.Event park/release — prefetch
must return while the backend search is still parked.
The tests/tools/test_mcp_tool.py hunk from the original PR is dropped:
main no longer carries the 'elapsed < 2.5' assertion it targeted
(superseded by a delay-relative bound).
Salvaged from #71913.
Re-port of PR #35464 onto the rewritten hermetic conftest:
- autouse _neutralize_webbrowser fixture records open/open_new/open_new_tab
and webbrowser.get() instead of launching a real browser
- autouse _neutralize_macos_keychain_creds defaults the Anthropic Keychain
reader to None, with an opt-in allow_macos_keychain marker
- regression tests in tests/test_hermetic_side_effect_guards.py
- tests/agent/test_anthropic_keychain.py opts in via pytestmark
Salvaged-from: #35464
Co-authored-by: y0shualee <yuxiangl490@gmail.com>
The restart-routing, systemd-support, and subprocess-HOME tests asserted
branch behavior but left part of the real probe surface unmocked, so they
fail when the suite itself runs inside a container (self-hosted CI) or a
launchd-descended shell:
- /restart routing tests: the handler also consults the real /.dockerenv —
extract the inline probe to gateway.restart.is_container_restart_context()
(patchable seam, no behavior change) and pin it False; scrub ALL four
supervisor env markers (ambient XPC_SERVICE_NAME on macOS flipped one).
- supports_systemd_services tests: pin shutil.which('systemctl') and
is_container() so the test asserts the branch, not the host.
- copilot ACP real-HOME test: pin is_container() (auto mode prefers profile
home in containers) and scrub ambient HERMES_REAL_HOME/TERMINAL_HOME_MODE.
97 tests green on macOS dev box AND inside a docker CI runner container.
Co-authored-by: Kyzcreig <9063726+Kyzcreig@users.noreply.github.com>
_fresh_modules() in test_vision_routing_31179.py deleted agent.image_routing
(+siblings) from sys.modules and never restored the ORIGINAL modules — the
reloaded copies leaked. Any later same-process test holding function refs to
the original module (test_image_routing.py) then patched the reloaded copy
via mock.patch string targets, making patches invisible: order-dependent
failures (repro: 31179 file first → 2 failures; reversed → green).
Autouse fixture now snapshots the affected sys.modules entries and restores
them on teardown. Both orders + solo runs verified green.
Masked by the per-file-isolated canonical runner; bites anyone running
pytest tests/agent/ directly.
Main's 243c9182b1/a16fd675df/7142dc4580 added load_config_readonly
sibling stubs across 38 files; our pruned versions of 11 of those files
kept only the load_config stubs. Re-applied the pairing at every
surviving site (26 patch()/setattr sites) — same return_value/
side_effect as the adjacent load_config stub. 494 tests green across
the 11 files.
Second stub shape the first sweep missed: monkeypatch.setattr(config_mod,
"load_config", ...) — attribute-form instead of string-form. Four files
(compression_max_attempts, preflight_compression_cap_e2e,
codex_gpt55_autoraise_notice, proactive_prune_config) stub agent_init
config reads that now go through load_config_readonly(). Swept the whole
tree for the attribute form; remaining hits stub modules that still use
the mutable loader. 1,210 tests green across the 19-file re-sweep.
The readonly swaps mean agent/ modules now read config through
load_config_readonly(); tests that stubbed only load_config stopped
intercepting those reads. Added sibling readonly stubs (same
return_value/side_effect/lambda) at every affected site across 11 test
files, found via a tree-wide sweep of load_config stubs cross-referenced
against the swapped modules. 3,915 tests green across the 38-file
sibling sweep.
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
Resolves the PR's conflict with main (2252 commits). Two conflicts, both
"each side added an independent block in the same place" — kept both:
- gateway/run.py — the housekeeping loop. This branch adds the Skill Sync
pulls inside the CURATOR_EVERY branch (12-space indent); main adds a
stale-session auto-archive as a sibling `if` at loop level (8-space).
Different scopes, so the naive union would have mis-nested the archive
block into the curator branch; kept each at its own indent level.
- tools/skill_manager_tool.py — the _edit_skill result dict. This branch
appends the org auto-propose note; main appends
_add_description_prompt_preview(). Independent, order-insensitive.
No behaviour dropped from either side.
Verified: 3552 passed / 0 failed across 63 suites (scope regenerated to
include main's new maybe_auto_archive / _add_description_prompt_preview
consumers) via scripts/run_tests.sh. `hermes sync` and `hermes sync status`
still work against a live token, resolving the production plane default.
The Pyright Optional-parameter warnings in skill_manager_tool.py are
pre-existing on main (`content: str = None` etc.), not introduced here.
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
Three provably-safe optimizations for O(n)-per-iteration history walks:
1. sanitize_tool_call_arguments: optional identity-keyed cursor (strong
refs to the exact validated message objects) skips re-json.loads-ing
already-validated history each loop iteration. Any list rewrite
(compression, repair, undo, steer) breaks the identity prefix match
and forces re-scan from the divergence point. Wired via a per-agent
cursor dict in conversation_loop.
2. estimate_messages_tokens_rough: per-message memo keyed on a deep
identity fingerprint (strings pinned by strong reference so id()
aliasing is impossible; scalars by value; dicts/lists structurally
with key order). Equal fingerprints imply identical str(shadow)
bytes, hence identical estimates. Unfingerprintable shapes fall
through to direct compute. Bounded FIFO cache (4096 entries).
3. _flush_messages_to_session_db_unlocked: bounded scan that skips the
identity-matched prefix of the previous successful flush's snapshot.
Snapshot only taken on full success; cleared on exception. Compression
rewrites use fresh copies, breaking identity and forcing full re-scan.
Parity proven in tests/agent/test_cursor_optimizations_parity.py:
500-message synthetic histories with tool calls, malformed args, unicode,
element-wise old==new across 3 iterations incl. simulated compression.
Measured (median of 5): sanitize 0.097ms->0.011ms, tokens 1.145ms->0.853ms,
persist-scan 179.5us->10.0us at 500 messages.
The TTL-expiry test aged only the in-process probe cache; with the new
disk L2 the fresh disk verdict (correctly) served 'ollama' and the swap
to lm-studio wasn't re-detected. In real time-flow the 300s disk TTL
always lapses before the 1h in-proc TTL — the test now compresses both
expiries, matching the scenario it describes. 25/25 pass.
Local-model users paid a fresh probe waterfall on EVERY CLI cold start
inside AIAgent.__init__: detect_local_server_type (up to 4 HTTP GETs,
2s timeout each on a hung server) + /api/show (3s timeout). The
existing caches were in-process only, so back-to-back invocations
(chat -q, cron ticks, subagents) re-paid the network every time.
- New 300s-TTL disk L2 at HERMES_HOME/cache/local_endpoint_probes.json
for detect_local_server_type verdicts and query_ollama_num_ctx
results. Only SUCCESSFUL probes persist (a down server never pins a
negative verdict); stale entries pruned on write; corrupted cache
degrades to a miss; atomic writes. 300s is strictly fresher than the
1h in-process TTL that already accepts server-swap staleness.
- models.dev fetch timeout 15 -> (5, 10) connect/read tuple: a
blackholed connect stalled the first-turn critical path 15s; now
fails in 5s (matches the OpenRouter fetch convention, #46620).
- _auto_detect_local_model timeout 5 -> (2, 3): runs inside
_get_model_config() at startup against a LOCAL endpoint; a hung local
server cost 5s before the banner.
E2E (real HTTP server, two fresh subprocesses, isolated HERMES_HOME):
proc1 = 2 HTTP hits, proc2 = 0 HTTP hits, identical results
(ollama/131072), probe wall 74.5 -> 35.5 ms. 222 targeted tests green
incl. 9 new disk-L2 contract tests.
The streaming hot loop computed len(repr(chunk)) on EVERY chunk to feed
the retry-diagnostic byte counter — a full recursive pydantic repr at
5.5-8.8 us per chunk (measured), ~20-30 ms of pure CPU per 3,000-chunk
response, paid on every streaming response on every platform.
New _estimate_chunk_bytes() sizes the chunk from its delta payload
strings (content / reasoning / tool-call arguments) plus a 40-byte
framing floor: 2.1-2.4 us per chunk (~3x cheaper), independent of
pydantic field count, never raises on unknown shapes (Anthropic events,
stub providers fall back to the floor). Both call sites switched
(chat-completions loop + anthropic event loop).
The counter feeds only the stream-retry diagnostic log line
(agent/stream_diag.py) — an estimate proportional to traffic preserves
its purpose (distinguishing 'died at 0 bytes' from 'died mid-stream').
6 new contract tests; 64 targeted stream tests green.
- extract _commit_registry/_note_refresh_failure shared by the background
worker and foreground stage-4 (identical 4-step success + failure paths
were duplicated); worker now commits under _models_dev_fetch_lock so a
failing background refresh can never re-arm the backoff immediately
after a successful force_refresh committed (unsynchronized-write race)
- add should_clear_context_pin_async to hermes_cli/route_identity.py
(matching the get_model_context_length_async precedent) and use it at
the 4 async gateway sites instead of inline asyncio.to_thread wraps;
the sync _format_session_info site keeps the sync call (already
off-loop via its callers' to_thread)
- test the background-refresh success path (the PR's primary new
behavior): disk saved, mem cache swapped, backoff cleared, in_flight
reset — mutation-checked
- replace the race-prone spin-wait on _models_dev_refresh_in_flight with
a named-thread join in the backoff test
The branched call shape in get_provider_info/get_provider is deliberate:
~69 test sites across tests/hermes_cli and tests/gateway monkeypatch
fetch_models_dev (and get_provider_info) with zero/single-arg lambdas.
Passing allow_network= unconditionally broke 5 tests in CI slices 2/3/7.
Documented the constraint inline.
- _mark_stale_cache_grace only moves cache_time forward so a completed
background refresh is never rewound to a 5-minute grace window
- clear _models_dev_refresh_in_flight if Thread.start() raises so a
one-off thread-exhaustion failure doesn't disable refresh forever
- move empty-registry validation into _fetch_models_dev_from_network
(was duplicated in the background worker and the foreground fetch)
- pass allow_network through as a plain kwarg in get_provider_info and
hermes_cli.providers.get_provider instead of the branched call shape
- refresh the stale module docstring (no bundled snapshot exists; the
resolution order now describes stale-serve + background refresh)
Follow-ups on top of @DonutsDelivery's salvaged reaper commit (#64141):
- create_from_config() now parses lsp.idle_timeout (invalid values fall
back to DEFAULT_IDLE_TIMEOUT) — previously the constructor knob was
unreachable from config.yaml (config exposure adapted from #36892 by
@0xbWy and #68091 by @9miya20)
- canonical default declared in hermes_cli DEFAULT_CONFIG so config
discovery surfaces the knob (per sweeper review note on #64980)
- reaper loop survives transient sweep errors instead of dying and
silently re-opening the leak (gap flagged in #68091 review)
- eventlog.log_reaped(): one INFO line per sweep + clears the
log_active announce cache so respawns re-announce at INFO
- docs: replace the stale 'no idle-timeout reaper' paragraph with the
new lifecycle description + config reference
- tests: reuse-refresh protection (the regression teknium's sweeper
requested on #64141), reaper-survives-error, config propagation,
invalid-value fallback, DEFAULT_CONFIG/manager-constant sync
gemini-2.5-flash shuts down Oct 16 2026 (Google deprecation schedule)
and gemini-3-flash-preview is superseded. Update every hardcoded
default to the current GA flash model:
- gemini_native_adapter: probe_gemini_tier + _create_chat_completion
default params; free-tier guidance de-pinned from a specific model's
RPD number so it doesn't stale again
- auxiliary_client: gemini/kilocode fallback aux models,
_OPENROUTER_MODEL, _NOUS_MODEL -> google/gemini-3.6-flash
(verified live on both OpenRouter and Nous portal /models)
- provider plugins: kilocode + vertex default_aux_model
- hindsight memory plugin: gemini provider default
- setup wizard gemini list: 3-flash-preview -> 3.6-flash (matches the
curated picker catalog)
- tests: aux-client assertions that pinned the old default literal now
reference the _NOUS_MODEL constant, so the next default bump can't
break them (change-detector cleanup)
Fixes#32360.
Encapsulates the feature behind one command for launch, and adopts the
official product name.
One command:
- `propose` moves from `hermes skills propose` to `hermes sync propose`, so
the whole feature is one command to learn and one to document. Its handler
moves from cmd_skills to cmd_sync accordingly.
- The `hermes sync` parser now documents both halves plainly: personal sync
across your devices, and sharing with your organisation. Added an examples
epilog; rewrote the verb help in user language ("Include a skill in your
sync" rather than "Opt a skill into sync").
- Every user-facing string that pointed at `hermes skills propose` now points
at `hermes sync propose` (8 sites, including the agent-visible guidance
returned by skill_manage and the org provenance header).
This also clears the way for #39343, which adds its own top-level `sync` for
git-repo profile backup — that feature nests under `skills`, this one owns
`sync`.
Naming:
- HSP / "Hermes Sync Protocol" is gone from prose, docstrings, and comments.
The feature is "Skill Sync".
- Public identifiers renamed: HSPClient -> SyncClient, HSPError -> SyncError,
HSPConflict -> SyncConflict, hsp_address -> wire_address, HSP_VERSION ->
WIRE_VERSION.
- The WIRE names are deliberately NOT renamed: the `hsp_version` capability
field and the `x-hsp-object-type` response header are set by the deployed
gateway-gateway sync plane (verified in src/sync/syncRouter.ts), so
renaming them client-side would break sync against a live server. A comment
at the version constant records why they differ from the product name.
- The version-mismatch error is now actionable ("this server speaks sync
version X, but this Hermes speaks Y — update Hermes to sync with it")
instead of leaking the protocol acronym.
Also fixes a wiring gap found on the way: the gateway housekeeping tick
pulled personal skills but never org skills — the same defect already fixed
for the CLI. Org pull now runs there too, gated on real org membership.
Tests: the jargon guard now also fails on a bare "HSP". The two tests that
asserted the old cross-command structure are replaced by three asserting the
new one (propose IS under sync, propose is NOT under skills, sync usage
lists it). 2294 passed / 0 failed across all 51 suites that import the
changed modules, via scripts/run_tests.sh.
Verified by running the real CLI: `hermes sync --help` lists all eight verbs,
`hermes skills --help` no longer mentions propose, `hermes sync propose
--help` parses, and `hermes sync status` still reports live org state.
* Revert "fix(credits): remove the 'Grant spent · $X top-up left' notice"
This reverts commit 5dc6a14c14.
* fix(credits): reintroduce grant_spent behind an in-session crossing gate
The removed notice nagged because its condition is a steady state for
accounts living on top-up, the latch is per-session, and the cold-start seed
runs the policy at every session open — every session re-announced
'Grant spent · $X top-up left'. Reintroduce it gated so only a session that
WATCHES the grant run out announces:
- seen_grant_unspent crossing gate, mirroring seen_below_90: opens when the
session observes the grant meaningfully unspent (>= GRANT_UNSPENT_MIN_MICROS,
1 cent — portal-seeded states derive micros from float dollars and can carry
sub-cent residue where headers report exactly spent). Seeds never prime it.
- The gate guards only the show branch and is consumed by the announcement —
one announcement per crossing. Header flicker (used_fraction None and back)
clears the sticky line but cannot re-announce; a renewal re-opens the gate.
- new_credits_latch() centralizes the latch shape (agent build, lazy re-init,
and the policy test helper all build through it).
- Tests: steady-state-open stays silent (seed + policy seams), live crossing
announces once, flicker/renewal/residual cases locked; the rendering test
primes the gate and asserts its leg count so a gate regression cannot
silently shrink coverage.
The codex app-server namespaces MCP tool call ids as
codex_mcp__<server>__<tool>_<codex_call_id>. With an exec-<uuid> component the
built-in hermes-tools server alone overflows the Responses API's 64-char
call_id limit, so the request 400s with a non-retryable "string too long".
The offending item sits near the head of the transcript and replays every
turn, permanently bricking the session — the only recovery is /reset.
Sibling defect to #10788, which clamped input[*].id via
_MAX_RESPONSES_ITEM_ID_LENGTH. Apply the same treatment to call_id at both
Responses emit sites in _chat_messages_to_responses_input: a deterministic
surrogate (call_ + sha256[:32]) for ids over the limit, short ids unchanged.
Because the surrogate is a pure function of the original id, a function_call
and its matching function_call_output — which carry the same original id — map
to the same surrogate and stay paired without correlating the two items.
The grant_spent notice fired for every subscription user with top-up
funds the moment their cap was reached and camped in the CLI/TUI status
bar and desktop toasts with no action to take — the account keeps
working off top-up. Remove it everywhere:
- agent/credits_tracker.py: drop grant_cond + the emit/clear block;
the dev fixture state now (correctly) produces no notice
- TUI: keep the turn-start clear of credits.grant_spent as back-compat
for older backends that still emit the key
- Desktop: drop the demo step and stale comment references
- Docs/config comments: remove grant-spent from credits_notices text
- Tests updated: policy/cold-start now assert the key never fires
Usage bands, depleted, and restored notices are unchanged; /usage still
reports the full balance breakdown.
