The staleness check added in #66052 resolved the timeout from env,
config.yaml, and the default only, while the build path also reads the
honcho.json host block (timeout/requestTimeout). With a timeout
configured in honcho.json, the two permanently disagreed: every
no-config get_honcho_client() call — i.e. every HonchoSessionManager
.honcho property access — interpreted the mismatch as a config change
and tore down and rebuilt the client, defeating the singleton on the
hot path it was meant to protect.
Teach the check to read honcho.json through the same host-aware chain
as from_global_config, memoized on the file's mtime_ns so the per-call
cost stays one stat(). A genuine honcho.json timeout change is now also
detected, extending #57437 to that config surface.
Adds a --from DIR flag to scripts/dev-sandbox.sh that copies an existing
HERMES_HOME directory into the sandbox as the starting point before the
command runs. Lets you spin up a sandbox pre-populated with your real
config, sessions, skills, etc.
scripts/dev-sandbox.sh --from ~/.hermes hermes desktop
Design:
- cp -a dir/. dest/ — preserves perms, symlinks, hidden files
- Clobber guard: only seeds when sandbox HERMES_HOME is empty, so
re-running --persistent doesn't blow away existing sandbox state
- Validates: errors on nonexistent dir, missing arg, flag-like arg,
empty --from=
- Supports both --from DIR and --from=DIR forms
- Backwards compatible: no --from = unchanged behavior
$workingSessionIds and $attentionSessionIds were independently maintained
atoms that updateSessionState had to manually keep in sync with the session
cache (paired setSessionWorking/setSessionAttention calls, plus a rotation
special-case in ensureSessionState). Make them computed() projections of
$sessionStates instead, so the data flow is one-directional:
gateway event → cache → $sessionStates → computed views.
Transition side-effects (watchdog arm/disarm, settle grace, unread marker,
compression id rotation signal) move into handleTransition, fired from
publishSessionState by diffing previous vs next — one choke point instead
of per-callsite bookkeeping. The watchdog's force-clear reaches the cache
through setWatchdogClearFn rather than a listener set.
Also:
- clearAllSessionStates disarms all watchdog timers and drops settle-grace
entries so a gateway switch can't leak stale timers or keep-set rows
- dropSessionState disarms the dropped runtime's watchdog timer
- watchdog tests now exercise the real timer→callback wiring instead of
manually simulating the clear
Model-picker audit follow-through — closes the remaining pieces of the
"switch one session, switches everywhere / can't tell whose session this
is" report class:
- tui_gateway: `config.set key=fast` with a session no longer writes the
global agent.service_tier to config.yaml (sibling of the earlier
`reasoning` scoping fix). It pins create_service_tier_override
("priority" / "" for explicit normal) so lazy builds and rebuilds keep
the choice; the desktop's per-model presets were rewriting the global
tier on every model pick. Fast-support validation now checks a draft's
picked model, and `config.get key=fast` reads the pre-build pin.
- desktop: owning-profile tag (initial chip + tooltip/aria label) on
pinned rows and search results in the All-profiles sidebar, and on the
chat header once a second profile exists (#66003).
- desktop: composer model pill shows a pin dot + tooltip when a manual
sticky pick is overriding the Settings default for new chats (#62055).
Closes#66003. Addresses #62055.
A cron job ("Daily Buzz Report") died with 'AIAgent' object has no
attribute '_claim_stream_writer'. The #65991 single-writer fence lives on
AIAgent (run_agent.py), but the streaming paths that use it live in other
modules — chat_completion_helpers (chat / anthropic / bedrock) and
codex_runtime (codex responses) — and called it directly as
agent._claim_stream_writer() / agent._stream_writer_is_current(). That makes
those modules hard-depend on the method being present on whatever object is
passed as agent.
The fence is an *additive* safety net that may only ever drop a provably
superseded stream, never the sole legitimate writer. But the direct calls
turned any agent that doesn't expose it — a version-skewed checkout (the
streaming helper module newer than run_agent), a hot-reloaded gateway mid
git-pull, a duck-typed agent, or a test double — into a fatal AttributeError
that aborts the whole turn (and, on cron, fails the job).
Route every cross-module claim/check through agent/stream_single_writer.py.
claim_stream_writer(agent) returns 0 when the fence is unavailable (or
raises), and stream_writer_is_current(agent, token) treats a 0 token or an
absent guard as "current" — so a guard-less agent degrades to "no fence"
instead of crashing, while a real AIAgent keeps the full single-writer
protection. Internal self.* uses inside run_agent are unchanged (self is
always a full AIAgent there).
Mirror CANONICAL_PROVIDERS so Fireworks sits directly under Nous Portal
(always visible) ahead of OpenRouter across onboarding, Settings → Providers,
and the API-key catalog.
Answering the review question on the PR table — why a hovered-cold
switch still showed ~440ms click → WS open: getConnection-only
pre-warming left the WS connect chain to the click, and its microtask
continuation can only run after the click's fresh-draft React flush
(unmounting a large open transcript costs ~300-400ms of render work),
so the socket didn't even START connecting until the flush finished.
Add openGatewayForProfile: the same spawn + connect chain as a real
switch, minus activation — so the hover leaves the profile's socket
fully OPEN and the click's ensureGatewayForProfile just activates it
(no ws:new after the click at all; measured ws open at hover+136ms on
a warm backend). No scheduleReconnect on failure: a hover is
speculative, so a dead backend must not start a background retry loop
— the real switch owns retry and error UX. Pruning semantics are
unchanged: a hover-opened socket for an idle profile is dropped by the
next pruneSecondaryGateways recompute, which just returns the click to
the previous behavior.
Tests updated: pre-warm asserts openGatewayForProfile is called and
that activation (ensureGatewayForProfile) is NOT.
* fix(credential-pool): throttle "no available entries" log to stop Windows log-lock storm
Credential selection runs on a hot path (every model call plus auxiliary
tasks), so an empty/exhausted pool logged "no available entries" at INFO on
*every* selection. On Windows, where multiple Hermes processes share one
rotating log guarded by concurrent-log-handler's cross-process lock, that
per-selection volume storms the lock (RuntimeError: Cannot acquire lock after
20 attempts), pegs a core, and stalls the asyncio event loop long enough that
the Desktop backend readiness probe times out ("Timed out connecting to Hermes
backend after 15000ms") even though the backend already announced
HERMES_BACKEND_READY.
Log the condition at most once per 60s window, re-arming on a successful
selection so recovery->re-exhaustion still surfaces promptly. Same fix class as
the warn-once dedup in #58265.
* test(credential-pool): cover no-available-entries log throttle
Assert the empty-pool INFO line logs at most once per throttle window, logs
again after the window elapses, and re-arms on a successful selection so a
recover->re-exhaust transition surfaces promptly. Uses a deterministic fake
monotonic clock (no sleeps, no network).
Complements the #65637 salvage (53d358838 + a9cc17fd8): the gateway
session store now retries transcript appends through its own queue, but
cron and CLI writers call SessionDB directly — a corrupt FTS index still
hard-failed their appends until the next process restart triggered the
offline repair.
_execute_write now detects the FTS-corruption error class (both the
generic 'database disk image is malformed' and newer SQLite's
'fts5: corrupt structure record' variant), performs a one-shot in-place
rebuild by delegating to the existing rebuild_fts(), and retries the
failed write. One-shot per instance so an unrecoverable database cannot
loop; lock/busy jitter-retry path untouched.
E2E-verified: corrupted messages_fts_data rejects appends; with this fix
the same append self-heals, persists, and FTS search works again.
Community feedback (@LSanapalli on X): the inline task-creation form is
cramped inside a ~280px column with no way to resize; board-level
workspace defaults can't be changed after board creation; and users
believe they must block a task, comment, then unblock just to talk to
a worker.
- Create-task dialog: replace the inline column form with a centered
modal (reuses hermes-kanban-dialog chrome, 36rem wide) with labeled
fields for title, assignee, priority, skills, workspace kind/path,
goal mode, and parent task. Same request shape; Enter/Escape behavior
preserved; submit disabled until a title is present.
- Board settings dialog: new Settings button in the board switcher opens
a modal to edit display name, description, and the board-level default
project directory (default_workdir). PATCH /boards/:slug now accepts
default_workdir (validated absolute existing dir; empty string clears;
omitted leaves unchanged) and returns the recomputed
default_workspace_kind so task-creation defaults follow immediately.
- Comment workflow hint: the task drawer's comment box now explains that
comments land on the thread immediately and reach the worker on its
next run/kanban_show() — no block/unblock dance needed — with a fuller
tooltip for when blocking IS the right tool.
- i18n: new keys optional in the kanban namespace with English fallbacks
in the bundle (established pattern; avoids churning 17 locale files).
- Docs: dashboard section updated for the dialog + Settings button.
A cold profile switch pays the full pool-backend spawn — Python boot,
port announcement, readiness probe, token adoption — before the
profile's gateway can even open. Measured with the new CDP harness
(scripts/measure-profile-switch.mjs, same family as
profile-session-switch.mjs): click → WS open is ~2.5-2.9s on a cold
profile, ~3-3.6s to a settled sidebar; a warm profile settles in
~0.5-0.8s. The pointer entering a profile square telegraphs the switch
hundreds of ms before the click lands, so start the spawn then.
- store/profile: prewarmProfileBackend(name) — fires the existing
hermesDesktop.getConnection IPC, which is idempotent (ensureBackend
returns the pooled connectionPromise), so the real switch joins the
in-flight spawn instead of starting it. Skips the active gateway
profile, throttles per profile (60s) so drive-by hovers can't spam
spawn attempts, and swallows failures — error UX belongs to the real
switch. No new IPC surface; the pool's existing LRU cap + idle reaper
still bound resource use, and the LRU guard never evicts a
keepalive-fresh backend for a hover spawn.
- sidebar/use-profile-prewarm: pointerenter/pointerleave handlers with
a 120ms dwell so sweeping the pointer across the rail or a
mixed-profile session list doesn't spawn a backend per element
crossed.
- Wired at the three switch surfaces: rail ProfileSquare, the condensed
ProfileDropdown items (extracted ProfileDropdownItem so each row owns
its dwell timer), and SidebarSessionRow (covers cross-profile resumes
from the all-profiles view; same-profile rows no-op inside the guard).
Measured E2E over CDP: synthetic hover on a cold profile square spawns
its backend in the background; the subsequent click settles in ~519ms
vs ~3.0-3.6s unhovered — and any hover shorter than the spawn still
shaves its dwell off the click's wait.
Verification: apps/desktop `npx tsc --noEmit` clean; full
`npx vitest run` 212 files / 1777 passed (new prewarm guard/throttle
tests in store/profile.test.ts); eslint + prettier clean.
A root-launched CLI session can leak /root into the terminal cwd state a
non-root gateway/cron process later resolves (#65583). os.path.isdir('/root')
is True for a non-root user — stat only needs search permission on / — so
_resolve_safe_cwd returned it and subprocess.Popen(cwd='/root') died with
PermissionError: [Errno 13], failing EVERY cron job's terminal/file/search
tool on every command until restart.
_resolve_safe_cwd now requires X_OK (new _cwd_usable helper) and climbs to
the nearest enterable ancestor, logging a WARNING that names the leak class
when an existing-but-denied cwd is skipped. Missing-cwd recovery (#17558)
behavior unchanged.
E2E-verified: LocalEnvironment constructed with an unenterable cwd now runs
commands from the fallback directory instead of raising.
The widened resume sites pass repair_alternation=True; the DB doubles in
tests/test_tui_gateway_server.py (separate from tests/tui_gateway/) needed
the same signature update as Frowtek's originals.
Sibling-site audit on top of #65672: the interactive TUI resume, the
profile-scoped resume, and the /undo history reload also feed LIVE
REPLAY (raw_history -> sanitize_replay_history -> working conversation;
session['history'] after rewind). Pass repair_alternation=True on the
model-fed copies; display_history stays verbatim so inspection/export
show what is actually stored. Display-only consumers (session.history
RPC, formatted transcript output) intentionally unchanged.
test_build_child_agent_ignores_acp_command_when_binary_missing patches
shutil.which globally and asserted the LAST call was which("copilot").
That is order-dependent: an unrelated which("uv") reached later in the same
process (which happens under some CI test-slice orderings) becomes the last
call, so assert_called_with("copilot") fails even though the copilot binary
was probed exactly as intended. Switch to assert_any_call("copilot"), which
verifies the actual intent and is robust to unrelated which() calls. The
behavioural assertions (provider, acp_command, acp_args) are unchanged.
The lazy session.resume path now calls
db.get_messages_as_conversation(target, repair_alternation=True), but the
fake _DB stubs in test_protocol.py still declared the pre-change signature,
so the resume raised "unexpected keyword argument 'repair_alternation'"
and the three session_resume_lazy tests failed.
Mirror the real get_messages_as_conversation signature in the stubs by
accepting (and ignoring) repair_alternation.
Follow-up to the restore-boundary alternation heal (#65492): get_messages_
as_conversation grew a repair_alternation flag, wired into gateway
load_transcript and the CLI startup resume. Three other LIVE-REPLAY
restore sites still loaded the transcript verbatim, so a durable
'user;user' violation there re-fires the pre-request defensive repair on
every request for the rest of the session (it only ever mutates the
per-request list, never the restored working conversation):
- acp_adapter/session.py::SessionManager._restore — the loaded history
becomes the resumed ACP (Zed) agent's SessionState.history.
- hermes_cli/cli_commands_mixin.py — the /resume slash command sets
self.conversation_history from the load (the startup resume was fixed,
this mid-session one was missed).
- tui_gateway/server.py — the resume handler feeds the load into the
deferred session record's working conversation.
Pass repair_alternation=True at all three so the wedge is healed once at
restore. Inspection/export consumers (trace upload, context guard,
api_server history, display_history) keep the verbatim default.
Adds an end-to-end regression test driving the ACP _restore path: a
seeded user;user session restores to an alternation-clean live history
with no user input lost.
Two more display gaps from #26541 grafted onto the merged bridge:
- webSearch: codex's built-in web search now produces a tool.started/
tool.completed bubble pair (query as preview + args). Previously the
item type wasn't in _CODEX_TOOL_ITEM_TYPES, so built-in searches
showed nothing.
- mcp.hermes-tools.* stripping: tools codex invokes through Hermes' own
hermes-tools MCP server display as their bare names (web_search,
browser_navigate) instead of mcp.hermes-tools.web_search. The inner
dispatch subprocess can't fire native progress events, so the
codex-level event is the display event — name it the way users know
the tool.
Credit: both behaviors designed and first implemented by @simpolism in
PR #26541 (May 15, earliest of the app-server display-bridge family).
The approval-drain loop in CodexAppServerSession.run_turn drains up to 8
pending notifications to keep per-turn state current before answering a
server-initiated approval request — but never forwarded them to the
on_event display hook. Tool bubbles for items drained alongside an
approval (e.g. the item/started for the very command awaiting approval)
silently disappeared.
Mirror the main notification path's on_event invocation in the drain
loop. Regression test demonstrates RED→GREEN.
Grafted from PR #26541 by @simpolism — the earliest submission of the
codex app-server display-bridge family (May 15). Confirmed independently
by #64698 and #65412.
The REFERENCE ONLY framing ('treat as background reference, NOT as active
instructions... Do NOT answer questions or fulfill requests') was observed
bleeding into general tool-use suppression: a production session went
narration-only for 7 consecutive turns immediately after a compression
event, describing edits instead of calling tools (#65848 report).
Fix is additive: one clause stating the note does not restrict HOW the
agent works — tools remain fully active for the active task. Every
anti-resumption protection stays intact; the previous prefix generation
is frozen into _HISTORICAL_SUMMARY_PREFIXES per the module contract so
persisted summaries still get the directive-strip on re-compaction.
The #65848 rewrite was not taken: dropping the 'Do NOT answer questions'
line and the four-heading discard directive risks re-opening the
stale-task-resumption class those clauses exist to prevent (the carveout
era regressions #41607/#38364/#42812 documented in this file).
Report and root-cause analysis: @yasserbousrih (#65848).
Widen the #65991 single-writer fence to run_codex_stream: each codex
attempt claims the delta sink before consuming events, and the consume
loop's interrupt_check now also stops the instant a newer attempt
supersedes this one. Parity with the chat_completions / anthropic /
bedrock paths from the salvaged fix.
Two regression tests: superseded codex stream is fenced mid-stream;
sole-writer codex stream delivers unchanged.
Assert that a superseded stream (older writer token, other thread) is fenced
from the delta sink, the active writer is never fenced, a non-claiming thread
is never treated as a writer, and the real consume loop stops the instant it is
superseded — so two streams can never interleave into one turn (#65991).
When the stale-stream detector reconnects past a stream whose socket abort
raced (the close never actually stopped the old worker), the superseded stream
and the retry's stream both write deltas into the same turn. The persisted
transcript is then two coherent responses interleaved token-by-token —
de-interleaving the stored text by alternation yields two complete, independent
answers to the same prompt, which is a dual-writer race in the harness, not a
model/context failure (#65991).
The interrupt path already positively cancels before force-closing (#6600), but
the stale-kill path relies only on the socket abort, and nothing fenced late
chunks from a superseded stream out of the shared delta sink.
Enforce a single-writer invariant on the sink itself, guarded by attempt id
rather than only socket state: every streaming attempt (chat_completions,
anthropic_messages, and bedrock paths) claims a monotonic writer token before
it begins consuming its stream. A newer claim supersedes any older one, so the
consume loop bails the instant it is superseded and _fire_stream_delta /
_fire_reasoning_delta / _record_streamed_assistant_text drop chunks from a
stale writer. The token is stored per-thread, so a thread that never claimed
(a non-streaming delta caller) is never fenced — the guard can only ever drop a
superseded stream, never the single legitimate writer. Discards are counted and
logged sparsely so a real provider problem stays visible instead of being
silently swallowed.
A hermes --tui session whose main thread wedges before app.run() returns
never executes the finally that calls _run_cleanup — the only place the
exit watchdog was armed — so a dead CLI lingered indefinitely (observed
~47 min at 4% CPU, the #65998 class).
Arm the backstop from the SIGTERM/SIGHUP handlers instead (both the
interactive and single-query paths), the earliest moment shutdown intent
is unambiguous. The signal-armed leash is 2x HERMES_EXIT_WATCHDOG_S so a
slow-but-progressing _run_cleanup (which still arms its own tighter timer)
is never cut short; the outer timer only wins when cleanup was never
reached. Idempotent across repeated signals; never raises from a handler.
Deliberately NOT armed at startup: the watchdog thread calls os._exit(0)
unconditionally after its sleep, so a startup-armed timer (the #65998
approach) would hard-kill every session that outlives the timeout.
Supersedes #65998; thanks @JeffStone69 for the report and root-cause gap
analysis.
Degrading models (observed with gpt-5.6 past ~350K input) emit tool-call
batches like 6 valid named calls + 1 blank-name call. Previously the
whole turn was voided — every valid call got 'Skipped: another tool call
in this turn used an invalid name' — and three such batches tripped the
3-strike stop, killing sessions that were still making progress.
Now a mixed batch error-results ONLY the invalid call(s) (terse
anti-priming error for blank names per #47967, catalog dump for typos)
and dispatches the valid subset for execution. The assistant message
keeps every emitted call so provider-side tool_call/result pairing stays
intact. The 3-strike counter only advances when a turn contains NO valid
call, so a fully-degenerate model still stops while a mostly-coherent
one keeps working. Broken JSON args on a never-executing invalid call no
longer trigger the whole-turn JSON retry loop.
Field evidence: July 2026 debug bundle showed gpt-5.6-sol emitting
6-call batches with one blank-name rider at 559K/384K-token context in
two separate sessions; 13 valid tool calls were discarded before the
session stopped as partial.
Follow-up fixes for salvaged PR #65637:
1. Clear _dirty_transcripts in rewrite_transcript + rewind_session —
stale pending messages were re-inserted after /retry, /undo, /compress.
2. Narrow _is_fts_corruption_error to specific SQLite error strings —
bare 'fts' substring matched 'shifts', 'gifts', etc.
3. Move DB write outside _transcript_retry_lock — holding the lock
during writes serialized all sessions' transcript appends and blocked
during FTS rebuild. Now the lock guards only the pending queue.
4. Push rebuild_fts() into SessionDB — SessionStore was reaching into
_conn/_lock private attrs. SessionDB.rebuild_fts() follows the same
pattern as optimize_fts().
5. Cap pending per session at 200 — prevents unbounded memory growth
when DB is persistently broken. Oldest messages dropped with warning.
Added 4 new tests: dirty-clear on rewrite/rewind, FTS matcher false
positives, pending cap enforcement.
Two changes to tests/agent/test_file_safety_session_state.py:
1. Drop the stale monkeypatch on tools.file_tools._get_live_tracking_cwd
— the helper was deleted in the cwd-tracking refactor (c80b244b5),
and monkeypatch.setattr on a missing attribute raises AttributeError,
breaking CI slice 4/8 on main for every PR. The patch was redundant:
the test writes an absolute path, so cwd resolution never engages.
2. Make the fixture stale-proof: instead of monkeypatching the private
_hermes_home_path/_hermes_root_path helpers (same failure class if
they're ever renamed), set HERMES_HOME to <root>/profiles/work and
let the real resolution chain (get_hermes_home /
get_default_hermes_root's profiles-parent rule) derive both paths.
The fixture now references zero private symbols and exercises the
production resolution path.
Port from anomalyco/opencode#35671: Z.AI / Zhipu GLM returns
'tokens in request more than max tokens allowed' (error code 1210) on
context overflow. This matched no pattern in _CONTEXT_OVERFLOW_PATTERNS,
so the error classified as unknown/retryable — the agent would retry the
oversized request instead of triggering context compression.
Proven live on main before the fix: classify_api_error() returned
FailoverReason.unknown for the exact Z.AI error shape; now returns
context_overflow.
