Assert that a superseded stream (older writer token, other thread) is fenced
from the delta sink, the active writer is never fenced, a non-claiming thread
is never treated as a writer, and the real consume loop stops the instant it is
superseded — so two streams can never interleave into one turn (#65991).
When the stale-stream detector reconnects past a stream whose socket abort
raced (the close never actually stopped the old worker), the superseded stream
and the retry's stream both write deltas into the same turn. The persisted
transcript is then two coherent responses interleaved token-by-token —
de-interleaving the stored text by alternation yields two complete, independent
answers to the same prompt, which is a dual-writer race in the harness, not a
model/context failure (#65991).
The interrupt path already positively cancels before force-closing (#6600), but
the stale-kill path relies only on the socket abort, and nothing fenced late
chunks from a superseded stream out of the shared delta sink.
Enforce a single-writer invariant on the sink itself, guarded by attempt id
rather than only socket state: every streaming attempt (chat_completions,
anthropic_messages, and bedrock paths) claims a monotonic writer token before
it begins consuming its stream. A newer claim supersedes any older one, so the
consume loop bails the instant it is superseded and _fire_stream_delta /
_fire_reasoning_delta / _record_streamed_assistant_text drop chunks from a
stale writer. The token is stored per-thread, so a thread that never claimed
(a non-streaming delta caller) is never fenced — the guard can only ever drop a
superseded stream, never the single legitimate writer. Discards are counted and
logged sparsely so a real provider problem stays visible instead of being
silently swallowed.
A hermes --tui session whose main thread wedges before app.run() returns
never executes the finally that calls _run_cleanup — the only place the
exit watchdog was armed — so a dead CLI lingered indefinitely (observed
~47 min at 4% CPU, the #65998 class).
Arm the backstop from the SIGTERM/SIGHUP handlers instead (both the
interactive and single-query paths), the earliest moment shutdown intent
is unambiguous. The signal-armed leash is 2x HERMES_EXIT_WATCHDOG_S so a
slow-but-progressing _run_cleanup (which still arms its own tighter timer)
is never cut short; the outer timer only wins when cleanup was never
reached. Idempotent across repeated signals; never raises from a handler.
Deliberately NOT armed at startup: the watchdog thread calls os._exit(0)
unconditionally after its sleep, so a startup-armed timer (the #65998
approach) would hard-kill every session that outlives the timeout.
Supersedes #65998; thanks @JeffStone69 for the report and root-cause gap
analysis.
Degrading models (observed with gpt-5.6 past ~350K input) emit tool-call
batches like 6 valid named calls + 1 blank-name call. Previously the
whole turn was voided — every valid call got 'Skipped: another tool call
in this turn used an invalid name' — and three such batches tripped the
3-strike stop, killing sessions that were still making progress.
Now a mixed batch error-results ONLY the invalid call(s) (terse
anti-priming error for blank names per #47967, catalog dump for typos)
and dispatches the valid subset for execution. The assistant message
keeps every emitted call so provider-side tool_call/result pairing stays
intact. The 3-strike counter only advances when a turn contains NO valid
call, so a fully-degenerate model still stops while a mostly-coherent
one keeps working. Broken JSON args on a never-executing invalid call no
longer trigger the whole-turn JSON retry loop.
Field evidence: July 2026 debug bundle showed gpt-5.6-sol emitting
6-call batches with one blank-name rider at 559K/384K-token context in
two separate sessions; 13 valid tool calls were discarded before the
session stopped as partial.
Follow-up fixes for salvaged PR #65637:
1. Clear _dirty_transcripts in rewrite_transcript + rewind_session —
stale pending messages were re-inserted after /retry, /undo, /compress.
2. Narrow _is_fts_corruption_error to specific SQLite error strings —
bare 'fts' substring matched 'shifts', 'gifts', etc.
3. Move DB write outside _transcript_retry_lock — holding the lock
during writes serialized all sessions' transcript appends and blocked
during FTS rebuild. Now the lock guards only the pending queue.
4. Push rebuild_fts() into SessionDB — SessionStore was reaching into
_conn/_lock private attrs. SessionDB.rebuild_fts() follows the same
pattern as optimize_fts().
5. Cap pending per session at 200 — prevents unbounded memory growth
when DB is persistently broken. Oldest messages dropped with warning.
Added 4 new tests: dirty-clear on rewrite/rewind, FTS matcher false
positives, pending cap enforcement.
Two changes to tests/agent/test_file_safety_session_state.py:
1. Drop the stale monkeypatch on tools.file_tools._get_live_tracking_cwd
— the helper was deleted in the cwd-tracking refactor (c80b244b5),
and monkeypatch.setattr on a missing attribute raises AttributeError,
breaking CI slice 4/8 on main for every PR. The patch was redundant:
the test writes an absolute path, so cwd resolution never engages.
2. Make the fixture stale-proof: instead of monkeypatching the private
_hermes_home_path/_hermes_root_path helpers (same failure class if
they're ever renamed), set HERMES_HOME to <root>/profiles/work and
let the real resolution chain (get_hermes_home /
get_default_hermes_root's profiles-parent rule) derive both paths.
The fixture now references zero private symbols and exercises the
production resolution path.
Port from anomalyco/opencode#35671: Z.AI / Zhipu GLM returns
'tokens in request more than max tokens allowed' (error code 1210) on
context overflow. This matched no pattern in _CONTEXT_OVERFLOW_PATTERNS,
so the error classified as unknown/retryable — the agent would retry the
oversized request instead of triggering context compression.
Proven live on main before the fix: classify_api_error() returned
FailoverReason.unknown for the exact Z.AI error shape; now returns
context_overflow.
Per the MCP spec the cursor is an opaque string; anything else
(including MagicMock auto-attributes in tests) means no more pages.
Fixes test_mcp_tool_session_expired mock-session runaway.
Port from anomalyco/opencode#35439/#35500: preserve full MCP catalogs
across paginated tools/list responses.
The MCP spec allows servers to paginate tools/list, resources/list, and
prompts/list via an opaque nextCursor token. The Python SDK's
ClientSession.list_* methods fetch exactly one page per call, and hermes
never passed the cursor back — on a paginated server every tool,
resource, and prompt past page 1 was silently invisible to the agent.
Adds _paginate_full_list() (cursor-draining helper with a 50-page
runaway cap and spec-correct opaque-string cursor validation) and
applies it at all three discovery sites: _discover_tools(), the
tools/list_changed refresh handler, and the list_resources/list_prompts
utility handlers. The keepalive probe intentionally keeps its
single-page call (liveness only).
E2E: real stdio MCP server serving 3 pages (2/1/1 tools) — old code
discovered 2 tools, new code discovers all 4.
Port from Kilo-Org/kilocode#11955: Gemini validates every object schema's
required list strictly against the same node's properties and fails the
ENTIRE GenerateContentRequest with HTTP 400 'required[0]: property is not
defined' when a name has no matching property. MCP servers (e.g. the GitHub
remote MCP) routinely emit array item schemas carrying required without
properties, which made every request on the native Gemini path fail before
any model output.
sanitize_gemini_schema() now filters required to names present in the node's
properties and drops the keyword when nothing valid remains. Applies
recursively (properties / items / anyOf). Tool handlers still validate
required fields at execution time, so nothing the model could actually use
is lost.
Scoped to the Gemini-facing sanitizer only — the universal
tools/schema_sanitizer.py already prunes typed object nodes, and its
remaining gap (untyped nodes) is contested by open PR #20151.
Port from openai/codex#31494: user-visible history replay must strip CSI
sequences and control characters. Stored conversation history can carry
raw terminal escapes (pasted content, gateway-origin text, model output
echoing injected tool results). Replaying it via /resume's recap panel or
build_recap (/status on CLI + gateway) wrote those bytes straight to the
terminal — an injected message could clear the screen, retitle the window,
move the cursor, or restyle the recap UI. Rich's Text() does not neutralize
raw escape bytes.
- tools/ansi_strip.py: add sanitize_display_text() — strip_ansi() plus
bare C0/C1 control removal, preserving \n and \t, normalizing \r to \n
(adapted to Python from Codex's sanitize_user_text; reuses the existing
ECMA-48 stripper instead of transcribing their char-walk)
- hermes_cli/cli_agent_setup_mixin.py: sanitize user + assistant text in
_display_resumed_history() before building the Rich recap panel
- hermes_cli/session_recap.py: sanitize preview lines in build_recap()
(_truncate choke point) so /status recaps are clean on every platform
- tests: 10 new sanitize_display_text cases (incl. the exact codex#31494
fixture), recap + resume-display leak assertions
The previous commit fixes the canonical agent/skill_utils.parse_frontmatter.
Six more modules reimplement the '---' fence check locally and had the
same bug:
- tools/skill_manager_tool.py _validate_frontmatter — rejected BOM'd
skill_manage create/edit content outright
- tools/skills_hub.py GitHubSource._parse_frontmatter_quick and
OptionalSkillSource._parse_frontmatter — hub browse/install metadata
- hermes_cli/skills_hub.py — local skill install validation
- gateway/run.py — skill slug discovery for disabled-skill hints
- agent/prompt_builder.py _strip_yaml_frontmatter — BOM'd context files
(AGENTS.md) leaked raw frontmatter into the system prompt
- tools/blueprints.py _split_frontmatter — str.lstrip() does not strip
U+FEFF (not whitespace), so the existing lstrip never covered it
Sibling-surface regression tests added.
Bug class also fixed upstream in cline/cline#12218 (found by the weekly
Cline PR scout).
A UTF-8 BOM saved into a SKILL.md (e.g. Notepad or PowerShell `>`) is kept by
read_text(encoding="utf-8"), so the string handed to parse_frontmatter starts
with the BOM and the startswith("---") fence check fails. The whole frontmatter
is then silently dropped: the skill loads with no name/description, `platforms`
gating falls open (a macOS-only skill becomes visible everywhere), and
required_environment_variables / metadata.hermes.config setup never fires.
Strip a single leading BOM at the top of parse_frontmatter, the shared
chokepoint for every local skill-loading path (_parse_skill_file,
discover_all_skill_config_vars, DESCRIPTION.md parsing, _inject_skill_config,
and the tools/skills_tool._parse_frontmatter re-export), so the whole class is
covered, not just the reported site. Only the leading marker is removed; a BOM
mid-content is left as data. Mirrors the existing file-tools BOM handling
(#35278) and CONTRIBUTING.md "File encoding".
Adds tests: BOM'd frontmatter parses identically to plain, the body is
BOM-free, platforms gating and config-var extraction survive a BOM, and an
end-to-end BOM-write / plain-read round trip (mirroring _parse_skill_file).
test_auto_reset_does_not_recover_session_being_ended asserted the old
end_session('session_reset') call; auto-reset now writes through
promote_to_session_reset with the specific auditable reason
('suspended' for a suspended entry). Missed in the local targeted run
because the file wasn't in the touched-suite list; caught by CI shard 6.
Unifies the two gateway subsystems that were fighting each other: the
'never lose a session' recovery machinery (#54878 stale-route self-heal,
find_latest_gateway_session_for_peer reopening agent_close/ws_orphan_reap
rows) and the session reset/expiry machinery (expiry watcher, /new,
/resume, resume_pending freshness gate).
The unified contract:
- INTENTIONAL boundaries (expiry finalization, auto-reset, /new,
/resume switch) are recorded durably via promote_to_session_reset(),
which upgrades accidental recoverable end_reasons (agent_close,
ws_orphan_reap) to the explicit boundary while preserving other
explicit reasons (compression, etc.). Recovery then correctly refuses
to resurrect them.
- ACCIDENTAL ends (crash, cleanup bug, mistaken reaper) stay
recoverable — genuine crash recovery is untouched.
On top of the cherry-picked contributor commits:
- promote_to_session_reset widened to ws_orphan_reap + parameterized
reason so auto-reset paths stay auditable (idle/daily/suspended/
resume_pending_expired) (#61220, #61993, #63539)
- get_or_create_session auto-reset, reset_session (/new), and
switch_session (/resume) all write through the promote path — the
first-reason-wins end_session no-op could previously leave a reset
session resurrectable behind a stale agent_close row (#61993)
- resume_pending freshness gate now honors session_reset.mode=none:
explicit opt-out of automatic resets also opts out of the zombie
gate (#61052)
- resume recovery note extracted to build_resume_recovery_note() and
made adapter-aware via a new interactive_resume capability flag:
webhook/api_server auto-resume turns now CONTINUE the interrupted
task instead of emitting an unanswerable 'session restored'
acknowledgement that abandoned the work (#57056)
- tests updated to call the real note builder instead of mirroring it
E2E-validated against a real SessionDB + SessionStore in a temp
HERMES_HOME: expiry->agent_close->no-resurrection, /new promote,
crash recovery preserved, mode=none opt-out, routing-table flag sync.
When a gateway session with resume_pending=True is not recovered within the
auto-continue freshness window (e.g. because repeated API calls timed out on a
large context), get_or_create_session correctly creates a new session. However
two gaps existed:
1. The user received no notification — resume_pending_expired fell through the
generic "inactive for Xh" else-branch in run.py, which produces wrong wording
and (for session_reset.mode: none users) is gated on policy.notify that
evaluates to False.
2. The old session was ended in state.db with the hardcoded generic reason
"session_reset", making it impossible to distinguish from a normal idle/daily
reset in post-mortem analysis.
Fix:
- gateway/run.py: add an explicit resume_pending_expired case for the agent
context note ("gateway restart recovery timed out") and the user-facing
notification. Always notify for this reason — like suspended — because the
user had an active session that was silently replaced.
- gateway/session.py: pass auto_reset_reason as the DB end_reason instead of
the hardcoded "session_reset", so all auto-reset paths are auditable.
- tests: extend TestResumePendingExpiredAutoReset in test_session_reset_notify.py
with five new cases that cover the reason, activity flag, DB end_reason,
non-regression of the idle path, and freshness-disabled bypass.
Closes#58933
Co-authored-by: Cursor <cursoragent@cursor.com>
When the #54878 self-healing path drops a stale sessions.json entry, the
fix at gateway/session.py:1765 now checks _should_reset() before falling
through to DB recovery. This test covers the case where the stale entry's
session is overdue under an idle/daily reset policy — it must create a
fresh session, set auto-reset metadata, and NOT call reopen_session().
When the session expiry watcher finalizes a session (daily/idle reset)
and the next message triggers the #54878 self-healing path
(get_or_create_session detects sessions.json / state.db mismatch),
the recovered session was silently reopened without checking whether
it should have been reset. This caused sessions to persist indefinitely
across reset boundaries.
Fix: after dropping the stale sessions.json entry in the self-healing
path, call _should_reset() against the old entry's updated_at. If a
reset is due, set db_end_session_id to skip DB recovery and create a
fresh session — matching the normal reset flow.
Address review feedback on #63068:
1. Replace unconditional reopen_session() + end_session() with a
conditional promote_to_session_reset() method in SessionDB.
The new method only promotes live rows or rows ended with
agent_close — explicit boundaries (compression, session_reset,
new_command) are preserved via first-writer-wins semantics.
2. Rewrite tests to use real SessionDB instead of MagicMock:
- 7 unit tests for promote_to_session_reset edge cases
- 3 integration tests verifying the actual recovery contract
in find_latest_gateway_session_for_peer after promotion
Port from anomalyco/opencode#36130: the Responses spec carries streaming
error details at the top level of the error frame, but the official OpenAI
SDK and several OpenAI-compatible proxies wrap them in an HTTP-style nested
envelope ({"type": "error", "error": {code, message, param}}).
_raise_stream_error only read top-level fields, so nested-envelope frames
collapsed to the generic 'stream emitted error event' placeholder with
code=None — the error classifier never saw the provider's real failure
reason, misrouting rate-limit / context-overflow / entitlement errors into
the generic retry path.
Top-level fields keep precedence; the envelope is a fallback. Null-tolerant
for spec-compliant frames with explicit nulls.
Only advertise finite watchdog deadlines that are still in the future, exercise the full MoA heartbeat path, and register the salvaged contributor attribution.
The dashboard console previously ran under a 'hosted' context that
blocked most commands (auth add, config set model.*, mcp add --command,
cron --script, ...) behind an allowlist + line-policy layer. With the
full Hermes CLI now built into the dashboard, that policy layer is
redundant gatekeeping: the console gets the same command surface
everywhere.
Removed:
- ConsoleContext/contexts plumbing on ConsoleCommand + engine
- EXPECTED_HOSTED_PATHS allowlist + _mark_hosted
- _enforce_hosted_line_policy + HOSTED_CONFIG_* allow/block tables
- _dashboard_console_context() and the context field on the ready frame
- hosted-context tests; context badge in HermesConsoleModal
Kept (mechanical, not policy): shell-syntax rejection, the
interactive/server command blocks (gateway, dashboard, mcp serve, ...),
mutating-command confirmations, output caps, and command timeouts.
Follow-ups on top of @xxxigm's salvaged bridge (#33294):
- Remove the now-dead narrow item/started-only mapper from #38835
(_codex_note_to_tool_progress) — the full bridge supersedes it and
keeps the same tool-name contract; its tests are repointed at the
bridge helpers.
- Preserve main's request_routing/approval-bypass wiring on the
CodexAppServerSession constructor (landed after the PR was filed).
- Gate agentMessage interim delivery on display.show_commentary so the
app-server runtime honors the same toggle as the codex_responses
commentary channel (tool progress is unaffected).
- Add json import (bridge helpers use json.dumps) and modernize the
wiring test's stub agent for main's usage-accounting attributes.
42 tests across five suites:
* ``TestCodexItemToToolName`` / ``TestCodexItemToArgs`` /
``TestCodexItemToPreview`` / ``TestCodexItemCompletionPayload`` —
pin the per-type mapping so the synthetic tool name + args the
UI sees match what ``CodexEventProjector`` writes into messages.
* ``TestStreamDeltaDispatch`` / ``TestToolProgressDispatch`` /
``TestAgentMessageInterimDispatch`` — drive each Codex
notification shape through the bridge and assert the right
agent callback fires with the right arguments (including the
duration / is_error / result kwargs the gateway renders).
* ``TestBridgeRobustness`` — defensive paths: non-dict
notifications, missing params, raising callbacks (must not
tear down the codex turn loop), and agents without callbacks
registered (cron / gateway-less contexts).
* ``TestBridgeWiredInRuntime`` — integration guard that
``run_codex_app_server_turn`` actually constructs the session
with ``on_event=<bridge>``, preventing a future refactor from
silently regressing live progress visibility again.
Pass ``on_event=make_codex_app_server_event_bridge(agent)`` when
spawning the per-session ``CodexAppServerSession``. The session has
always had a raw event hook but ``run_codex_app_server_turn`` never
supplied one, so Discord / Telegram / TUI users saw nothing while
codex was working — only the final answer landed.
Now each ``item/started`` for a tool-shaped item fires
``tool_progress_callback("tool.started", ...)``, ``item/completed``
fires the matching ``"tool.completed"`` with duration + result,
``item/agentMessage/delta`` flows through ``_fire_stream_delta`` and
each completed ``agentMessage`` surfaces through
``_emit_interim_assistant_message`` so the gateway's
``already_streamed`` dedupe keeps interim commentary in the channel
without duplicating text the stream already showed.
Adds ``make_codex_app_server_event_bridge(agent)`` plus four small
mapping helpers (``_codex_item_to_tool_name`` / ``_codex_item_to_args``
/ ``_codex_item_to_preview`` / ``_codex_item_completion_payload``)
that translate codex JSON-RPC ``item/*`` notifications into the
exact shape Hermes' gateway UI callbacks expect — tool names match
``CodexEventProjector`` so the progress bubbles and the projected
``tool_calls`` entries use the same identifiers.
No behaviour change yet: the next commit wires the bridge into
``run_codex_app_server_turn`` (#33200).
Review follow-up: the store-internals tests proved _current_cron_store()
resolves lazily, but not that the PUBLIC job I/O honors it. This exercises
save_jobs()/load_jobs() after a late env repoint and asserts the
import-time jobs.json stays byte-identical to a planted sentinel.
Review follow-up: tests that monkeypatch CRON_DIR/JOBS_FILE/OUTPUT_DIR (the
documented process-wide compatibility surface) were bypassed by the lazy
env fallback — 3 file-permission tests, the cross-process lock test, and
the heartbeat roundtrip regressed. _current_cron_store() now snapshots the
constants at import and honors any deliberate re-point of them ahead of
the env resolution, so the precedence is: use_cron_store() override >
patched constants > fresh HERMES_HOME > import defaults. Adds a test
pinning constants-beat-env; the late-env sentinel behavior is unchanged.
tests/cron: failure set byte-identical to unpatched main on this box
(the 5 regressions gone); 138 pass in the touched files.
Complements ec0227b43 (context-scoped cron store): the ContextVar override
is the right tool for deliberate cross-profile scoping, but with no
override active, _current_cron_store() returned the import-time constants —
so a HERMES_HOME set AFTER cron.jobs import (the filed incident: test
fixtures patching the env too late) still read/wrote the user's real
jobs.json. The fallback now resolves the active profile home fresh via
get_hermes_home() (context-local override, then env) and scopes the store
to it; when the home is unchanged since import, the exact module-level
constants are returned as before (zero change in the common path, and they
remain the documented compatibility surface). use_cron_store() still wins.
Three tests: late env repoint scopes the store; unchanged home returns the
import-time constants identically; an active use_cron_store() override
beats the env.
