Builds on jbbottoms's #65178 takeover fix (cherry-picked as the previous
commit). Windows --replace already tree-kills via taskkill /T, but the
POSIX paths signalled only the recorded gateway PID — adapter
subprocesses that outlived their parent kept holding scoped token locks
and blocked the replacement gateway.
- gateway/status.py: _snapshot_gateway_children() captures the old
gateway's descendants (psutil, recursive) while it is still alive;
reap_gateway_children() SIGTERMs verified orphans after the main PID
is confirmed dead, waits bounded, SIGKILLs survivors. Identity-aware
(psutil is_running is PID+create-time), skips zombies and children
whose ppid still equals the old gateway (parent actually alive), and
never raises — best-effort with debug/info logging only.
- take_over_scoped_lock_holder() snapshots before terminating and reaps
only on a confirmed successful handoff.
- gateway/run.py: start_gateway --replace snapshots before SIGTERM and
reaps after the old PID is confirmed gone, mirroring taskkill /T.
- tests/gateway/test_replace_child_reap.py: reap/skip/never-raise unit
coverage plus end-to-end --replace ordering (snapshot → terminate →
reap) and the no---replace path never touching the old process.
When --replace misses a cross-HERMES_HOME Telegram token holder, platform
connect used to retry forever. Terminate a verified gateway holder once
(with the takeover marker) and re-acquire the scoped lock (#65176).
Co-authored-by: Cursor <cursoragent@cursor.com>
Verified: applies cleanly and the patched module compiles. Tests are
described in the PR body (not bundled in this commit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the unlink()+O_EXCL sequence in acquire_scoped_lock with an
atomic os.replace() of the stale lock to a <lock>.stale tombstone
followed by the existing O_EXCL create. With plain unlink(), two racing
starters could both judge the lock stale and the second unlink() would
silently delete the first racer's freshly-created lock — both would then
'win'. os.replace() guarantees exactly one racer claims the stale file;
the loser gets FileNotFoundError and falls through to O_EXCL, which
admits at most one winner. Tombstones are cleaned up immediately;
behavior is otherwise identical.
Widen the PermissionError handling from is_gateway_runtime_lock_active
(#42689) to the sibling open() in acquire_gateway_runtime_lock: a stale
root-owned gateway.lock left by a launchd Background session previously
crashed the acquiring process. Unlink the stale file and retry once; if
the unlink or retry fails, return False cleanly instead of raising.
When the macOS launchd service runs in a Background session, the gateway
process spawns as root and creates a root-owned gateway.lock. On restart
as the normal user, open() on that file raises PermissionError, crashing
the gateway immediately and entering a launchd crash loop.
Catch PermissionError in is_gateway_runtime_lock_active(), remove the
stale lock file, and return False so the new process can start cleanly.
Fixes#42685
On macOS, the lock record's start_time is None (no /proc at creation),
but psutil.Process(recycled_pid).create_time() returns a valid float
for the unrelated process that now owns the PID. The old condition
required both sides to be None before falling back to cmdline checking,
so the recycled PID was never detected as stale.
Change the fallback condition from AND to OR: when either side's
start_time is missing, fall back to cmdline-based gateway detection.
Fixes#53763
Widen the allow_session tier from Matrix to every adapter the gateway
notifies: Telegram, Discord, Slack, Feishu, and Teams gate their Session
button on it; WhatsApp Cloud and qqbot accept the kwarg (no session tier
in their button sets). Also thread allow_session through the plugin-
escalation gate, the execute_code guard payload, and the plain-text
fallback so every notify path carries the same capability flags.
Adds an allow_session flag to the gateway approval payload so adapters
can render the session tier independently of the permanent tier. Matrix
gains a session reaction (🌀) and a reaction legend; pure-tirith prompts
now offer once/session/deny instead of collapsing to once/deny.
Salvaged from PR #67312, adapted to the allow_permanent semantics that
landed in #68597 (Always offered when any dangerous-pattern warning is
persistable; pure-tirith prompts stay session-max).
Three related messaging-approval fixes:
1. approvals.timeout default 60 -> 300. PR #63501 collapsed the gateway
wait onto the canonical approvals.timeout (previously
gateway_timeout=300), silently shrinking messaging approval windows
to 60s. Push-notification approvals routinely arrive later than a
minute; taps landed after the wait had already failed closed.
2. Stale-tap honesty: adapters resolved the approval AFTER rendering
'<checkmark> Approved by <user>' (Telegram/Discord/Slack), or ignored a zero
resolve count (WhatsApp Cloud/Feishu). A tap on an expired prompt
claimed approval while the command had already been denied. All
button paths now resolve first and render 'Approval expired -
command was not run' when nothing was waiting.
3. Mixed-warning prompts (dangerous pattern + tirith finding) now offer
Always: the persistence layer already permanently allowlists the
pattern key and downgrades the tirith key to session scope, but the
UI hid Always whenever ANY tirith warning was present. Pure-tirith
prompts still withhold Always (content findings are session-max by
design), and Smart-DENY overrides remain once-only.
The relay adapter re-attaches an egress discriminator on outbound replies
so the connector can resolve the owning tenant. It captured scope_id for
scoped (guild) messages and user_id for DMs, but as MUTUALLY EXCLUSIVE:
a scoped inbound hit an early return, so the author's user_id was never
recorded, and _with_scope only attached user_id when there was no
scope_id. Guild replies therefore went out with scope_id only.
That's fine while the guild has a provision-time route row. But a MANAGED
Discord agent joins guilds dynamically (the shared bot is added to /
removed from servers at runtime), and GATEWAY_RELAY_ROUTE_KEYS — the only
thing that writes guild route rows — is a self-hosted, static field never
stamped for managed agents. So their guild has no route row, the
connector's guild-route lookup misses, and with no user_id on the frame
there's nothing to fall back to → every guild reply is declined
"discord egress declined: target not routed to an onboarded tenant"
even though INBOUND resolved the same guild fine (via the author-first
SharedSocketRouter.targets() fallback).
Fix: capture the authentic author user_id for EVERY inbound (DM and
scoped alike) and re-attach it on the outbound reply alongside scope_id.
The connector consults it only on a route/scope miss, so carrying both
never overrides routing-table resolution. This is the gateway half of the
paired gateway-gateway change (makeDiscordTenantOf guild-route-miss
author-binding fallback); together they make guild replies resolve the
same observed-author way inbound already does.
Tests (tests/gateway/relay/test_relay_adapter.py): a guild reply now
carries both scope_id AND user_id; a scoped inbound with no author still
yields scope_id only (never invents one). Verified fail-without /
pass-with.
* fix(billing): rename user-facing "terminal billing" copy to Remote Spending
The capability was renamed Remote Spending on the portal (consent CTA:
"Allow Remote Spending"; per-terminal states Granted/Stopped), but the
terminal, desktop, and docs still said "terminal billing" everywhere.
- Feature name: Remote Spending in titles/labels, lowercase mid-sentence.
- Step-up action verb is now "allow", matching the portal consent CTA.
- Kill-switch-off recovery copy points at the actual control ("a billing
admin can turn it on from the portal's Hermes Agent page") instead of
the dead-end "manage it on the portal".
- Per-terminal revoke copy uses the portal vocabulary ("stopped").
- Wire identifiers (cli_billing_enabled, cli_billing_disabled, ...) are
unchanged; copy, comments, docs, and test expectations only.
* fix(billing): correct the post-step-up denial diagnosis + finish the desktop rename
Adversarial review findings: (1) a repeated insufficient_scope after a
successful step-up is a per-terminal authorization failure, but the copy
blamed the org kill-switch and pointed at the wrong recovery control —
now: "Remote Spending still isn't active for this terminal — the
authorization didn't take. Retry, or make this change on the portal."
(2) the desktop step-up flow started in Remote Spending vocabulary but
finished in "billing management access" — renamed both end states.
(3) prettier formatting on the touched files (matches the post-merge
fmt bot).
* fix(desktop): preserve interim assistant text wiped at message.complete
When the agent emits interim text (commentary alongside tool calls, or the
attempted final answer before a verify-on-stop nudge), all UI surfaces
streamed it live but then wiped it at message.complete — keeping only the
final response. The user saw text appear during inference, then disappear.
This is the complete fix across all three layers: agent core, gateway
transport, and all UI surfaces (desktop + Ink TUI).
The verify-on-stop and pre_verify paths flagged the assistant's attempted
final answer as _verification_stop_synthetic, suppressing it from both
state.db and the UI. The user only saw the terse post-verification reply.
Now the assistant response is real content: it's persisted to state.db and
emitted as an interim message via _emit_interim_assistant_message(force_display=True)
before the verification loop runs. Only the synthetic nudge messages keep
the synthetic flags. The turn finalizer drops nudges from live history and
compares content (not just role) to avoid duplicating a published candidate.
Message sequence repair collapses verification candidates in the
consecutive-assistant merge.
Wire agent.interim_assistant_callback both at construction (_agent_cbs())
and per-turn (defense-in-depth), emitting a new message.interim event with
{text, already_streamed}. Gated on display.interim_assistant_messages
(default true). Cleared in the finally block so a stale closure can't
fire on a later turn.
Add message.interim to the GatewayEventName union (apps/shared) and a
typed payload to the TUI's GatewayEvent discriminated union.
The TUI already had the segment-anchoring machinery (flushStreamingSegment +
finalTail) but had no handler for message.interim. Added recordInterimMessage
+ interimBoundaryIndex to seal segments mid-turn, and updated
recordMessageComplete to only dedupe segments after the interim boundary.
Replaced the fragile sealed-set approach with a proper interimBoundaryPending
state flag on ClientSessionState. finalizeInterimAssistantMessage finalizes
the streaming bubble in place (or creates a standalone one), rotates the
stream ID so next deltas create a new bubble, and sets the flag. When the
final text equals an already-sealed interim, they stay as distinct messages.
Extracted mergeFinalAssistantText() as a pure function in chat-messages.ts,
used by both completeAssistantMessage and finalizeInterimAssistantMessage.
Split the bidirectional dedup predicate: reasoning is a restatement only when
the final FULLY covers it. A short final ("Done.") no longer swallows a
longer reasoning block that merely starts with it.
Honor display.interim_assistant_messages (default true) across all layers:
the tui_gateway gates the callback, the desktop wires it to a nanostores
atom via use-hermes-config. Updated hermes_cli/config.py and
cli-config.yaml.example comments to document the Desktop behavior.
_split_segment_tokens now accepts posix=False and _find_ad_hoc_match tries
both posix modes so ad-hoc verification scripts with Windows backslash
paths are matched correctly. (response_previewed forwarding from #53553
is not included — our emit-interim + persist approach makes it unnecessary
since the attempted answer is now surfaced before the verification loop.)
- tsc: clean (desktop + TUI + shared)
- vitest desktop: 73/73 pass (7 interim-sealing + 5 mergeFinalAssistantText + 4 config atom)
- vitest TUI: 83/83 pass (4 new message.interim tests)
- python: 390 tests pass (340 tui_gateway + 33 verification/finalizer + 6 config gating + 3 evidence + 8 continuation budget)
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
* fix: prefix-match interim streamed content to avoid benign duplicate bubbles
_interim_content_was_streamed used exact equality (streamed == visible_content),
so a final response that was the streamed text plus a trailing delta — or a
partial stream before the verify nudge fired — failed the match and left
_response_was_previewed false. The turn then showed two bubbles (interim +
identical final) instead of settling the interim in place.
Relax to a prefix check (visible_content.startswith(streamed)) in both the
core match and the desktop's settle-in-place gate. The TUI already used
prefix matching via finalTail. The reverse direction (streamed longer than
final) is intentionally not matched — that could suppress a needed resend
in the gateway path where already_streamed=True calls on_segment_break().
* test(desktop): add partial-stream-then-nudge dedup edge case
Third edge case for the interim-sealing dedup: model streams part of its
answer via message.delta, verify nudge fires, interim seals the streamed
prefix, then the final response is the same text plus a trailing delta.
Asserts one bubble (not two) containing the full final text.
Acceptance protocol #2 — covers all three dedup edges:
1. interim == final (existing)
2. interim = strict prefix of final (existing)
3. partial-stream-then-nudge (this commit)
---------
Co-authored-by: Liam Zhang <yingliang-zhang@users.noreply.github.com>
Co-authored-by: Lucas D'Alessandro <lucasfdale@users.noreply.github.com>
Co-authored-by: Eric Manganaro <superposition@users.noreply.github.com>
Co-authored-by: sweetcornna <sweetcornna@users.noreply.github.com>
Co-authored-by: DECK6 <DECK6@users.noreply.github.com>
Co-authored-by: matantsevs <matantsevs@users.noreply.github.com>
Co-authored-by: gitcommit90 <gitcommit90@users.noreply.github.com>
A successful turn returning exactly NO_REPLY (or another exact silence
marker) leaked the literal control token when a second message was queued
before the first turn finished. The queued-follow-up recovery branch sends
the first final_response directly through adapter.send() and predates the
silence filter added to the normal completed-turn path.
Use the finalized task result for that recovery delivery rather than the raw
result_holder copy, then apply the existing
is_intentional_silence_agent_result() predicate before sending. This keeps
the established contract: only successful exact-marker turns suppress;
substantive prose and failed results still send; stream-confirmed responses
still skip the resend; and persisted history is untouched. Using finalized
output also preserves normal empty/failure normalization on this direct path.
Integration regressions run the real Slack _run_agent queued-follow-up flow:
one proves NO_REPLY never reaches the adapter while the second turn still
runs; another proves an empty failed first turn sends its normalized error
before the queued follow-up. Existing filter tests cover every supported
marker, prose mentions, and failed-result semantics.
Address PR #62873 review:
- Bare YAML `mode: off`/`on` parse to Python False/True (YAML 1.1). Stringifying
False yielded "false" (not "off"), so `mode: off` wrongly enabled streaming.
Add _normalize_transport_token() to map booleans to canonical off/auto tokens,
mirroring the normalization documented in gateway/display_config.py.
- Only the `mode` alias infers `enabled`; a bare `transport` no longer enables
streaming, preserving `streaming.enabled` as the documented master switch
(website/docs/user-guide/configuration.md).
- Update tests to the corrected contract and add YAML-boolean coverage plus
loader-level regressions for unquoted `mode: off` and nested mode enable.
- StreamingConfig.from_dict now treats `mode` as an alias for `transport`
that also implies `enabled`, so `streaming: {mode: auto}` turns streaming
on instead of being silently ignored (enabled defaulted to False, which
buffered the whole reply and sent it in one message)
- `mode: off` disables streaming; an explicit `enabled` key still wins; an
explicit `transport` takes precedence over `mode`
- Add regression tests covering mode/transport/enabled precedence and the
real-world `mode + preloader_frames` block
Follow-up for salvaged PR #59779: the session_reset and stt fallbacks
used truthiness/type checks, so a present-but-empty top-level value was
silently replaced by the nested gateway.* form — inconsistent with the
key-presence precedence every other key in the block uses. Switch both
to 'key not in yaml_cfg' gating and add precedence regression tests.
load_gateway_config() already accepted both the top-level key and the
nested gateway.<key> form (written by `hermes config set gateway.<key>
...`) for multiplex_profiles, max_concurrent_sessions, streaming, and
write_sessions_json — each fixed one at a time as users hit it (most
recently #59320 for multiplex_profiles). Nine sibling top-level keys
never got the same nested fallback: session_reset, quick_commands, stt,
stt_echo_transcripts, group_sessions_per_user, thread_sessions_per_user,
reset_triggers, always_log_local, and unauthorized_dm_behavior.
`hermes config set gateway.<any-of-these> ...` builds exactly this nested
shape (hermes_cli/config.py's _set_nested has no schema, so it accepts
any dotted path), so a user following the same pattern that legitimately
works for gateway.multiplex_profiles/gateway.streaming gets a silent
no-op for these nine keys instead.
Fix: read `gateway: {...}` into a single `gateway_section` variable once
(consolidating three separate `yaml_cfg.get("gateway")` calls already in
the function) and add the same top-level-wins/nested-fallback check for
each of the nine keys, mirroring the existing write_sessions_json
precedent exactly.
Note: because every fallback here is guarded by
`isinstance(gateway_section, dict)`, this also makes the streaming
fallback tolerate a scalar `gateway:` block (e.g. `gateway: disabled`)
without crashing — the same crash #40837 (open) targets specifically for
streaming. This change doesn't set out to fix that PR's issue, but the
consolidated guard covers it as a side effect; flagging it for the
reviewer rather than leaving it to be found in review.
* fix(fast): default /fast to session scope on CLI and gateway
Completes the session-first policy from #67946 for the /fast toggle
(the remaining half of #54084). A bare /fast fast|normal now applies to
the current session only; --global persists agent.service_tier to
config.yaml.
Gateway: new _session_service_tier_overrides dict (registered in
_CONVERSATION_SCOPED_STATE so /new clears it) resolved at both agent
turn sites via _resolve_session_service_tier(); the /fast handler and
its choice picker apply session overrides and evict the cached agent.
The TUI config.set fast path was already session-scoped.
CLI: /fast parses --global (parity with /reasoning); bare toggles
mutate self.service_tier only.
* fix(sessions): /new resets model, reasoning, and fast to config defaults
/new and /reset are full conversation boundaries: session-scoped
runtime overrides do not carry into the next session (#48055, #23131).
CLI new_session(): clears the one-turn model restore, re-derives
service_tier from config, and — when the session's model differs from
the config default — switches back via the shared switch_model()
pipeline (live agent swap included; best-effort so an unreachable
default never blocks /new).
TUI _reset_session_agent(): stops forwarding model_override /
create_reasoning_override / create_service_tier_override into the
rebuilt agent and pops the pins so later rebuilds can't resurrect
them. The gateway already cleared its per-session overrides via
_clear_conversation_scope on /new.
Cross-session contamination stays impossible: nothing here touches
process-global env or other sessions' pins.
Flip the resolve_persist_behavior() fallback from persist-to-config to
session-only. A plain /model <name> (typed or via any picker — CLI,
TUI/Desktop, gateway) now affects only the current session; --global
persists explicitly, and model.persist_switch_by_default: true restores
the old opt-out behavior for users who want switches to stick.
This is the root cause behind the recurring 'session switch applied
globally' bug class (#61458, #63083, #58290, #61190): every surface
funnels its no-flag default through this one function, so per-surface
patches kept missing paths. Fixing the default fixes all surfaces at
once: CLI typed + picker, TUI/Desktop config.set + slash, gateway typed
+ inline picker.
Builds on liuhao1024's #58371 (--provider session scoping, cherry-picked
as the previous commit) and supersedes the per-surface #61488.
When /model is called with --provider but without --global or --session,
the switch now defaults to session-only instead of persisting to
config.yaml. Provider switches are typically exploratory — the user is
trying a different backend for this conversation, not reconfiguring the
default. --global can still force persist when desired.
This addresses a regression from fad4b40d9 where /model switched to
persist-by-default, causing /model xxx --provider xxx to overwrite the
global config when the user only intended a temporary switch.
Fixes#58290
Address review on PR #36998: the inline ri.<service>..<ns>. stripper in
_get_status_bar_snapshot was a one-off heuristic that:
* lived in cli.py with no shared call site, so the switch-confirmation
banner ("✓ Model switched: ri.language-model-service..…") and the
[Note: model was just switched from … to …] system-prompt nudge still
printed the full opaque RID — exactly what the screenshot reported;
* split on '..' and re-split on '.', which would mis-handle any RID
whose namespace token isn't a single dotted segment.
Refactor:
* New module-level helper hermes_cli.model_switch.format_model_for_display
matches on a startswith() allow-list (_OPAQUE_MODEL_PREFIXES) and
returns the trailing slug. Falls through to the original string for
every non-Palantir id, so HF paths (meta-llama/Llama-3.3-70B-Instruct),
plain Claude/GPT names, .gguf paths, and aliased ids are untouched.
Allow-list is extensible — add a prefix tuple entry for future
proxies that wrap real names in a namespace (Bedrock ARNs are
already covered by the slash-split fallback and have a different shape).
* _get_status_bar_snapshot() now delegates to the shared helper after
the reverse-alias miss (so configured aliases still win over the
helper output).
* cli.py::_handle_model_command — both confirmation-print blocks
(~7720 and ~7975) now run result.new_model AND old_model through
the formatter before they hit _cprint() and the
_pending_model_switch_note text.
* gateway/run.py model-switch handler (~10915) — same treatment for
_pending_model_notes[_session_key] and the
t('gateway.model.switched', model=…) confirmation line returned to
the gateway client.
The formatter is DISPLAY-ONLY. The session_model_overrides map,
ModelSwitchResult.new_model, persistence to config.yaml, alias lookups,
and every wire call still carry the full opaque RID — Palantir's API
requires it.
Verification: unit reproducer covers (a) all four Palantir model RIDs
from this user's config stripped to the trailing slug, (b) plain
model names (claude-4-7-opus-20260101, gpt-5.4, HF paths, empty
string) passed through unchanged, (c) prefix-only edge preserved
(no infinite-loop / empty-output regression).
Refs: PR #36998 review feedback; screenshot showed model banner still
printing the long RID after the original status-bar-only fix landed.
The delivery ledger durably records a final response before the send so a
crash between finalize and platform ACK can redeliver it on the next boot.
attempts is that redelivery budget, capped at MAX_ATTEMPTS=3.
sweep_recoverable() claims every dead-owner row and increments attempts
before the caller knows whether it can send. self.adapters only holds a
platform after its connect() succeeded, so when the platform failed to
connect this boot _redeliver_pending_obligations() hits its "adapter is
None" branch and continues WITHOUT sending — but the attempt is already
spent. Three such boots and the row abandons, having never been sent once.
That is the loss the ledger exists to prevent, and the trigger correlates
with the crash that created the obligation: the network trouble that killed
the send tends to still be there on the next boot. Worse, the message stays
lost — once abandoned it is never retried even after the platform recovers.
Reproduced against the real runner with an unconnected adapter:
boot 1: claimed=1 state='attempting' attempts=1 (0 sends attempted)
boot 2: claimed=1 state='attempting' attempts=2 (0 sends attempted)
boot 3: claimed=1 state='attempting' attempts=3 (0 sends attempted)
boot 4: claimed=0 state='abandoned' attempts=3 (0 sends attempted)
Let the caller declare which platforms it can send on, and skip claiming
rows for the others. attempts then only ever buys a real send. Rows for a
platform that never returns are still bounded by the stale cutoff, so
nothing accumulates. The parameter is keyword-only and optional — omitting
it keeps the previous claim-everything behaviour for other callers.
* fix(gateway): serialize concurrent turns per resolved session_id with a turn lease
Closes the serialization half of #64934. The busy guards are keyed by
routing key, but the durable transcript is owned by session_id — and
switch_session() makes the key→id mapping many-to-one (/resume from a
second chat/topic, CLI-continuity rebinding, async-delegation pinning,
topic-binding tip-walks). Two routing keys mapped to one session_id ran
concurrent turns on two different agent objects, invisible to every
per-key guard: flushes persisted in completion order, the identity-marker
dedup swallowed rows, and the second turn ran on a stale history base —
leaving a permanent user;user alternation wedge.
The fix: an asyncio lease keyed by RESOLVED session_id (gateway/turn_lease.py),
acquired in _handle_message_with_agent after session resolution is final
(post switch_session/tip-walk), immediately before the transcript load, and
released in _handle_message's finally on every exit path. Tokens are granted
per (routing key, run generation) so a stale unwind can never release a newer
turn's lease (#28686 ownership lesson). Same-key messages never reach the
acquisition point mid-turn (both routing-key guards hold them), so the lock
is uncontended outside the alias-key route — where the second turn now waits
for the first turn's flush and logs one WARNING naming the session and both
routing keys (pairs with the #67371 tripwire).
Fail-open: a stuck holder degrades to today's unserialized behavior with a
loud ERROR after agent.gateway_timeout — never a wedged session; a degraded
token holds nothing and can't steal the lease. Registry is size-capped and
never evicts a live lease. Persist-disabled review forks never dispatch
through _handle_message, so they cannot contend.
Known limits (tracked on #64934): CLI-continuity cross-process pairs need a
DB-level lease; mid-turn compression rotation leaves a small alias window
for a follow-up at the binding-sync sites.
Validation: 8 behavior tests (alias-key wait + flush order, no cross-session
contention, generation-scoped idempotent release, timeout fail-open without
lease theft, bounded registry, bare-runner-safe release wiring) + E2E against
a real SessionStore reproducing the issue's switch_session alias route —
strict alternation and arrival order preserved.
* refactor(gateway): conversation-scope funnel + mid-turn lease rebind
Completes the #64934 system beyond the point fix. Two structural changes,
both eliminating whole bug classes rather than instances:
1. _clear_conversation_scope — THE single conversation-boundary funnel.
/new, /resume, auto-reset, expiry finalization, and the
compression-exhausted reset each carried a hand-copied pop-list of the
per-session dicts, and the lists drifted every time a new dict was
added (#48031, #58403, #10702, #35809 were all 'boundary X forgot
dict Y' bugs). All five sites now make one funnel call driven by the
_CONVERSATION_SCOPED_STATE registry; adding a new conversation-scoped
dict means adding one name to the registry, and every boundary picks
it up automatically. Scope rules documented at the registry: turn-scoped
state, the monotonic generation counter, and the agent cache are
deliberately excluded (different lifecycles).
2. SessionTurnLeaseRegistry.rebind — the held turn lease now FOLLOWS
mid-turn compression rotation. Both rotation sites (session-hygiene
pre-compression, agent-result session_id swap) alias the same
_SessionLease object under the new id, so an alias routing key
resolving the fresh child (topic tip-walk) still serializes against
the in-flight turn. Closes the rotation-alias window flagged as a
known limit on #64934. Ownership-checked like release; when the
target id already has a live lease the rebind fails open with a loud
WARNING (never a mid-turn deadlock).
Tests: 3 new rebind behavior tests + 5 funnel behavior tests (including
a real-setter drift guard); the two AST change-detector pins in
test_10710/test_48031 were re-pointed at the funnel and the #58403 pin
converted to a behavioral test. E2E: rotation-alias scenario against a
real SessionStore + SessionDB — turn B on the fresh child waits behind
the rotated holder, sees its rows, alternation intact.
The per-message ephemeral context prompt re-renders every turn, and
any byte change (Discord auto-thread rename, reset notes, voice
channel state) both breaks the provider prompt-cache prefix at the
head of every request and changes the gateway agent-cache signature,
forcing a full agent rebuild per message. Pin the rendered block per
session keyed by a hash of exactly the fields it renders, so only a
real input change (rename, topic edit, /sethome, redact_pii flip)
re-renders; deliver one-shot per-turn facts (auto-reset note,
first-contact intro, voice-channel changes) on the current user
message via the api_content sidecar instead of the system prompt; sort
get_connected_platforms for byte-stable ordering.
A final response generated but not confirmed-delivered was the one
artifact the gateway could lose without a trace: crash or planned
restart between finalize and platform ACK dropped it silently, and the
resume path re-ran the whole turn at full cost (#58818 P1, #41696,
#63695's gateway half).
gateway/delivery_ledger.py records each outbound final response in
state.db (same conventions as the async-delegation ledger: WAL, owner
pid + process-start-time liveness, bounded retention):
pending -> attempting -> delivered | failed
startup sweep on dead-owner rows -> redeliver | abandoned
Contract (the lessons from the closed delivery-outbox attempt #61790):
- obligation recorded BEFORE the first send attempt; cleared only on
SendResult.success (destination acceptance, #51184)
- ambiguity is labeled, never silently retried: rows that were mid-send
when the process died redeliver with a visible '♻️ Recovered reply —
may be a duplicate' prefix (honest at-least-once)
- stable ids from session_key + inbound message id + content, so
distinct threads/topics can never collide
- poison rows bounded: 3 attempts / 24h freshness -> abandoned; claim
atomically re-stamps ownership so racing sweeps can't double-claim
- redelivery clears resume_pending for the session so the resume path
never re-runs a turn whose answer the ledger already holds
- best-effort everywhere: ledger failure can never block or delay a send
- slash-command/ephemeral/empty responses are not recorded; cron and
proactive delivery stay on DeliveryRouter (separate subsystem)
Config: gateway.delivery_ledger (default on; no version bump needed).
Validation: 30 ledger+producer tests; 352 blast-radius gateway tests
green; cross-process E2E (record in process A, kill it mid-send, claim
+ marker + redeliver in a fresh process B against the same state.db).
Rebased onto current main, where _ensure_session_db grew a per-profile
cache (get_hermes_home()-keyed) for /p/<profile>/ multiplex — after this
PR's base. The PR's async rewrite assumed the old single-self._session_db
model, so on current main `session_db=self._session_db` in _create_agent
would pass None in production (the real DB lives in the per-home cache).
Split the concern: keep a SYNC _ensure_session_db (per-profile, used by
_create_agent + the many sync-patching create_agent tests) and add an
async _ensure_session_db_async that captures the profile home on the loop
thread then offloads only the SQLite open via to_thread (single-flight).
Both share _open_and_cache_session_db. Request handlers use the async
variant; _create_agent reverts to the sync call. Updated the first-request
test's FakeDB to accept db_path to match main's SessionDB(db_path=...).
Co-authored-by: necoweb3 <sswdarius@gmail.com>
- Make create sequence (check + insert + title) atomic via single
_execute_write call with BEGIN IMMEDIATE, closing the TOCTOU window
where two concurrent same-ID creates could both return 201.
- Offload _ensure_session_db() to asyncio.to_thread with single-flight
lock so first-request SQLite init doesn't block the event loop.
- Add concurrent same-ID create test (one 201, one 409) and
first-request path test covering the initialization.
Deduplicates the sidecar handling across 9 sites:
- substitute_api_content(): 2 API-bound pop+substitute sites
(chat_completion_helpers, transport — conversation_loop keeps its
inline pop because the current-turn compose fallback needs the value)
- drop_stale_api_content(): 4 rewrite-drop sites
(context_compressor x2, replay_cleanup, agent_runtime_helpers)
- extract_api_content_sidecar(): 3 gateway forwarding sites
(gateway/session, gateway/slash_commands, cli_commands_mixin)
Also: restore the eager _pending_cli_user_message clear on the early
row-creation try (was lost when the crash-persist moved after prefetch —
a crash in compression between the two tries would leave a stale staged
input), and fix a comment indentation nit in run_agent.py.
Co-authored-by: Soju06 <qlskssk@gmail.com>
The first LLM call of every gateway turn gets ~0% provider prompt-cache
hit rate (in-turn calls: 97-99%) because the bytes sent for a turn's
user message are not the bytes replayed next turn: memory-prefetch and
pre_llm_call context are injected into the API copy only, the #48677
persist override writes cleaned content to the DB row, and
get_messages_as_conversation sanitize/strips user and assistant content
on load. Any of these diverges the request prefix at that message and
re-prefills everything after it — measured 27.9s for the first call vs
2.4-5.8s cached at a median ~156k-token context.
Persist what you send: a nullable messages.api_content column stores the
exact content string sent to the API when it differs from the clean
stored content, and replay substitutes it verbatim (no sanitize, no
strip). The injection composition lives in one helper
(turn_context.compose_user_api_content); the turn prologue stamps its
output onto the live user message, the api_messages build sends the
stamped bytes, and every outgoing copy pops the field so it never
reaches a provider. The crash-resilience user-turn persist moves after
prefetch/pre_llm_call so the user row is written once with its final
sidecar; _ensure_db_session stays before preflight compression (session
rotation needs the parent row under PRAGMA foreign_keys=ON). The
current-turn index trackers are re-anchored after compaction rebuilds
the message list, in-place preflight compaction backfills the stamp onto
the already-inserted row, and gateway replay forwards the sidecar only
when the replay pipeline did not rewrite the content. Rewrite paths that
would leave stale bytes (historical image strip, merge-summary-into-
tail, consecutive-user repair merge, stale-confirmation redaction) drop
the sidecar; the chat-completions transport and the max-iterations
summary path strip it defensively. codex_app_server and MoA turns are
excluded from stamping because their wire bytes differ from the
composition. A missing or dropped sidecar degrades to today's behavior
(one cache-boundary miss), never to wrong content.
The busy/priority/monitor/backup×2/drain paths each had near-identical
try/except blocks for transcribe+echo with only log_context, adapter,
and metadata varying. Extract into _transcribe_and_echo_pending_voice
which handles the cache lookup, echo dedup, and exception logging in
one place.
Uses a _UNSET sentinel to distinguish 'caller did not pass metadata'
(use the rich _thread_metadata_for_source fallback) from 'caller
explicitly passed None' (monitor/backup/drain paths that use the
simpler thread_id-only dict).
~120 lines of copy-paste collapsed into one 30-line helper.
merge_pending_message_event extends the existing event's media_urls in
place when two media-bearing messages arrive in quick succession (photo
bursts, consecutive voice messages). The gateway runner caches STT
transcripts on the event via _gateway_pending_stt_text; if the cached
event gains new media after the cache was populated, the stale transcript
was returned instead of transcribing the merged attachments.
Add _invalidate_pending_stt_cache() and call it from both media-merge
branches in merge_pending_message_event so the next transcription call
re-runs against the full merged media list.
Closes the merge-race edge case identified during review of PR #61519.
The function was introduced in d55304c39 but never had a single caller —
verified via git log -S and search_files across the whole repo. The
drain path at _stream_confirmed_final_delivery inlines its logic directly.
Keeping a dead function around that duplicates live logic is a maintenance
hazard: future changes to the live path won't propagate to the dead one.
Fixes the two review defects that kept PR #29923 open, plus docs:
- gateway: exclude --once from the session-store write-through. The
once-override lived only in memory before, but the write-through
persisted it, so a gateway restart before the finally-restore
rehydrated a supposedly one-turn model permanently.
- TUI: skip _sync_agent_model_with_config while a one-turn restore is
pending. The once-model is deliberately not pinned as a session
model_override, so the config sync saw a model mismatch and clobbered
the once-override back to the config model before the turn ran.
- tests: real _handle_model_command drive asserting --once never
touches set_model_override while --session still does; restore-pop
idempotency.
- docs: /model --once in configuring-models.md with an honest
prompt-cache cost note (one-shot switch breaks the cached prefix
twice; wins for short sessions and cheap-to-expensive escalation).
Adds --once to /model across CLI, TUI, and gateway: switch model for the
next turn only, restoring the previous model in a finally block so
success, exception, and interrupt all revert. Parsing extends
parse_model_flags_detailed(); resolve_persist_behavior() treats --once
as a persistence opt-out; --global + --once is rejected.
Salvaged from PR #29923 (image-generation lane split to #59815 per
review; conflict resolution against current main by the maintainers).
Preserve monotonic final-delivery completion, isolate recovery config and storage per adapter/profile, coalesce reconnect scans, release cancelled claims, bypass split-message debounce for historical events, and move bounded ledger setup into a short-timeout state module.
Route recovered messages through the live Discord ingress policy, preserve dedup and completion invariants, bound and retain the recovery ledger, and expose the opt-in config with docs and backup coverage.
Gateway tests build adapters via object.__new__() without __init__ (the
documented bare-instance pattern), so the new _status_text dict must be
accessed through getattr guards in set_status_text, the _keep_typing
finally cleanup, and the Slack send_typing read — same treatment as
other post-hoc __init__ attributes. Fixes CI shard 2/8
(test_active_session_text_merge).
Builds on the salvaged typing_status_text plumbing (PR #62007): instead
of a static 'is thinking...', Slack's assistant status line now updates
live as the agent works — 'is running pytest tests/…', 'is reading
docs/api.md…' — and reverts to the static text between tool calls.
Mechanics:
- agent/display.py: build_status_phrase() derives a <=49-char present-
tense phrase from the existing _TOOL_VERBS table (+ 'is using <name>'
for plugin/MCP tools; None for _thinking).
- base adapter: supports_status_text capability flag + set_status_text()
per-chat store, cleared when the typing loop winds down.
- Slack adapter: send_typing() renders the live phrase when set, falling
back to typing_status_text then 'is thinking...'.
- gateway/run.py: progress_callback stashes the phrase on tool.started
and clears on tool.completed. Rendering rides the existing
_keep_typing refresh cadence — zero additional Slack API calls, no
rate-limit exposure. Works with tool_progress: off (Slack default);
the callback is now armed whenever the adapter supports status text.
- display.live_status config (full|verb|off, default full): 'verb' hides
argument previews for shared/customer-facing channels.
Also fixes a latent crash in the cherry-picked from_dict: malformed
non-dict 'extra' sections broke typing_status_text resolution (uses the
already-coerced extra dict).
Design notes: status text is a side-effect display channel only — never
enters the transcript, no prompt-cache impact. Lifecycle guarantees from
the stuck-status fix family are preserved (per-thread tracking,
clear-on-finish via existing stop_typing paths). Related: #45109
(closed; same direction via lifecycle states), #59010/#51363 (native
task cards — complementary, larger scope).
Adds PlatformConfig.typing_status_text for the two platforms that render
text for the working-state line: Slack's assistant.threads.setStatus
status (hardcoded 'is thinking...') and Google Chat's visible marker
message (hardcoded 'Hermes is thinking…'). None keeps each platform's
built-in default; to_dict omits the field when unset so existing configs
serialize unchanged. Plumbing mirrors typing_indicator exactly (typed
field, from_dict extra fallback, shared-key bridge).
Also documents that Slack's status line requires the assistant:write
scope — without it setStatus fails silently and Slack shows its own
generic placeholder, which previously made the behaviour undiagnosable
from config alone.
/simplify-code findings on the #66222 salvage:
- consume_detached_task_result moves to agent/async_utils.py (shared home);
gateway/run.py and the Discord adapter both had near-identical copies of
the same pattern (a third lives in the telegram adapter). One canonical
implementation, both new callsites import it.
- Reconnect backoff formula min(30 * 2^(n-1), 300) was copied verbatim at
3 sites in run.py (primary watcher x2, secondary-profile reconnect), the
third hardcoding the cap. Hoisted to module-level _reconnect_backoff()
with a single _RECONNECT_BACKOFF_CAP so a future tune can't silently
miss one path.
Behavior-preserving: 70 gateway teardown/liveness/reconnect tests green.
Replace REST-based Discord liveness probe with local WebSocket/heartbeat
state detection. REST success doesn't prove Gateway event delivery — a
half-closed WebSocket can leave Bot.start() alive while REST returns 200.
Now samples ready/open/ACK state and heartbeat latency; consecutive
unhealthy samples emit one retryable fatal code so GatewayRunner rebuilds
the adapter through the existing reconnect path.
Also fixes three lifecycle gaps in the recovery path:
1. asyncio.wait_for() can remain blocked if adapter cleanup swallows
cancellation — now uses bounded asyncio.wait() with task detachment.
2. Multiplexed secondary-profile adapters had no profile-scoped reconnect
owner — now uses one runner-owned reconnect slot per profile.
3. An in-flight turn could send its final text through the disconnected
adapter after a replacement was registered — now resolves the live
same-profile replacement for unsent final responses only (message IDs
never migrate, edits/deletes stay on the old transport).
Adds an opt-in Linux/systemd event-loop watchdog (gateway.systemd_watchdog_seconds,
default 0) for the failure mode where the whole asyncio loop stops making
progress and no in-process liveness task can run. stdlib-only sd_notify,
Type=notify/WatchdogSec generation, READY/STOPPING lifecycle.
Co-authored-by: 王鑫 <wx.xw@bytedance.com>
C1: The watchdog's os._exit(1) bypassed drain_log_queue() and
remove_pid_file()/release_gateway_runtime_lock() — the three things
_exit_after_graceful_shutdown does before exiting. The watchdog's own
logger.critical('shutdown watchdog fired') was silently dropped because
it was still in the async QueueListener queue when os._exit ran. PID
file and runtime lock were stranded on next boot.
W1: Dropped 2MB infographic PNG — dead asset with zero references in
the repo. Binary blobs in git history are permanent; every clone pays
the cost forever.
Authorship: @HexLab98's commits preserved via rebase-merge.
