hermes-agent/gateway
Teknium 19527db731
fix(gateway): per-session turn lease + conversation-scope funnel (#64934) (#67401)
* fix(gateway): serialize concurrent turns per resolved session_id with a turn lease

Closes the serialization half of #64934. The busy guards are keyed by
routing key, but the durable transcript is owned by session_id — and
switch_session() makes the key→id mapping many-to-one (/resume from a
second chat/topic, CLI-continuity rebinding, async-delegation pinning,
topic-binding tip-walks). Two routing keys mapped to one session_id ran
concurrent turns on two different agent objects, invisible to every
per-key guard: flushes persisted in completion order, the identity-marker
dedup swallowed rows, and the second turn ran on a stale history base —
leaving a permanent user;user alternation wedge.

The fix: an asyncio lease keyed by RESOLVED session_id (gateway/turn_lease.py),
acquired in _handle_message_with_agent after session resolution is final
(post switch_session/tip-walk), immediately before the transcript load, and
released in _handle_message's finally on every exit path. Tokens are granted
per (routing key, run generation) so a stale unwind can never release a newer
turn's lease (#28686 ownership lesson). Same-key messages never reach the
acquisition point mid-turn (both routing-key guards hold them), so the lock
is uncontended outside the alias-key route — where the second turn now waits
for the first turn's flush and logs one WARNING naming the session and both
routing keys (pairs with the #67371 tripwire).

Fail-open: a stuck holder degrades to today's unserialized behavior with a
loud ERROR after agent.gateway_timeout — never a wedged session; a degraded
token holds nothing and can't steal the lease. Registry is size-capped and
never evicts a live lease. Persist-disabled review forks never dispatch
through _handle_message, so they cannot contend.

Known limits (tracked on #64934): CLI-continuity cross-process pairs need a
DB-level lease; mid-turn compression rotation leaves a small alias window
for a follow-up at the binding-sync sites.

Validation: 8 behavior tests (alias-key wait + flush order, no cross-session
contention, generation-scoped idempotent release, timeout fail-open without
lease theft, bounded registry, bare-runner-safe release wiring) + E2E against
a real SessionStore reproducing the issue's switch_session alias route —
strict alternation and arrival order preserved.

* refactor(gateway): conversation-scope funnel + mid-turn lease rebind

Completes the #64934 system beyond the point fix. Two structural changes,
both eliminating whole bug classes rather than instances:

1. _clear_conversation_scope — THE single conversation-boundary funnel.
   /new, /resume, auto-reset, expiry finalization, and the
   compression-exhausted reset each carried a hand-copied pop-list of the
   per-session dicts, and the lists drifted every time a new dict was
   added (#48031, #58403, #10702, #35809 were all 'boundary X forgot
   dict Y' bugs). All five sites now make one funnel call driven by the
   _CONVERSATION_SCOPED_STATE registry; adding a new conversation-scoped
   dict means adding one name to the registry, and every boundary picks
   it up automatically. Scope rules documented at the registry: turn-scoped
   state, the monotonic generation counter, and the agent cache are
   deliberately excluded (different lifecycles).

2. SessionTurnLeaseRegistry.rebind — the held turn lease now FOLLOWS
   mid-turn compression rotation. Both rotation sites (session-hygiene
   pre-compression, agent-result session_id swap) alias the same
   _SessionLease object under the new id, so an alias routing key
   resolving the fresh child (topic tip-walk) still serializes against
   the in-flight turn. Closes the rotation-alias window flagged as a
   known limit on #64934. Ownership-checked like release; when the
   target id already has a live lease the rebind fails open with a loud
   WARNING (never a mid-turn deadlock).

Tests: 3 new rebind behavior tests + 5 funnel behavior tests (including
a real-setter drift guard); the two AST change-detector pins in
test_10710/test_48031 were re-pointed at the funnel and the #58403 pin
converted to a behavioral test. E2E: rotation-alias scenario against a
real SessionStore + SessionDB — turn B on the fresh child waits behind
the rotated holder, sees its rows, alternation intact.
2026-07-19 03:49:29 -07:00
..
assets chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
builtin_hooks remove: BOOT.md built-in hook (#17093) 2026-04-28 09:50:27 -07:00
platforms feat(gateway): durable delivery-obligation ledger for final responses (#67181) 2026-07-19 00:45:32 -07:00
relay fix(relay): normalize a 0/negative max_message_length at the descriptor boundary 2026-07-16 07:21:23 -07:00
__init__.py docs(gateway): mention Weixin in gateway help and docstrings 2026-05-12 17:08:51 -07:00
authz_mixin.py fix(gateway): multiplex secret_scope for authz, Slack, webhooks 2026-07-16 05:39:58 -07:00
cgroup_cleanup.py fix: satisfy ruff encoding + windows-footgun lints for cgroup reaper 2026-06-28 02:05:50 -07:00
channel_directory.py fix(gateway): offload channel directory session scans 2026-07-09 06:27:04 -07:00
code_skew.py fix(gateway): refuse model switch on stale checkout to avoid env_float ImportError 2026-06-24 04:16:54 +05:30
config.py perf(gateway): byte-stable session context prompts 2026-07-19 14:58:59 +05:30
cwd_placeholder.py fix(gateway): resolve terminal.cwd placeholders per backend and mount mode 2026-07-04 13:28:47 -07:00
dead_targets.py fix(gateway): skip confirmed-dead delivery targets (deleted groups, blocked bots) (#55115) 2026-06-29 13:23:29 -07:00
delivery.py refactor(gateway): share error-text blob between not_found classifiers 2026-07-01 15:11:38 +05:30
delivery_ledger.py feat(gateway): durable delivery-obligation ledger for final responses (#67181) 2026-07-19 00:45:32 -07:00
display_config.py feat(gateway): live per-tool status line on Slack 2026-07-18 12:28:59 -07:00
drain_control.py feat(gateway): suppress home-channel shutdown broadcast on flagged drains (#54824) 2026-06-29 12:18:11 -07:00
hooks.py feat(hooks): expose thread_id and chat_type in agent:start/end context (#41672) 2026-06-07 19:16:36 -07:00
kanban_watchers.py fix(kanban): route notifier wake via profile chokepoint; harden review findings 2026-07-02 00:05:48 +05:30
memory_monitor.py Port from cline/cline#10343: periodic gateway memory logging (#27102) 2026-05-16 12:55:23 -07:00
message_timestamps.py feat(gateway): inject stable human-readable message timestamps 2026-06-16 15:49:59 -07:00
mirror.py refactor: consolidate gateway session metadata into state.db (#58899) 2026-07-05 14:01:03 -07:00
pairing.py fix: merge split gateway pairing stores 2026-07-07 02:18:17 -07:00
platform_registry.py perf(startup): lazy-load gateway platform adapters (#54448) 2026-06-28 15:11:59 -07:00
profile_routing.py fix(gateway): profile routing — conjunctive matching + universal gateway_runner 2026-07-15 09:50:05 -07:00
readiness.py fix(gateway): ground readiness in live runtime state 2026-07-11 08:42:21 -07:00
response_filters.py fix(gateway): tolerate punctuation on silence markers 2026-07-05 02:12:26 -07:00
restart.py fix(gateway): preserve external supervisor ownership 2026-07-16 05:08:56 -07:00
restart_loop_guard.py fix(gateway,cron): guard cron model-tool path + add auto-resume loop breaker (#30719) 2026-07-01 02:48:36 -07:00
rich_sent_store.py style(profile): frame comments around what the code does 2026-06-30 15:30:06 -07:00
run.py fix(gateway): per-session turn lease + conversation-scope funnel (#64934) (#67401) 2026-07-19 03:49:29 -07:00
runtime_footer.py chore: prune unused imports and duplicate import redefinitions 2026-05-28 22:26:25 -07:00
scale_to_zero.py feat(gateway): scale-to-zero idle detection + dormant-quiesce (Phase 0) 2026-06-24 18:47:18 -07:00
session.py perf(gateway): byte-stable session context prompts 2026-07-19 14:58:59 +05:30
session_context.py fix(delegate): declare stateless channel in one-shot and cron so delegate_task returns results 2026-07-18 00:05:25 -07:00
shutdown_forensics.py chore: ruff auto-fixes — collapsible-else-if, if-stmt-min-max, dict.fromkeys (#23926) 2026-05-11 11:03:29 -07:00
shutdown_watchdog.py fix: drain logs + release PID/lock before watchdog os._exit, drop infographic PNG 2026-07-18 19:53:00 +05:30
slash_access.py feat(gateway): per-platform admin/user split for slash commands (salvage of #4443) (#23373) 2026-05-10 12:33:54 -07:00
slash_commands.py fix(gateway): per-session turn lease + conversation-scope funnel (#64934) (#67401) 2026-07-19 03:49:29 -07:00
status.py fix(gateway): launchd ThrottleInterval + portable respawn-storm circuit breaker 2026-07-18 19:38:21 +05:30
status_phrases.py chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
sticker_cache.py fix: guard yaml.safe_load, flock unlock, TOCTOU races, and atomic writes 2026-05-19 00:12:41 -07:00
stream_consumer.py fix(discord): persist streamed final delivery 2026-07-18 14:01:33 -07:00
stream_dispatch.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
stream_events.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
systemd_notify.py fix(gateway): recover Discord websocket and event-loop stalls 2026-07-18 20:01:55 +05:30
turn_lease.py fix(gateway): per-session turn lease + conversation-scope funnel (#64934) (#67401) 2026-07-19 03:49:29 -07:00
whatsapp_identity.py fix(whatsapp): resolve LID aliases on modern platforms/ session layout 2026-06-28 02:05:26 -07:00