hermes-agent/gateway
Jason c346f018d4 fix(gateway): evict stale-self-heal agent cache entries pointing at dead sessions
The #54878 self-heal (SessionStore.get_or_create_session) drops a routing
key pointing at a session already ended in state.db and recovers/recreates
a fresh session_id under the same session_key. The #54947 fix (agent-cache
cache-hit guard in gateway/run.py) treats a cached agent whose snapshot
session_id differs from the current session_id, under the same
session_key, as an intentional /resume-/branch-style switch between two
live sibling conversations, and reuses it unchanged to protect the prompt
cache.

These two fixes compose incorrectly: when the #54878 self-heal just fired,
the cached agent's session_id is not a live sibling — it's the dead session
just routed away from. #54947's "different session_id -> reuse freely" rule
reuses it anyway. The stale agent runs the turn, and the post-run "session
split" sync (agent.session_id != session_id) then writes the routing key
straight back onto the dead session_id, undoing the self-heal. This repeats
on every subsequent message until an interrupt (e.g. /stop) happens to race
in before that post-run sync, silently discarding conversation context.

Reproduced live on the engineering gateway (2026-07-12, routing key
agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines
over ~40 minutes, each followed by the dead session_id being reused and
re-synced back, until an interrupted /stop finally let a fresh session
stick — at which point all prior context was gone.

No open upstream issue tracks this specific interaction as of 2026-07-12
(checked #54878, #54947, #59580, #59597, #61220 — all cover adjacent but
distinct edges of the self-heal / agent-cache system).

Fix: before applying #54947's reuse-on-mismatch rule, check (outside the
cache lock, via SessionStore._is_session_ended_in_db) whether the cached
snapshot's session_id is itself ended in state.db. If so, treat it as a
stale self-heal artifact and evict/rebuild fresh -- same as a genuine
cross-process write -- instead of reusing it. Re-validates the peeked
verdict against the tuple actually held under the lock so a race can't
apply a stale verdict to a different (possibly live) cache entry.

Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new
cases: dead-session eviction, live-sibling reuse preserved [#54947 intact],
cross-process invalidation preserved [#45966 intact], same-session_id dead
edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed,
9040 passed, 11 skipped -- all 14 failures verified pre-existing on
unpatched main (confirmed via git stash + re-run), unrelated to this
change.
2026-07-16 00:11:22 +05:30
..
assets chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
builtin_hooks remove: BOOT.md built-in hook (#17093) 2026-04-28 09:50:27 -07:00
platforms fix(gateway): gate profile routing on multiplex_profiles + widen batch-key routing to all adapters 2026-07-15 09:50:05 -07:00
relay feat(relay): consume channel context from the connector (#64649) 2026-07-15 09:31:34 +10:00
__init__.py docs(gateway): mention Weixin in gateway help and docstrings 2026-05-12 17:08:51 -07:00
authz_mixin.py fix(gateway): per-profile pairing whitelist isolation in multiplex mode (#53045 salvage) (#59330) 2026-07-07 18:25:52 +10:00
cgroup_cleanup.py fix: satisfy ruff encoding + windows-footgun lints for cgroup reaper 2026-06-28 02:05:50 -07:00
channel_directory.py fix(gateway): offload channel directory session scans 2026-07-09 06:27:04 -07:00
code_skew.py fix(gateway): refuse model switch on stale checkout to avoid env_float ImportError 2026-06-24 04:16:54 +05:30
config.py fix(gateway): gate profile routing on multiplex_profiles + widen batch-key routing to all adapters 2026-07-15 09:50:05 -07:00
cwd_placeholder.py fix(gateway): resolve terminal.cwd placeholders per backend and mount mode 2026-07-04 13:28:47 -07:00
dead_targets.py fix(gateway): skip confirmed-dead delivery targets (deleted groups, blocked bots) (#55115) 2026-06-29 13:23:29 -07:00
delivery.py refactor(gateway): share error-text blob between not_found classifiers 2026-07-01 15:11:38 +05:30
display_config.py fix: normalize display boolean strings 2026-07-05 06:29:26 -07:00
drain_control.py feat(gateway): suppress home-channel shutdown broadcast on flagged drains (#54824) 2026-06-29 12:18:11 -07:00
hooks.py feat(hooks): expose thread_id and chat_type in agent:start/end context (#41672) 2026-06-07 19:16:36 -07:00
kanban_watchers.py fix(kanban): route notifier wake via profile chokepoint; harden review findings 2026-07-02 00:05:48 +05:30
memory_monitor.py Port from cline/cline#10343: periodic gateway memory logging (#27102) 2026-05-16 12:55:23 -07:00
message_timestamps.py feat(gateway): inject stable human-readable message timestamps 2026-06-16 15:49:59 -07:00
mirror.py refactor: consolidate gateway session metadata into state.db (#58899) 2026-07-05 14:01:03 -07:00
pairing.py fix: merge split gateway pairing stores 2026-07-07 02:18:17 -07:00
platform_registry.py perf(startup): lazy-load gateway platform adapters (#54448) 2026-06-28 15:11:59 -07:00
profile_routing.py fix(gateway): profile routing — conjunctive matching + universal gateway_runner 2026-07-15 09:50:05 -07:00
readiness.py fix(gateway): ground readiness in live runtime state 2026-07-11 08:42:21 -07:00
response_filters.py fix(gateway): tolerate punctuation on silence markers 2026-07-05 02:12:26 -07:00
restart.py fix(gateway): exit 78 (EX_CONFIG) on fatal startup errors, s6 finish script stops restart loop 2026-06-24 16:34:51 +10:00
restart_loop_guard.py fix(gateway,cron): guard cron model-tool path + add auto-resume loop breaker (#30719) 2026-07-01 02:48:36 -07:00
rich_sent_store.py style(profile): frame comments around what the code does 2026-06-30 15:30:06 -07:00
run.py fix(gateway): evict stale-self-heal agent cache entries pointing at dead sessions 2026-07-16 00:11:22 +05:30
runtime_footer.py chore: prune unused imports and duplicate import redefinitions 2026-05-28 22:26:25 -07:00
scale_to_zero.py feat(gateway): scale-to-zero idle detection + dormant-quiesce (Phase 0) 2026-06-24 18:47:18 -07:00
session.py fix(gateway): sanitize sender-name prefix in shared multi-user sessions 2026-07-15 23:48:34 +05:30
session_context.py fix(delegation): route async results to origin session 2026-07-08 07:06:15 -07:00
shutdown_forensics.py chore: ruff auto-fixes — collapsible-else-if, if-stmt-min-max, dict.fromkeys (#23926) 2026-05-11 11:03:29 -07:00
slash_access.py feat(gateway): per-platform admin/user split for slash commands (salvage of #4443) (#23373) 2026-05-10 12:33:54 -07:00
slash_commands.py fix(slack): clear stuck assistant status on /stop and via explicit metadata 2026-07-14 21:31:32 -07:00
status.py fix(dashboard): cache gateway PID status probes 2026-07-09 01:19:07 +05:30
status_phrases.py chore: limit generic status phrases to long-running notifications 2026-07-05 06:29:26 -07:00
sticker_cache.py fix: guard yaml.safe_load, flock unlock, TOCTOU races, and atomic writes 2026-05-19 00:12:41 -07:00
stream_consumer.py fix(telegram): harden flood fallback recovery 2026-07-11 11:13:50 +05:30
stream_dispatch.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
stream_events.py feat(gateway): structured stream-event protocol + Telegram draft formatting parity (#37250) 2026-06-02 00:33:50 -07:00
whatsapp_identity.py fix(whatsapp): resolve LID aliases on modern platforms/ session layout 2026-06-28 02:05:26 -07:00