feat(gateway): session activity watchdog, stall notify, compress timeout (#72424)

Three mechanisms to detect and notify when gateway sessions stall silently:

1. Mid-turn activity heartbeats stamped to SessionDB so hermes sessions list
   and hermes status show progress during long turns without new message rows.

2. Stall watchdog: when a busy session has pending inbound and the shared
   activity clock is idle past agent.session_stall_timeout (default 300),
   log a WARNING and notify the user once to try /new. Notify-only; does
   not kill the turn.

3. Compaction timeout: fenceless compress_context callers get a progress-aware
   host budget (compression.context_timeout_seconds default 120 idle,
   compression.context_total_ceiling_seconds default 600 ceiling). On timeout,
   cancel via commit fence, skip compaction without dropping messages, and
   continue the turn.

Closes #72016 (slices 1-3; slice 4 cumulative SSE stream-retry deadline
remains a follow-up).

Cherry-picked from PR #72424 by @fangliquanflq.
This commit is contained in:
fangliquanflq 2026-07-27 23:16:24 +05:00 committed by kshitij
parent 5fde131eb2
commit cfb206fe2e
24 changed files with 2638 additions and 107 deletions

View file

@ -793,6 +793,12 @@ agent:
# Set to 0 to disable the warning.
# gateway_timeout_warning: 900
# Session stall watchdog (seconds). When a busy session has a pending
# inbound follow-up and the agent activity clock is idle this long, the
# gateway logs a WARNING and notifies the user to try /new. Does not kill
# the turn (see gateway_timeout). 0 = disable. Default 300.
# session_stall_timeout: 300
# Graceful drain timeout for gateway stop/restart (seconds).
# Default 0 = no drain: a restart interrupts in-flight agents immediately,
# cleans up, and exits. Set a positive value only if you want a grace