mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-04-25 00:51:20 +00:00
fix(gateway): cancel_background_tasks must drain late-arrivals (#12471)
During gateway shutdown, a message arriving while
cancel_background_tasks is mid-await (inside asyncio.gather) spawns
a fresh _process_message_background task via handle_message and adds
it to self._background_tasks. The original implementation's
_background_tasks.clear() at the end of cancel_background_tasks
dropped the reference; the task ran untracked against a disconnecting
adapter, logged send-failures, and lingered until it completed on
its own.
Fix: wrap the cancel+gather in a bounded loop (MAX_DRAIN_ROUNDS=5).
If new tasks appeared during the gather, cancel them in the next
round. The .clear() at the end is preserved as a safety net for
any task that appeared after MAX_DRAIN_ROUNDS — but in practice the
drain stabilizes in 1-2 rounds.
Tests: tests/gateway/test_cancel_background_drain.py — 3 cases.
- test_cancel_background_tasks_drains_late_arrivals: spawn M1, start
cancel, inject M2 during M1's shielded cleanup, verify M2 is
cancelled.
- test_cancel_background_tasks_handles_no_tasks: no-op path still
terminates cleanly.
- test_cancel_background_tasks_bounded_rounds: baseline — single
task cancels in one round, loop terminates.
Regression-guard validated: against the unpatched implementation,
the late-arrival test fails with exactly the expected message
('task leaked'). With the fix it passes.
Blast radius is shutdown-only; the audit classified this as MED.
Shipping because the fix is small and the hygiene is worth it.
While investigating the audit's other MEDs (busy-handler double-ack,
Discord ExecApprovalView double-resolve, UpdatePromptView
double-resolve), I verified all three were false positives — the
check-and-set patterns have no await between them, so they're
atomic on single-threaded asyncio. No fix needed for those.
This commit is contained in:
parent
1d1e1277e4
commit
62ce6a38ae
2 changed files with 167 additions and 5 deletions
|
|
@ -2033,12 +2033,26 @@ class BasePlatformAdapter(ABC):
|
|||
Used during gateway shutdown/replacement so active sessions from the old
|
||||
process do not keep running after adapters are being torn down.
|
||||
"""
|
||||
tasks = [task for task in self._background_tasks if not task.done()]
|
||||
for task in tasks:
|
||||
self._expected_cancelled_tasks.add(task)
|
||||
task.cancel()
|
||||
if tasks:
|
||||
# Loop until no new tasks appear. Without this, a message
|
||||
# arriving during the `await asyncio.gather` below would spawn
|
||||
# a fresh _process_message_background task (added to
|
||||
# self._background_tasks at line ~1668 via handle_message),
|
||||
# and the _background_tasks.clear() at the end of this method
|
||||
# would drop the reference — the task runs untracked against a
|
||||
# disconnecting adapter, logs send-failures, and may linger
|
||||
# until it completes on its own. Retrying the drain until the
|
||||
# task set stabilizes closes the window.
|
||||
MAX_DRAIN_ROUNDS = 5
|
||||
for _ in range(MAX_DRAIN_ROUNDS):
|
||||
tasks = [task for task in self._background_tasks if not task.done()]
|
||||
if not tasks:
|
||||
break
|
||||
for task in tasks:
|
||||
self._expected_cancelled_tasks.add(task)
|
||||
task.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
# Loop: late-arrival tasks spawned during the gather above
|
||||
# will be in self._background_tasks now. Re-check.
|
||||
self._background_tasks.clear()
|
||||
self._expected_cancelled_tasks.clear()
|
||||
self._pending_messages.clear()
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue