mirrors/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-07-25 17:18:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 6

fix(agent): prevent shared OpenAI client FD-recycle corruption from stale stream watchdog

Browse source 
The streaming stale watchdog was calling
_replace_primary_openai_client() from its polling thread, which closes
the shared client's connection pool. Worker threads from previous
stale-killed attempts may still be unwinding their SSL BIOs, causing
TLS application-data to overwrite SQLite file headers via FD reuse.

This is the same corruption vector documented in #67142 for Anthropic,
where the fix was to never close the shared client from a non-owner
thread. Apply the same pattern to the OpenAI-wire path:

- Stale stream watchdog: skip shared client replacement
- Mid-tool-retry cleanup: skip shared client replacement
- Stream retry cleanup: skip shared client replacement

The request-local client is already closed via _close_request_client_once.
The shared client is replaced lazily by _ensure_primary_openai_client
on the next request, which runs on the owning thread.

Closes #70773.
This commit is contained in:
JonthanaHanh 2026-07-24 21:38:34 +07:00 • committed by Teknium
parent 66cc0075a2
commit e51abb266d
1 changed files with 15 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

33
agent/chat_completion_helpers.py
View file

@ -3492,13 +3492,9 @@ def interruptible_streaming_api_call(agent, api_kwargs: dict, *, on_first_delta=
# already worker-owned-closed by _close_request_client_once
# above; the next attempt builds a fresh one. The shared
# _anthropic_client is never closed from inside a request.
if agent.api_mode != "anthropic_messages":
try:
agent._replace_primary_openai_client(
reason="stream_mid_tool_retry_pool_cleanup"
)
except Exception:
pass
# #70773: same FD-recycle corruption vector for OpenAI.
# The shared client will be replaced lazily by
# _ensure_primary_openai_client on the next attempt.
continue
# SSE error events from proxies (e.g. OpenRouter sends
@ -3557,13 +3553,9 @@ def interruptible_streaming_api_call(agent, api_kwargs: dict, *, on_first_delta=
# above; next attempt builds fresh), so the shared
# _anthropic_client is never closed from inside a
# request — only the OpenAI-wire primary is refreshed.
if agent.api_mode != "anthropic_messages":
try:
agent._replace_primary_openai_client(
reason="stream_retry_pool_cleanup"
)
except Exception:
pass
# #70773: same FD-recycle corruption vector for OpenAI.
# The shared client will be replaced lazily by
# _ensure_primary_openai_client on the next attempt.
continue
# Retries exhausted. Log the final failure with
# full diagnostic detail (chain, headers,
@ -3806,10 +3798,15 @@ def interruptible_streaming_api_call(agent, api_kwargs: dict, *, on_first_delta=
# FD-recycle corruption vector. Nothing further is needed.
pass
else:
try:
agent._replace_primary_openai_client(reason="stale_stream_pool_cleanup")
except Exception:
pass
# #70773: same FD-recycle corruption vector as #67142.
# The shared OpenAI client's connection pool must NOT be
# closed from this watchdog/poll thread — worker threads
# from previous stale-killed attempts may still be
# unwinding their SSL BIOs. The request-local client is
# already closed above via _close_request_client_once.
# The shared client will be replaced lazily by
# _ensure_primary_openai_client on the next request.
pass
# Reset the timer so we don't kill repeatedly while
# the inner thread processes the closure.
last_chunk_time["t"] = time.time()