mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-31 19:16:29 +00:00
perf(agent): reuse the per-request OpenAI wire client across sequential LLM calls
Every LLM call built a fresh openai.OpenAI wire client (new httpx pool, TCP+TLS handshake, measured 19.2ms p50 / 35.5ms p95 per call at ~5 calls per tool-loop turn) and closed it when the request finished. Cache ONE reusable wire client on the agent, keyed by the effective client kwargs: - _create_request_openai_client hands back the cached client when the effective kwargs are identical; any change (credential rotation, provider failover, vision default_headers) evicts and rebuilds. - Only a request that produced a response reports a reuse close reason (request_complete / stream_request_complete); error unwinds report *_error_cleanup and really close, so a retry after a request error always builds a fresh pool. - Cross-thread aborts poison the slot: a pool whose sockets were shutdown(SHUT_RDWR) from a stranger thread is never reused (#29507) — the owner-thread close discards it and the next create rebuilds. The holder read and the abort are atomic (under the holder lock) at all three abort sites, so a late abort can never poison the NEXT request's checked-out client. - Worker-side interrupt breaks close the half-read SSE stream on the owning thread before building the partial response; a failed close poisons the slot (otherwise each interrupt leaked one checked-out connection until the pool hit PoolTimeout). run_codex_stream gets the same poison-on-close-failure handling. - Single checked-out slot (in_use): a concurrent call gets an untracked client with the old per-request lifecycle. - release_clients() / close() really close the cached client when idle; if a worker has it checked out they abort the sockets and detach the slot, deferring the FD release to the worker's own close. - MoA facade and Mock passthroughs never enter the cache; max_retries=0 is preserved on all request clients.
This commit is contained in:
parent
a41dc65ba7
commit
82e2c9ce40
6 changed files with 899 additions and 30 deletions
|
|
@ -1351,7 +1351,20 @@ def run_codex_stream(agent, api_kwargs: dict, client: Any = None, on_first_delta
|
|||
try:
|
||||
close_fn()
|
||||
except Exception:
|
||||
pass
|
||||
# A failed close can leave this response's connection
|
||||
# checked out of the httpx pool while the caller's finally
|
||||
# reports a reuse-reason close (e.g. interrupt_check broke
|
||||
# the event loop with collected output) — caching the
|
||||
# client with the leaked connection. Poison the slot so
|
||||
# that close really closes the pool (owner-thread abort;
|
||||
# mirrors the chat-streaming interrupt-break handling).
|
||||
# ``client is None`` means the shared primary client,
|
||||
# which is never reuse-cached and must not have its
|
||||
# sockets force-shut here.
|
||||
if client is not None:
|
||||
agent._abort_request_openai_client(
|
||||
active_client, reason="codex_stream_close_failed"
|
||||
)
|
||||
|
||||
|
||||
def run_codex_create_stream_fallback(agent, api_kwargs: dict, client: Any = None):
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue