fix(photon): inspect port listeners off the gateway event loop

`_reap_stale_sidecar` is `async`, but it identified the processes holding
the sidecar port with two blocking helpers called inline:

* `_find_listener_pids` -> `subprocess.run(["lsof", ...], timeout=5.0)`
* `_pid_is_sidecar` -> `subprocess.run(["ps", ...], timeout=5.0)`, once
  per candidate pid

so the inspection can hold the shared gateway loop for 5 + 5·N seconds
while nothing else on it is serviced. It only runs once the /healthz probe
finds something already listening — the orphaned-sidecar recovery path —
and `_reap_stale_sidecar` is awaited from `_start_sidecar`, which runs on
every reconnect (`connect(is_reconnect=True)`). The stall therefore lands
on a live gateway that is still serving every other platform, right when a
crashed sidecar has already left an orphan behind.

Move the whole inspection to one `asyncio.to_thread` hop (one hop rather
than N+1 round trips). The reaping semantics are untouched: SIGTERM for
verified orphans, SIGKILL escalation, and both foreign-listener
RuntimeErrors behave exactly as before.

Same off-the-loop class as the inbound-image decision (#66688) and the
cron-fire verifier.

Adds a regression test asserting both the lsof lookup and the per-pid ps
check execute on a worker thread rather than the loop thread.
This commit is contained in:
Frowtek 2026-07-18 17:17:50 +03:00 committed by Teknium
parent 1b7db6e037
commit a90a2b3c3c
2 changed files with 61 additions and 3 deletions

View file

@ -1115,9 +1115,20 @@ class PhotonAdapter(BasePlatformAdapter):
)
except httpx.RequestError:
return # nothing listening — the normal case
pids = self._find_listener_pids(self._sidecar_port)
stale = [pid for pid in pids if self._pid_is_sidecar(pid)]
foreign = [pid for pid in pids if pid not in stale]
# Off the event loop: _find_listener_pids shells out to `lsof`
# (timeout=5s) and _pid_is_sidecar runs a `ps` per candidate pid
# (timeout=5s each), so this inspection can hold the loop for
# 5 + 5·N seconds. _reap_stale_sidecar is awaited from
# _start_sidecar, which runs on every reconnect — exactly when a
# crashed gateway left an orphan behind — so the stall lands on a
# live gateway that is still serving every other platform. One hop
# covers the whole inspection instead of N+1 round trips.
def _inspect():
found = self._find_listener_pids(self._sidecar_port)
mine = [pid for pid in found if self._pid_is_sidecar(pid)]
return mine, [pid for pid in found if pid not in mine]
stale, foreign = await asyncio.to_thread(_inspect)
if not stale:
raise RuntimeError(
f"port {self._sidecar_port} is in use by another process "